Unfortunately, you can not avoid that dependency in EXM 3.4.
The CD's do not have access to the Marketing Operations part of Sitecore, which is needed for List Subscription and Message Sends. Therefore, the Client API uses a direct connection to the CM.
In Sitecore 9.0.1 and beyond, this dependency is removed and replaced with a distributed message queue that the CD creates a message for all Client API operstions and adds it to a Rebus message queue.
The CM server receives the message through Rebus and handles the work. This prevents the CD's from needing a direct connection.
But, this is not in EXM 3.4.
If you need to avoid this dependency, it is suggested to Upgrade Sitecore to the most current version.