Working multiple teams in one Sitecore solution is a question of discipline and organisation. By organisation I primarily mean how and where assets for the sites are stored and organised.
To start with your example however; different sites needing different membership providers. You don't actually map sites directly to a provider, you map them to a default
Extranet by default, but this can be changed.
<site name="sitea" domain="domaina" ... />
<site name="siteb" domain="domainb" ... />
And each membership provider in your solution would be set up, to deliver these new domains to your solution.
As for how to organise yourselves; I recommend following the guidelines laid out in the Helix architecture recommendations. Specifically the section named Patterns, Principles and Conventions that directly discusses everything related to multi-tenant implementations, file structure, organising Sitecore content, security setup and so on.
In short. Yes, there are some more things to consider when working multiple teams in the same solution, but it is definitely doable and done by many teams around the world every day.
Breaking up your solution into two separate Sitecore instances has the added implication of needing a whole new set of licenses. One set for each site. Depending on your customer, this might not be an option at all.