I'm still new to using SXA so take my opinion with a grain of salt. I will provide some examples to explain my position.
What I can say is that there are enough benefits to using SXA that I would recommend it but I can't say it's all green grass and rose petals. There's quite a learning curve as it relies a lot on SPE to create many things which aren't obvious to find if you need to serialize it all with TDS. It also introduces a number of new concepts and moves many things to the content section of the tree which makes it confusing what to serialize and what not to.
On the up side, it fixes a lot of odd behavior that comes with a stock installation of Sitecore such as managing multisite systems, duplicating content trees properly, providing easy access to placeholder settings, easier domain configuration and having a page local or site global datasource folder. It even has a concept of reusable page chunks (header, footer, nav) which I did't realize how much I needed until I had it. All of these things make it worth the using over building it yourself.
I think the biggest mental paradigm shift for me is with what components you're provided and how they are rendered. One such example is that components are no longer just a rendering item pointing to a razor file or controller method. They are now modules with a scaffolding infrastructure. On the one hand it's a lot of extra. On the other hand much of it is managed by SPE commands and is easy to add groups of components to one or many of your sites consistently.
All the components require you to create a new controller even though they all basically do the same thing. You can hotwire it so they all use the same controller and you should. It's the exception not the rule that you'll need to customize for every component.
The other side of the coin is that the components that come OOTB aren't worth keeping. You really should build your own. For example, search uses 'scope' items which use a lucene type language which is hard to write and are not as flexible as ContentSearch predicates and can really put you in a fix if you already committed to using it but can't fix the type of problems you may come across.
Navigation items do not store settings in datasources. They use rendering parameters. This means if you want to have a sidebar nav across 100 pages and reuse the same settings you'll be doing it by hand. Rebuild them to use datasources.
One of the most troublesome parts of it is how our front end developers lives changed with it. With SXA the styles and scripts are stored in the media library and synced up through a gulp task. This works ok on a local environment. As soon as you need to deploy it to a complex web of systems through permission environments it falls apart. No one I work with wants to deploy theme files through packages. It's tedious error prone and time consuming. You can optionally just create items called 'HTML Includes' in the media library to point to the static files. I'd do that and just deploy the js/css with the build like normal.