By the default, the renderings in the toolbox are grouped by the folders where they are located. This means that it doesn’t matter where you will put it in under the Available Renderings:
But you can change this behaviour and have toolbox section based on the Available Renderings items when you will check this checkbox on the Available Renderings item:
This will rearrange toolbox and have better control of the sections available in the toolbox.
This is how you can configure the toolbox. But the issue you are having is 100% sure (at least in my opinion) related to your custom theme which interferes with SXA styles for a toolbox. The other possible cause of such behaviour is a script error (also located in a custom theme or injected with one of your custom renderings) which stops the execution of SXA Experience Editor scripts.