From a architectural point of view there is no reason not to have everything in a single project. This will certainly make load and build times quicker - if that is the aim.
The reason for splitting up modules into projects is to clearly define boundaries for modules so that dependencies are as visible as possible, which increases productivity, reduces cost of maintenance and simplifies extensibility.
In other words: if you are comfortable with having your development team managing cross-module dependencies within a single Visual Studio project there is no reason to split up into several projects. There are even tools which can help you control the module dependencies in solutions with fewer projects.
Projects Define Boundaries
However, you should keep in mind that your project lives beyond the initial development phase - and most often beyond your and the initial development team's involvement. In a typical application lifecycle, the technical debt, dependency creep and other productivity decreasing factors happens in the delivery and maintenance phases. The ambitions of clean code and architecture often crumbles with deadlines, team handovers and quick patches. Solution structure is there to define the architecture - both in the short and long run. Defining boundaries as clearly as possible is very helpful in making sure your architecture stays clean.
I recommend watching Anders Laubs talk from the Helix workshop at Symposium 2016 which covers boundaries and why we need them: https://youtu.be/C1OvZOVYous?t=14m38s.