Thanks for the code Koen, your fix is the same fix that SXA uses to change the size of their dialog. So there is validity to your solution.
Here is what I came up with. Doesn't require modification of any Sitecore dll code. I use JavaScript to increase the size.
SelectRendering.xml
Here is my SelectRendering.xml. Notice I added a script link to a new js file called dialog.js. https://gist.github.com/dnstommy/6af6cd4905bd226ef7e1b01621c0cb18
<?xml version="1.0" encoding="utf-8" ?>
<control xmlns:def="Definition" xmlns="http://schemas.sitecore.net/Visual-Studio-Intellisense">
<Sitecore.Shell.Applications.Dialogs.SelectRendering>
<FormDialog ID="Dialog" Icon="Core/32x32/open_document.png" Header="Open Item"
Text="Select the item that you wish to open. Then click the Open button." OKButton="Open">
<Stylesheet Src="SelectItemWithThumbnails.css" DeviceDependant="true" />
<Script Src="/sitecore/shell/Applications/Dialogs/SelectRendering/dialog.js" />
<CodeBeside Type="Client.Web.Custom.Dialogs.CustomSelectRenderingForm, Client.Web"/>
<DataContext ID="DataContext" Root="/"/>
<GridPanel Width="100%" Height="100%">
<GridPanel Width="100%" Height="100%" Columns="3" GridPanel.Height="100%" Style="table-layout:fixed">
<Scrollbox Width="100%" ID="TreeviewContainer" Height="100%" Class="scScrollbox scFixSize scFixSize4" style="border-right: 2px solid #474747;" Background="white" Padding="0px" GridPanel.Width="200px" GridPanel.Height="100%">
<TreeviewEx ID="Treeview" DataContext="DataContext" ShowRoot="true" Click="Treeview_Click" />
</Scrollbox>
<VSplitter ID="TreeSplitter" GridPanel.Class="scThinSplitter" Target="left" />
<Scrollbox ID="Renderings" Width="100%" Height="100%" Class="scScrollbox scFixSize scFixSize4" Background="white" Padding="0px" GridPanel.Height="100%">
</Scrollbox>
<Tabstrip ID="Tabs" Width="895px" Height="100%" Background="white" Padding="0px" style="position:absolute; width:100%;top:0px"/>
</GridPanel>
<Border ID="PlaceHolderNameBorder" Visible="false" style="padding:12px 0px 0px 0px">
<GridPanel Columns="3" Width="100%">
<Literal Text="Add to Placeholder: " GridPanel.NoWrap="true" />
<Space Width="4" />
<Edit ID="PlaceholderName" Name="PlaceholderName" GridPanel.Width="100%" class="scQuirksBoxModel" Width="100%" />
</GridPanel>
</Border>
</GridPanel>
<Border ID="OpenPropertiesBorder" def:placeholder="Buttons" Visible="false" style="padding:0px 0px 8px 0px">
<Checkbox ID="OpenProperties" Header="Open the Properties dialog box immediately." />
</Border>
</FormDialog>
</Sitecore.Shell.Applications.Dialogs.SelectRendering>
</control>
dialog.js
Here is dialog.js. It processes out of the iframe container and finds the surrounding div that we need to resize. Then we also resize the iframe itself. https://gist.github.com/dnstommy/448bd832c79cb28b77a26a3e28b35d50
document.addEventListener('DOMContentLoaded', function(){
var parentDiv = parent.document.getElementsByClassName("ui-dialog ui-widget ui-widget-content ui-corner-all ui-front ui-draggable ui-resizable");
if (parentDiv)
{
parentDiv[0].style.width = "1100px";
parentDiv[0].style.height = "600px";
}
var iFrame = parent.document.getElementById("scContentIframeId0");
if(iFrame)
iFrame.style.height = "600px";
}, false);
Code Behind
Anyone coming across this that needs the code behind to make tabbed renderings, it can be found here. https://gist.github.com/dnstommy/c2ea97506b0c4d59f17c58358ffaa160