I found something to go off of with the validation button in the Experience Editor's ribbon which also opens a modal. Copied that item that represented the validation button and edited some fields in the presentation details of the large button. I had to edit the command which had to point to my javascript command.

I also needed to edit the PageCodeScriptFileName to point to my javascript file.

I made a request because I had some functionality I wanted to do server side (get the url of an item) but if I didn't have that requirement then the added request wouldn't have been necessary.
Button Item Renderings:
Speak js:
define(["sitecore", "/-/speak/v1/ExperienceEditor/ExperienceEditor.js"], function (Sitecore, ExperienceEditor) {
Sitecore.Commands.OpenModal =
{
execute: function (context) {
ExperienceEditor.PipelinesUtil.generateRequestProcessor("Custom.OpenModal", function (response) {
var dialogFeatures = "dialogMinWidth: 850px;";
ExperienceEditor.Dialogs.showModalDialog(response.responseValue.value, "", dialogFeatures, null);
}).execute(ExperienceEditor.generatePageContext(context, ExperienceEditor.getPageEditingWindow().document));
}
};
});
Request:
namespace Common.CustomSitecore.Requests.SharedContent
{
public class OpenModal : PipelineProcessorRequest<PageContext>
{
public OpenModal()
{
}
public override PipelineProcessorResponseValue ProcessRequest()
{
var item = base.RequestContext.Item;
var pipelineProcessorResponseValue = new PipelineProcessorResponseValue()
{
Value = new UrlString($"/sitecore/client/Your Apps/MyModal?sc_lang=en&itemUrl={item.GetUrl()}").ToString()
};
return pipelineProcessorResponseValue;
}
}
}
Request config:
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
<sitecore>
<sitecore.experienceeditor.speak.requests>
<request name="Custom.OpenModal" type="Common.CustomSitecore.Requests.SharedContent.OpenModal, Common"/>
</sitecore.experienceeditor.speak.requests>
</sitecore>
</configuration>