I see the issue that you described. Actually the problem is in the `Sitecore.Shell.Applications.Workbox.WorkboxForm` class which represents the *Workbox Form*.

There are two methods:

        /// <summary>Refreshes the page.</summary>
        protected virtual void Refresh()
        {
          this.Refresh((Dictionary<string, string>) null);
        }
    
        /// <summary>Refreshes the page.</summary>
        /// <param name="urlArguments">The URL arguments.</param>
        protected void Refresh(Dictionary<string, string> urlArguments)
        {
          UrlString urlString = new UrlString(WebUtil.GetRawUrl());
          urlString["reload"] = "1";
          if (urlArguments != null)
          {
            foreach (KeyValuePair<string, string> urlArgument in urlArguments)
              urlString[urlArgument.Key] = urlArgument.Value;
          }
          Context.ClientPage.ClientResponse.SetLocation(WebUtil.GetFullUrl(urlString.ToString()));
        }

In general, the problem is in the `Context.ClientPage.ClientResponse.SetLocation(WebUtil.GetFullUrl(urlString.ToString()))` line. And is in `WebUtil.GetFullUrl(urlString.ToString())` method particularly. This method always returns the URL with the port extension. 

Then, looking at details of `GetFullUrl` method where we can find that it calls GetServerUrl:

    public static string GetFullUrl(string url)
    {
      Assert.ArgumentNotNull((object) url, nameof (url));
      return WebUtil.GetFullUrl(url, WebUtil.GetServerUrl());
    }

The `GetServerUrl` method looks like:

    public static string GetServerUrl()
    {
      return WebUtil.GetServerUrl(false);
    }

Where the `false`, is `forcePort` parameter, which is always 'false'.

Let's skip a calls chain and have a look at the final view of `GetServerUrl` method:

    public static string GetServerUrl(Uri url, bool forcePort)
    {
      if (url == (Uri) null)
        return string.Empty;
      string scheme = url.Scheme;
      string host = url.Host;
      string str1 = url.Port.ToString();
      string str2 = string.Format("{0}://{1}", (object) scheme, (object) host);
      if (forcePort || str1 != "80")
        str2 += string.Format(":{0}", (object) str1);
      return str2;
    }

Looking at this method we can see that even `forcePort` is `true`, the URL will still include port as it is not equal to 80.

So, it is why you always have the port in your URL. 

You can create your `CustomWorkboxform` class and inherit it from the 'WorkboxForm'. Then try to override `protected new void Refresh(Dictionary<string, string> urlArguments)` method (using `new` keyword as this method is not virtual). Then you will have to update `Workbox.xml` with new CustomWorkboxForm code behind implementation (\sitecore\shell\Applications\Workbox\Workbox.xml). There can be some another difficulties with overlapping other methods. But idea might work!