Sitecore PowerShell Remoting doesn't work with http://localhost/ and http://127.0.0.1/ addresses

Question

I have Sitecore 9.3 with SXA and SPE 6.0 installed in a Docker container.

docker-compose.yml map containers 80 port to 80 on local machine.

ports:
  - "80:80"

I have an additional configuration in hosts file:

127.0.0.1   test.dev.local

Website is available in a browser on my machine by any of these 3 addresses:

1. http://test.dev.local/
2. http://localhost/
3. http://127.0.0.1/

And I can confirm that it is the same website.

But Sitecore Powershell Remoting works only with http://test.dev.local/

You can see that Powershell script was able to return get admin user only when I used http://test.dev.local/ as ConnectionUri

What does need to be changed to allow Sitecore Powershell Remoting to work with http://localhost/ or http://127.0.0.1/ URL addresses?

P.S. I want to use either http://localhost/ or http://127.0.0.1/ because inside the container Windows and IIS doesn't know anything about test.dev.local host. And host is important for SXA Export context(SXA export fails if address is not resolved). I am able to overcome this issue by settings

networks: 
  dev.local: 
    aliases: 
      - test.dev.local

in the docker-compose.yml file. But is not a very convenient solution for my case.