Server refuses all connections from public address

Have a look at threads like this https://discourse.mailinabox.email/t/how-do-i-disable-ipv6-in-ubuntu-22-04-on-linode/11794

It appears that some MIAB components expect IPv6 internally on the server, even if it’s not available to access the rest of the world. So if your provider doesn’t do IPv6, or you don’t provide a global (external) address, that’s fine but I’d suggest you don’t completely disable IPv6.

IPv6 does involve some “interesting” setup (for reverse DNS stuff), so it’s probably easier (at least initially) to remove any IPv6 external address. There are a few threads about running with/without IPv6.