I have my box setup for email just fine, and it now has two domains linked:
In web, I changed the file directory for mail.example.com from www/default, so it just shows “mail example”. That works just fine.
But now when I go to custom DNS, and set an A record on example.com to another IP where I have my webserver setup, it just redirects back to mail.example.com.
Am I doing something wrong or is there an actual issue here?
Run on your personal computer:
dig @mail.example.com example.com
to verify that the box is serving the correct DNS.
If that’s correct, then run:
dig example.com
to see if your personal computer is resolving the domain to the updated IP address. DNS results are cached throughout the Internet so it can take some time (~1 hour) for changes in DNS to really take effect.
If that’s correct, then run:
wget -S -O/dev/null http://example.com
wget -S -O/dev/null https://example.com
Check that it’s connecting to the right IP address. Which server is sending the redirect? Is the output the same for both http and https?
dig @mail.example.com example.com
dig example.com
wget -S -O/dev/null http://example.com
These three point to the right IP.
wget -S -O/dev/null https://example.com
This however, times out.
Nothing on the actual webserver I’m trying to direct to, redirects. It’s just static HTML files with a line of text.
If wget is connecting to the other server, and you are getting a redirect from that sever, then your problem is with that server and not your Mail-in-a-Box. 
It’s not getting a redirect from the other server though. If I save the result of what wget downloads from http://example.com, it downloads what I have set as the default Apache page.
If I ping example.com, it returns the proper IP but times out, yet wget can download the default Apache page from the same IP/server?
Pings can be blocked as a part of normal network operations, so that may not indicate a problem. If wget is working, then things are working fine.