Hello everyone,
Thank you so much for your helpful replies and ideas! After a lot of testing and troubleshooting, I’ve finally discovered the root cause of my previous issues.
The Problem (Initial Setup - Missed Configuration):
The core problem was that my Mail-in-a-Box server (running inside a KVM on Machine A) was incorrectly configured due to being behind an FRP (Fast Reverse Proxy) connection. Additionally, there was an unexpected open relay on my server, which has now been resolved.
To visualize the initial problematic setup:
Internet <--> Machine B (FRP Server - Public IP 2) <--> Machine A (Host - Public IP 1)
|
+--> Machine C (Mail-in-a-Box KVM - NAT'ed IP from A)
With this FRP setup, while incoming traffic to Machine B could be forwarded to Machine C, the outgoing traffic from Machine C was incorrectly using the public IP of Machine A. This caused various issues with email deliverability and potentially security.
The Solution (New Architecture - WireGuard Network):
I’ve moved away from the FRP setup and have now implemented a WireGuard private network to connect Machine B and Machine C. This allows for proper IP routing and NAT networking configurations.
Here’s a diagram of the new architecture:
Internet
|
|
+--> Machine A (Main Mail-in-a-Box Server - Public IP 1)
| (Runs KVM)
|
+--> Machine B (WireGuard Gateway - Public IP 2) --- WireGuard Tunnel ---> Machine C (Mail-in-a-Box KVM - Private IP on WireGuard)
(Handles In/Out Traffic) (NAT'ed internally on Machine A)
Detailed Explanation of the New Setup:
Let’s break down the components and traffic flow:
-
Machine A (Main Mail-in-a-Box Server):
- Has its own public IP address (let’s call it Public IP 1).
- Acts as the host for the KVM.
- The original Mail-in-a-Box instance runs directly on this machine.
-
Machine B (Digital Ocean Droplet - WireGuard Gateway):
- Has its own distinct public IP address (let’s call it Public IP 2).
- Acts as the WireGuard gateway for Machine C.
- Will be configured with
iptables rules to handle both incoming and outgoing traffic for Machine C, making it appear as if the traffic originates from Public IP 2.
-
Machine C (Mail-in-a-Box KVM):
- Runs inside a KVM on Machine A.
- Is on a private network managed by KVM/NAT on Machine A.
- Connects to Machine B via a WireGuard tunnel.
- All outgoing traffic from Machine C is routed through the WireGuard tunnel to Machine B.
- Incoming traffic destined for the Mail-in-a-Box on Machine C will be forwarded from Machine B through the WireGuard tunnel.
Traffic Flow:
-
Incoming Traffic to Machine C:
Internet --> Machine B (Public IP 2) --> WireGuard Tunnel --> Machine C (Private IP)
-
Outgoing Traffic from Machine C:
Machine C (Private IP) --> WireGuard Tunnel --> Machine B (Public IP 2) --> Internet
Current Challenges:
I’m currently working on configuring fail2ban. My plan is to install and configure fail2ban on Machine B (the WireGuard gateway server). This will allow me to monitor and block malicious attempts targeting the Mail-in-a-Box instance running on Machine C, as all relevant traffic will pass through Machine B.
In Summary:
The move to a WireGuard network with proper IP routing and NAT configuration on Machine B should resolve the issue of outgoing traffic using the wrong public IP. By centralizing the external network interface and security measures (like fail2ban) on Machine B, I aim to have a more stable and correctly configured Mail-in-a-Box instance within the KVM.
Finally: The main issue was “SMTP Open Relay” nothing related to sendmail method 
