I have a lightly used personal server:
$ free -h
total used free shared buff/cache available
Mem: 985M 445M 140M 51M 399M 327M
Swap: 1.0G 36M 987M
My opinion is that 512MB doesn’t cut it for a mail server.