In this post Storage Consumed by /boot/efi and /var/log/journal I was looking to recover some space for my DO Droplet based MIAB instance and noticed an increase in disk utilisation for /boot/efi from a Munin Monitoring point of view (increase to 8.51% from about 3.43% of a 25gb disk - just over 2gb).
Now please bear in mind I can work my way around a Linux machine but I am very much still learning and you know what they say about a little knowledge, so feel free to tell me I’m just being paranoid…
Just prior to the increase there was a spike in Firewall throughput for several days:
Which leads my suspicious mind to think that someone might have been playing around on the machine and deployed malware or similar. Having said all this I’ve seen no adverse effects from any of this other than the potential increase in disk consumption from that point - which if Munin is reporting differently to the command line might not even be real.
One point to note that I need to change in hindsight - my root account remains enabled (but has password based authentication disabled so I can only SSH to it using my private key).
So:
Does anyone else have a similar experience on their MIAB?
Am I being paranoid - /boot/efi would seem a logical place to compromise a machine and hide your tracks…
Unfortunately in the drive to free up space I have deleted most of the logs that would have been useful from this time so have no real means to investigate what happened at this point, unless anyone can identify a possible route to identify what might be running now.
I’ve never had a server hacked, but that just looks like noise to me or maybe some new client, likely a Microsoft one.
The likely value of your server to a hacker is the mail server. I suspect you would hear from people you have emailed that they are receiving spam from you. Then you would hear from admins of other servers they are receiving spam from you. Then you would find you IP address and domains on blacklists, and you would hear from your ISP that they are suspending services until you fix the problem.
If the attack is more targeted, maybe they want to gain access to things you manage with your email accounts.
Some basic things you can do:
Change all of your passwords and keys.
Configure external firewall to only permit destination port 22 from your IP addresses you use to SSH into MiaB.
Add AllowUsers username to /etc/ssh/sshd_config and service sshd restart
Run grep username /var/log/auth.log (and its archive files) and verify the IP addresses are only your IP addresses.
Run grep 'r=<username@example.com' /var/log/mail.log (and its archive files) and verify the IP addresses are only your IP addresses.
Reinstall your desktop operating system you use to access MiaB, or install a fresh one to a different drive and change SSH keys and admin passwords from the newly installed operating system.
Some good suggestions which I’ll follow up on. There is also the option I’ve subsequently discovered to re-build the server as that retains the IP address and would be a good recovery test anyway. When re-built I can assign new keys, create an admin equivalent, disable root and add the basic controls you’ve highlighted.
This is a small MIAB with only personal email so I guess there is little value to any real hacker and if it had been compromised to use as a SPAM relay I would probably already have seen some evidence as you’ve suggested so feeling a little less queasy now.
If I detect anything further that’s out of the ordinary I’ll report back.
