EMAIL SERVER HACKED! ~ Postfwd + Swatchdog Setup Guide

[ZOMG MY EMAIL SERVER GOT HACKED!!!!]

But not in the typical way you would think. In fact nothing was technically wrong with my mail setup, no open relays or anything like that. No data was leaked, nothing was lost except hours of my time configuring new stuff so this HOPEFULLY can’t happen again. I could place the blame on linux, but really the only reason this was able to happen was understandable human error.

What happened was some bot that tries common usernames and passwords was able to access my smtp server directly with a valid username and password. This bot then began using it to spoof tons of spam mail to random 3rd world countries like Australia. Thankfully it didn’t use the shystudios.us domain name! The moment I was informed of this by my host (thank you gex) I was able to shut it down, so really the damage was minimal. Most larger email providers immediately rate limited my server by deferring the mails. As far as I can tell I can still send email to the large providers so I think everything is back to normal. One funny thing I noticed is that after this happened a bunch of scanning services like shodan started showing up in my server logs for a few days.

So how did this happen? Well a few things had to go wrong. The big ones were my preconceived notions of how things work. The other one was using the password “steam” for my user “steam” which I use to run steamcmd to host an unlisted test gmod server occasionally. The bot was able to guess the username and password because they were the same. Now I was under the impression of a few things here. ONE reason is my server uses self signed certs and if you try to login without those certs fail2ban immediately should ban you. Another was the ‘mail’ user group. I was completely convinced that any user not in the ‘mail’ group was UNABLE to access any mail functions like sending or receiving email.

Well it turns out the ‘mail’ group in linux just doesn’t do anything at all, or at least I don’t think it does. Finally, my fail2ban was not working with the postfix-sasl filter. Which is what SHOULD (I THINK) stop login attempts to the smtp server. All because of a single invalid “port” name in the config. It was like this since I set up my mail server on November 11th 2022. So sadly it didn’t make a whole year without problems. Fail2ban did print a message out in the log but I just never caught it, and every other filter was working fine so I never had any reason to think something was wrong.

A lot of “tech” youtubers and other idiots always like to say that you should never host your own email server. But there is no reason NOT to host your own mail server if you are capable of doing it. Hosting an email SERVICE is different, you probably do NOT want to host email for other people. Also you probably shouldn’t use weird setups like local mail relays using dynamic dns bullshit since it is much more annoying to correctly configure. But if you are an individual or small business then you absolutely should be using your own email server!

What happened to me is probably the worst case scenario situation, minus total loss of my server or some unknown exploit. A valid 1st try password guess which allowed a bot to send some spam for a very short amount of time, THAT’S IT! And now with my new setup even if someone can perfectly guess a username and password the first try I will be immediately notified if they send spam and it will automatically be stopped (I HOPE!). This is achieved using two pieces of software called postfwd and swatchdog, but first I will go over a few things I did before setting those up.

[Fixing The Problem!]

When I was first made aware of the problem the first thing I did was stop the postfix and dovecot services, thus shutting down any possible way of sending email. I then locked the steam account using the command “passwd -l steam”, if I had done this when I first made that account this whole situation wouldn’t have happened. Next I did something VERY important, I flushed out all pending mail from postfix. If I didn’t do this then the moment I relaunched the postfix service it would begin re-sending the zillion spam mails that were queued up. I did this by running the following command:

postsuper -d ALL 

(DELETES all mail in the postfix queue)

Then I fixed fail2ban. In the config file that enables my postfix-sasl filter the port argument had “imap3” in the list. Well it turns out “imap3” just completely broke the entire filter so I removed it and now suddenly fail2ban was able to ban smtpd logins! Here is what it should look like:

[postfix-sasl]
enabled   = true
port      = smtp,ssmtp,imap,imaps,pop3,pop3s
filter    = postfix-sasl
logpath   = /var/log/mail.log 

I should mention I am running debian 11 with systemd so everything here is going to be based on that, it might change in the future (hopefully for the better). If you are running something like devuan I am sure you can figure out the alternatives, however I make extensive use of systemd. For example to read only logs relating to postfix I run the following command:

journalctl -e -u postfix*

(the -e jumps to the most recent logs and -u whatever* will only show logs containing that word)

[Setting Up Postfwd]

My main goal was to make it so that it would be impossible to spam with a valid user account. To do this I use postfwd which is a rule based filter that can be made to work with postfix. I installed this like any other software on debian, however the out of box configuration is completely broken. All of the official documentation is bad too. This is mainly why I am writing this article, because there are probably other people that want to have this EXACT setup. To begin configuration I add the postfwd user account and group.

groupadd postfwd

(add the postfwd group)

useradd -g postfwd -d /var/empty -s /bin/false -c "postfwd daemon user" postfwd

(add the user postfwd with no home dir and no shell)

passwd -l postfwd 

(LOCK the account so only root can access it via su, -l is better than -d because -d only removes the password and -l makes it so the password is NEVER correct)

Next you need to change the default postfwd config so it will actually work. This is located in “/etc/default/postfwd” so run the command:

nano /etc/default/postfwd

(this file should exist)

Change STARTUP to 1

Change the username in the RUNAS to postfwd

(this is the user that was just made, however on my system I also have the user “postfw” which I think might have been automatically created when I installed the software? Idk I didn’t make it but it exists in my etc/passwd so maybe just use “postfw” and skip adding the “postfwd” user?)

In ARGS change the summary to something less annoying, I have mine set to 2000 by default it is 600 aka 10 min, this just determines how often postfwd prints out the stats to the log.

Finally, change the CONF path to where you want the config file to live, this is what I use:

CONF=/etc/postfix/postfwd.cf

Now you still are not done fixing the program’s default config, next you need to fix the init script.

nano /etc/init.d/postfwd

Scroll down about to where it says “stop)”

by default the start-stop-daemon command will be “start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE && rm -rf $PIDFILE”. This is incorrect, you need to add “--user=${RUNAS}” after the $PIDFILE and before the &&. It should look like this:

start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE --user=${RUNAS} && rm -rf $PIDFILE

now run

systemctl daemon-reload 

I only figured this out because of this french website that explained the exact same issue I was having! [Translated page]

Now it is time to create the postfwd ruleset. I put mine in the postfix folder, as defined in the /etc/default/postfwd confg above.

nano /etc/default/postfwd/postfwd.cf

Here is my config:

## Ruleset

#ratelimit
id=RULE-RATE
        sasl_username=~/^(\S+)$/
        action=rcpt(sasl_username/10/600/REJECT not allowed)
# reject if same sender recip
id=RULE-SAME
        sender==$$recipient
        action=REJECT not allowed

There are two rules here, the first “RULE-RATE” is the rate limiting rule obviously. The ID can be anything you want but should be somewhat unique so you can actually find it in your logs. sasl_username is defined as some kind of mystery regex that I can’t decipher, all I know is that it just werkz. The action it takes is RCPT, which is some kind of cumulative counting system with four arguments separated by slashes, the first being the sasl_username. This is followed by two numbers, the first is the number of messages that can be sent within the 2nd number which is the period of time in seconds. So this rule allows 10 messages to be sent every 10 min before rejecting the messages.

This is a perfect amount for me and might actually be too high since it is VERY unlikely I will ever send more than 1 message per minute but it allows me to send UP TO 10 in a 10 minute period. I recommend setting this to something low like 3 for the time being so you can test it out to make sure it is working later on.

Before you test it though, you have to configure postfix to actually use postfwd, which is easy. You need to modify your postfix main.cf to have these two lines

smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unknown_recipient_domain, check_policy_service inet:127.0.0.1:10040

smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10040

The smtpd_recipient_restrictions line might be different for you, really you just need to add “check_policy_service inet:127.0.0.1:10040” to the end of whatever you already have there. BUT this is not enough, you need to also add the smtpd_end_of_data_restrictions parameter because if a mail is allowed via the other options then it just skips checking with the rest of the options, IE it never sends the mail to postfwd. So really you might be able to get away with adding JUST the smtpd_end_of_data_restrictions line but you might as well have the policy service check in both just in case.

Now you can enable and run postfwd. Remember to never “reload” the service, I also always just do start/stop and never restart because it seems like it sometimes breaks. Run these commands to get postfwd going:

systemctl enable postfwd.service 

systemctl start postfwd.service

systemctl restart postfix

(I don’t think you need the “.service” but I usually add it anyway since it always auto completes to that.)

[Getting EMail Alerts with Swatchdog AKA Swatch]

Swatchdog is a perl script that monitors log files and can do actions based on what it sees. For my use case, I want to have an email sent to me any time the rate limit rule is triggered. If you thought postfwd was weird to configure, strap in because swatchdog is even less ready to go out of the box on debian! To get started install swatchdog, I think the package name is just “swatch”.

The cool thing with swatch is that you can use it to monitor really any files so you can apply this to ANYTHING you want. I believe swatchdog is intended to be run as a user with no permissions, however when running as a user without permissions it is unable to directly access the system log. So in this setup I just run as root but it would be pretty trivial to have another piece of software constantly running which ‘clones’ the system log into another file that a swatchdog instance without permissions could access.

First I am going to create a directory where the swatchdog pid files can live and a place to store my config:

mkdir /var/run/swatchdog

mkdir /etc/swatchdog

Now I will create the config, this can be named anything and can go anywhere so long as swatchdog has permissions to open it. Since it is running as root here it just goes in “/etc/swatchdog/” I call the file “email_limit_alert.cfg”

nano /etc/swatchdog/email_limit_alert.cfg

Here is what it should be inside:

watchfor /id=RULE-RATE/
        mail addresses=shy@shystudios.us,subject=EMAIL_RATE_LIMIT_ALERT

The config is pretty simple to understand here, and swatchdog can do a lot more than just this. The “watchfor” command will look for a perl regex, so in this case it watches for the exact string between the two slashes “id=RULE-RATE” specifically. Here is a snippit of what comes up in my log when the postfwd rate limit rule is triggered:

Oct 17 06:03:26 shyserver postfwd2/policy[000]: [RULES] rule=0, id=RULE-RATE, queue=xxx, client=unknown[my ip address], user=shy, sender=, recipient=, helo=<[my local lan ip?]>, proto=ESMTP, state=END-OF-MESSAGE, rate=rcpt/4/0.00s, delay=0.00s, hits=RULE-RATE, action=REJECT not allowed

As you can see I am watching for “id=RULE-RATE” specifically, but you could also watch for “hits=RULE-RATE” or even “action=REJECT” if you want. I chose this because it will trigger swatchdog IMMEDIATELY when the rule is caught. Postfwd will occasionally print its status to the log with messages like:

Oct 17 08:56:20 shyserver postfwd2/master[000]: [STATS]   1 matches for id:  RULE-RATE

So I don’t want to JUST watch for “RULE-RATE” because it will send me an email every single time postfwd logs its status.

The last parts of the swatchdog config are easy to understand, it just sends me an email with the subject “EMAIL_RATE_LIMIT_ALERT”. The sender of this email will be whatever user swatchdog is running as, so it will be ROOT in this case but if you are running it as a different user make sure that user is allowed to send mail.

(Who am I kidding? Every user clearly can send mail on linux by default and I have no idea how to limit who can and cant send mail!)

We are not done yet, next is the fun part! You have to create from scratch your own systemd service, which is actually quite easy. You can name this anything you want. I put this file in “/etc/systemd/system/” and I call it “swatchdog_email_limit.service”

nano /etc/systemd/system/swatchdog_email_limit.service

These are the contents:

[Unit]
Description= Swatchdog email rate limit notifs

[Service]
Type= forking
Restart=on-failure
RestartSec=10
User= root
WorkingDirectory=/var/run/swatchdog
ExecStart= /usr/bin/swatchdog --config-file=/etc/swatchdog/email_limit_alert.cfg --script-dir=/var/run/swatchdog -t /var/log/syslog --daemon

[Install]
WantedBy=multi-user.target

once you have this reload systemd again:

systemctl daemon-reload

now you should be able to start/stop swatchdog using:

systemctl enable swatchdog_email_limit.service
systemctl start swatchdog_email_limit.service

The systemd service config is pretty simple to understand, the important things are setting the type to forking, setting the user correctly, and having all the correct arguments for swatchdog. Most importantly is the -t argument which tells swatchdog what file to read. Postfwd logs to /var/log/syslog by default so it needs to be set to that or it will not work.

To test this out, to make sure everything is working you could attempt to trigger the rate limit by sending a few messages. If you set it to 3 like I recommended earlier, that means you can send 3 messages and the 4th will be rejected. When that happens you should also receive an email. Alternatively you could change your swatchdog watchfor line to be “id=RULE-SAME” this is the 2nd postfwd rule that was setup and is a lot easier to trigger since you just need to send yourself a single email. This is useful for troubleshooting but make sure to change it back later on.

Remember any time you change swatchdog or postfwd’s config you need to restart those services. If things break journalctl -e and htop is your friend. Both of these programs can get stuck in a running state where systemd can’t stop them if not configured correctly, I use htop to close them out when that happens.

[Closing Remarks]

Hopefully this helped you out, I might make a video on this in the future too because honestly, this functionality should be included in every single email server setup guide by default. (so let me know if I should change anything here before I record that vid) People need to know about this. It is a shame it is so complicated to get working especially on an os like debian where stuff normally JUST WERKZ. The documentation for both postfwd and swatchdog are LACKING at best and at worst completely incorrect in some instances.

Imagine if everyone who ran a small email server like mine had this functionality, it would pretty much eliminate the bots that randomly attempt to log into stuff, since even if they DO get in, the best they can do is send 10 emails instead of 10zillion. Though I wonder how this works when they are spoofing the sender name to something random? Who knows maybe this entire setup would be ineffective. Really, I find email spam so strange like why would anyone ever bother sending spam? how is it even created? And does anyone even fall for it? I guess so but I’d assume email spam is going to become a lot less effective soon since the baby boomers who believe anything anyone on a screen tells them are not gonna be around for much longer.

Could it be that email spam is all created by some kind of large corp or rogue gov who’s main purpose is using spam as a way to halt independent communications through the use of spam blacklists that small independent providers are permanently added to that are used by all major mail providers!?!?? The main reason I run my own email server is because my old email, shy@memeware.net was getting totally filtered by all the large email providers. In fact every cock li email address is currently filtered which I think is a heinous crime. It isn’t getting sent to spam, it is just getting REJECTED! So if you ever sent me an email and didn’t get a reply that might be why! Why does a private individual like myself suffer because someone decided my email provider isn’t on their special good boy list?

PROBABLY because cock.li is the only email provider that isn’t a honeypot! Much like the US Postal service, email is one of if not the MOST open, decentralized, and reliable communications systems in existence. All the modern so called “private” chat programs ain't got shit on email. There are rumors of companies or gov regulations trying to make a phone number required to have an email address, which is such a blatant move to track people it is laughable.

Remember, you don’t need a real name, phone number, or even real return address to write and mail a physical letter via the post office! The post office is COMPLETELY anonymous if you want it to be. You can physically mail things that are a LOT more harmful than what you can email on-line! It is entirely about tracking and control. The main proponents of this are large corporations who want to have closed networks that can only be accessed if you are in their little APPROVED circle.

This should be completely illegal, no mail provider should be able to fully blacklist every single user of another email service. It is complete discrimination and may even be tortious interference in some cases! Gmail and the other large mail providers are actually the biggest source of email spam anyway! It is already difficult enough to interface with the large providers, you need weird ‘verification’ stuff like dmarc and DKIM just to get emails through. On the other hand, large providers DO by default accept an email server’s self signed messages. Unlike web browsers which freak out when you self sign your ssl certs, email still just works. I don’t have to worry about people decrypting my emails using a CA masterkey or some shit because I don’t let other men sign my certs! All private businesses should be using self signed certs!

If YOU want to host your own mail server and need a VPS provider that is cool with that, go checkout cockbox and use my referral link because it helps pay for THIS server! I have already had a few cool people do this and it will REALLY help in the event where I lose my bitcoin guy!

(cockbox is bitcoin only and I don’t handle crypto I just get someone else to do it.)