…or how not to set up your tiny server
In a surprise to absolutely noone, you need to adapt your Apache configs to your server setup – who knew!? ;]
In my case, I’m running this on a small Linode Nanode VPS with all of 1GB of RAM. In that 1GB I managed to fit:
- This WordPress instance
- Mariadb (for WordPress)
- A separate personal website
- 3 websites served from Docker containers with Apache handling the SSL termination
- A wireguard server (also running in Docker)
- A remote access server
- A private syncthing relay node
- A Watchtower instance to keep my containers up to date
Now, all of these are EXTREMELY low traffic – this is really all just for my personal use – but as you can see, you can fit a surprising amount of stuff into this one measly GB of RAM! ;]
However, I had one problem: the default Apache config allowed for WAY too many workers! The default setup uses mpm_prefork workers, which means Apache spins up one process per connection. Given my tiny websites and their minimal usage, this is totally fine in principle. However, occasionally I do get hit by script kiddies probing for vulnerabilities. In that scenario, a number of simultaneous connections are made as they probe for various vulnerabilities. That is fine in and of itself, however it does lead to OOM kills for mariadb… Wait, huh? Why?
The symptom
This is what I would see in my logs: mariadb gets OOM killed. “Weird!”, I thought, so I set up a general log for mariadb to see what kind of queries were running. However, that turned up nothing.
But then I started looking at my access logs and found that during the times I was getting OOM kills, there were a number of connection attempts to my various websites using non-existent PHP script paths and the like. So that’s when I got suspicious of the number of Apache workers running.
The fix
So it turns out that in the default Apache setup (that I had carried forward from the Debian 10 days), I had this in /etc/apache2/mods-enabled/mpm_prefork.conf:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 150
MaxConnectionsPerChild 0This config would start up to 150 Apache worker processes if there were lots of simultaneous connections. At 50-100MB RAM usage each, you can see how that would quickly overwhelm my poor little Nanode! And then for some reason the kernel always chose to kill mariadb (probably because it uses a little more RAM than the individual Apache workers), so I was left with this red herring of “hey, mariadb is using all this memory, why!?” when in fact it was the fault of all these Apache workers!
To make matters worse, when this would happen I would of course not be able to log into the server via SSH anymore, since it was spending all its time trying to swap and run all these Apache workers. So I could never quite figure out what was actually going on.
Luckily this happened again this morning and I was able to jump on this time and see all the extra workers. That finally made me twig on to what was actually happening! So, I changed the config to this:
StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxRequestWorkers 10
MaxConnectionsPerChild 0Now, this is obviously a VERY limited config, but given my teensy little VPS and my modest needs, it actually works just fine for me. Now I can hammer my node with all kinds of requests, and while a lot of them simply time out, and it becomes very slow to respond, it also no longer hits OOM kills, and recovers immediately after the onslaught is over. That’s really all I need!
Conclusion
So there we have it – a much more sane Apache config for this tiny little instance with very modest needs. Is it “web-scale”? Heck no! Do I need that? Heck no! ;] At least now I will stop getting intermittent alerts from my monitors saying that my website is down… well, at least until the next thing goes wrong. ;]
PS: this is literally the FIRST thing that they talk about in the Apache Performance Tuning docs… Guess I shoulda read that one… ;]
PPS: Yeah, this is old news, but it’s new to me. Sue me! ;]