Mitigating a DoS attack with GoAccess and Cloudflare

In this off-topic post, I’m going to discuss some behind-the-scenes “fun” that may happen when managing the web server used to host the CNX Software website. From time to time, the server becomes unreachable, but I can still access its console, and notice a very high CPU load (over 100) on a VPS with four cores, while the CPU load is typically 0.5 to 2 under normal circumstances.

That’s usually due to a DoS (denial of service), DDoS (distributed denial of service), or some bug I can’t identify. An easy way to solve this issue is to log in to Cloudflare and set the “Under Attack Mode” to on.

Cloudflare Under Attack Mode

It will show all visitors a JavaScript challenge the first time they visit the website, and the CPU usage typically drops back to normal level within a minute or so. That means I can access my website and so do regular visitors. It’s now without drawback though, as the RSS feed won’t work anymore, and some browsers seem to have some issues with the JavaScript challenge going into a loop.  So I typically turn it on temporarily for a few hours until the (D)DoS attack is gone.

In the case of a DoS attack where one or only a few hosts may be the cause of the high CPU usage, as opposed to a DDoS where many hosts are involved, it’s also possible to use GoAccess open-source web analytics application. to monitor the IP addresses with lots of requests. In a Ubuntu/Debian server, it can be installed as follows:


If one would want to analyze an access.log generated by nginx, we can run the following command:


and select “Common Log Format”.

Goaccess Configuration

It will process all the entries in access.log and provide a summary of key metrics. But it’s not super useful to fight against DoS attacks since the data not only contains recent entries, and in my case, it was about 5 days of traffic, so the top IP addresses were from Googlebot and Bingbot which are both so-called “good bots”.

Luckily the tool can also only show real-time statistics. We’ll first need to edit /etc/goaccess/goaccess.conf configuration file to enable the parameters shown in the screenshot above. It’s just a question of uncommenting a few lines:


Now we can disable “Under Attack Mode” in CloudFlare and run the following command:


We can let it run for a minute or two, and will show the most various metrics including the most accessed files, the top visitor IPs, and so on.

GoAccess Dashboard to list IP addresses used for a DoS attack

The next step is to check the IP addresses. For example, 2a06:98c0:3600::103 is a CloudFlare IPv6 address designed for workers. While it could be an issue, I’m not going to block that one. The 66.249.65.xx 66.249.75.xxx addresses typically come from various Google bots, so we’ll leave those alone too. But it’s unclear what 4.227.36.7 is for and some websites report it as a spam IP address, so I’ve added it to the IP Access Rules in Cloudflare->Security-WAF->Tools along with other suspicious IP addresses reported in GoAccess and enabled a “Managed Challenge” for each of them to try to mitigate the current DoS attack.

CloudFlare WAF Tools Block IP address

After a while, it looks like it partially worked as the CPU load dropped to 3 to 5. I’m not sure it’s the best solution, but that’s what I’m doing to try to keep the website running. It’s a continuous fight since the bots will change IP addresses and attack vectors, so I’m also using other mechanisms, but those are WordPress-specific.

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX RK3588 mini-ITX motherboard
Subscribe
Notify of
guest
The comment form collects your name, email and content to allow us keep track of the comments placed on the website. Please read and accept our website Terms and Privacy Policy to post a comment.
5 Comments
oldest
newest
Marten
Marten
8 days ago

Have you enabled rate limiting rules on Cloudflare? You should not have to manually enable under attack mode when everything is properly configured.

Marten
Marten
7 days ago

Browse around the site quickly and see if you are getting rate-limited, if not it’s perfectly estimated. Check the WAF statistics periodically to see if you are blocking ‘normal’ looking clients. Have you looking into the cache settings aswell? This can reduce the load dramatically.

Carlos
7 days ago

Since I work in webhosting industry this is quite common issue there are ways to mitigate some of this, malicious bots are getting more advanced and even the javascript checks Cloudflare uses are getting defeated from time to time. The approach I would take since you’re running WordPress, anything touching xml-rpc.php wp-login.php admin-ajax.php wp-comments-post.php, GET requests with ‘?s=’ should all be set to Managed Challenge since all of these requests are dynamic and can’t be cached so will hit your server directly, where as requests to /feed/ exclusively should be allowed, but make sure Cloudflare is always delivering a cached… Read more »

Boardcon Rockchip RK3588S SBC with 8K, WiFI 6, 4G LTE, NVME SSD, HDMI 2.1...