The site has slowed down, the server log is full of repeated requests from a single IP, and real users are abandoning the page halfway through. If you've just reached this point, you're probably thinking about rate limiting. But before you write a simple rule in Nginx and apply it to the entire site, you should know that this tool is double-edged. Misconfiguring it can cut off your legitimate traffic too.
How do we distinguish useful bots from harmful ones?
The first mistake I see is blocking all bots at once. Search engine bots like Googlebot and Bingbot must be able to crawl your site freely. If you restrict them, your site's ranking in search results drops, and a few weeks later you'll notice organic traffic has been halved.
The way to tell them apart is by checking the User-Agent and verifying the IP. Google publishes its official list of IPs. A useful bot usually:
- Comes from announced and fixed IPs.
- Has predictable behavior and reads pages at reasonable intervals.
- Respects robots.txt.
In contrast, a harmful bot (a scraper, password-testing bots, or content-extraction tools) usually has a fake User-Agent, comes from varying IPs, and sends burst requests in short intervals. To see the real pattern, examine the log with this command:
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20This command shows the top 20 IPs by traffic. If a single IP has sent 5,000 requests in an hour, that's a harmful bot, not a user behind a browser.
Where should we apply rate limiting?
Where you apply rate limiting is as important as the limit itself. There are three layers, and each has its own use case.
Layer one: the web server (Nginx or Apache)
This is the layer closest to the traffic and gives the fastest response. In Nginx, this is done with the limit_req module:
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
server {
location /login/ {
limit_req zone=mylimit burst=5 nodelay;
limit_req_status 429;
}
}This setting says each IP can send a maximum of 10 requests per second, with 5 extra requests allowed as a burst. The status code 429 is returned instead of 403 so the client understands it's temporarily limited, not that its access is forbidden.
Layer two: firewall or CDN
If your traffic passes through a CDN, apply the limit there. The advantage is that harmful requests don't even reach your server and don't consume bandwidth. For volumetric attack scenarios, combining this layer with DDoS protection makes more sense than trying to manage everything on a single server alone.
Layer three: the application
Limiting at the code level (for example in PHP or Python) gives more flexibility. You can define limits based on the logged-in user, not just the IP. But the cost is that harmful requests reach the application layer and consume server resources. Keep this layer for protecting sensitive endpoints like login and registration, not for the entire site.
This is where they go wrong
The biggest mistake I've seen in practice is applying rate limiting to the entire site with a very low rate. For example, someone sets rate=2r/s on all paths. The result? A user who opens a page and whose browser sends 15 simultaneous requests for CSS, JS, and images immediately gets a 429 error. The site loads incompletely for them, images don't load, and they think the site is broken.
The sign is also clear: in the log, 429 errors are recorded for .css and .js requests, not for the main pages. If you see this pattern, remove the limit from static files and apply it only to dynamic endpoints.
The cost of over-blocking
Every limit you apply has a cost: it may drive out real users too. Users coming from shared networks (office, university, or behind a mobile carrier's NAT) all share a single IP. If you set the limit based on IP, one heavy user can victimize the rest.
The solution is to define the limit based on a combination of IP and path, not just IP. Be stricter for sensitive paths like /login and more generous for public pages. A simple table of decisions:
| Path | Suggested rate | Reason |
|---|---|---|
| /login | 5r/m | Prevent password testing |
| /api/search | 20r/s | Protect the database |
| Public pages | No limit | User experience |
| Static files | No limit | Normal browser loading |
A point that gets less attention: rate limiting is not a substitute for authentication. If your login endpoint is vulnerable to a dictionary attack, rate limiting only slows the attack down, it doesn't stop it. Alongside it, you should also have account lockout after several failed attempts and two-factor authentication.
How do we know the limit is working correctly?
After applying the settings, you should measure two things. First, the 429 error rate in the log. If this number suddenly rises, it means the limit is too strict. Second, the server response time (TTFB). If the limit is working correctly, TTFB should drop because harmful requests no longer reach the backend.
For testing, use tools like ab or wrk and simulate the site's behavior under load:
ab -n 1000 -c 50 https://example.com/login/If after this test the site becomes slow or errors out for regular users, your limit is too aggressive. To check infrastructure health at this very moment, check the real-time service status to make sure the problem is the limit, not the server.
A practical tip: always keep your rate limiting logs. If you later want to understand who attacked and when, these logs are your only source. I've explained more about the importance of keeping logs in What are audit logs and how do we keep them intact?
Frequently asked questions
What's the difference between rate limiting and concurrency limiting?
Rate limiting controls the number of requests per unit of time (for example, 10 requests per second), while concurrency limiting restricts the number of simultaneous open connections. For dealing with scraper bots, rate limiting is more effective. For preventing a single client from hogging resources, concurrency limiting works better. In practice, use both together.
Does rate limiting apply to Googlebot too?
Yes, if your rule is based on IP and you don't exempt Google's IPs, Googlebot will be limited too. This causes your site's ranking to drop. The right way is to whitelist Google's official IPs or use reverse DNS verification so that only the real Google bot is exempt from the limit.
Why do users complain about 429 errors after applying rate limiting?
Usually because you've set the rate for the entire site without accounting for static files. A browser sends dozens of requests to display a single page. Apply the limit to dynamic and sensitive paths and exempt CSS, JS, and image files from it.
Is it better to configure rate limiting in Nginx or in the CDN?
If your traffic passes through a CDN, applying the limit in the CDN is better because harmful requests don't reach your server and resources are saved. But if you need more precise control over application logic (for example, limiting based on the logged-in user), the Nginx layer or the application itself is a better option. The best result comes from combining both layers.
If you're facing a bot attack right now and don't know where to start, first check the log, then apply the limit only to sensitive paths, and measure the site's behavior after every change. For more complex scenarios or volumetric attacks, ServerNet's security services can provide defense layers without involving you in manual configuration.
Comments 0
No comments yet — be the first!