Server Monitoring: Metrics, Alert Thresholds, and External Monitoring

It's two in the morning and the site won't come up. You SSH in and get no response. The last thing you remember is that everything was normal yesterday and you received no alerts. This is exactly where it becomes clear that you either had no server monitoring, or you had it but it wasn't alerting on what it should. This article is about setting up monitoring from scratch: what to measure, how to set thresholds, and why monitoring that only runs on the server itself is half the job.

Before any tool, measure these four metrics

Monitoring tools give you dozens of metrics, and that's exactly why people turn them all on and then look at none of them. To start, four metrics are enough. If these four are healthy, the rest usually follow.

• Load average relative to core count. The uptime command gives three numbers: the 1, 5, and 15 minute averages. The raw number is meaningless; it must be divided by the number of cores. On a 4-core server, a load of 4 means full saturation. A load of 8 means the wait queue is twice the processing capacity. If load rises but CPU is idle, the problem is disk I/O, not the processor; the guide to diagnosing the cause of high server load walks through this exact scenario step by step.
• Memory and swap usage. Look with free -m, not at the free column but at available. The available number is what you can actually give to new processes. If swap is being used continuously, the server is alive but slow; this state is worse than a full crash because no one notices.
• Disk space and inodes. df -h shows block space and df -i shows inode count. Take the latter seriously. A server that has 20% space left but has run out of inodes can't create any file and gives a No space left on device error while df shows free space. It's usually the fault of PHP sessions or countless small cache files.
• Service response time, not just whether it's alive. It's not enough that nginx responds with a 200 code. If the response takes 8 seconds, the site is down from the user's perspective.

How to set alert thresholds so they don't wake you up at night

The biggest mistake in server monitoring is choosing thresholds based on guesswork. Someone says "alert me if CPU goes above 80%" and then gets an email every day at 3 AM because the nightly backup runs at that hour. Two weeks later, they filter that email and never see any alert again. This is exactly what happens when you have monitoring and still get caught off guard.

The right approach is: first, collect data for a week without any alerts. Then look at what the normal pattern is. Set the threshold at something that is never touched under normal conditions, not something that is touched every day.

The "Duration" column is more important than the threshold column. A momentary CPU spike to 95% for two seconds is not bad news; it's what a server is supposed to do. The alert should be on persistence, not on peaks. In Prometheus you write this with for: 10m in the rule block:

- alert: HighLoad
  expr: node_load1 > (count(node_cpu_seconds_total{mode="idle"}) * 1.5)
  for: 10m
  labels:
    severity: warning

The for: 10m line is what prevents false alerts. Without it, every time a heavy cron runs, your phone rings.

Internal monitoring isn't enough; look from the outside too

This is where many people go wrong. They install Netdata or node_exporter on the server, build a nice dashboard, and feel at ease. But when the server itself goes down, that same tool goes down too. If the server reboots, if the network card dies, if the datacenter has a power issue, if a firewall is mistakenly closed, you get no alert because the alerter was on the very machine that died.

External monitoring means an independent point, outside your server, sends an HTTP request or a ping every 60 seconds and notifies you if there's no response. This service can be a cheap VPS elsewhere or one of the uptime monitoring services. The important point is that the alert path be separate from the failure path.

Let me also mention the cost: external monitoring can't tell you why the server died. It only says it died. To diagnose the cause, you still need internal metrics and logs. So these two don't replace each other; they complement each other. If you had to set up only one, choose external monitoring, because a late alert is better than complete silence.

What not to monitor on purpose

Every metric you add has a storage cost and an attention cost. Deliberately set aside a few things:

• Process-level metrics for services that restart themselves. If systemd brings the service back up within 5 seconds, alerting on it is just noise.
• 404 error rate. This number is always high on public sites and is almost never a sign of failure.
• CPU temperature on virtual servers. There is no sensor and the number you see is fake.

Practical setup in half an hour

If you want to start today, let's go through this order:

1. On the server, install node_exporter and enable it with systemd. Its default port is 9100 and it should only be reachable from the monitoring server's IP, not from the internet. See the list of ports that shouldn't be open in the reference for common service ports.
2. Bring up a Prometheus on a separate machine and add the target in prometheus.yml. If you don't have a separate machine, a small cloud server is enough for this job.
3. Write the four alerts from the table above. Connect Alertmanager to a real channel; an email that doesn't give a notification on your phone is not an alert.
4. Set up an external HTTP check on your site's homepage, with a 10-second timeout.
5. Let it run for a week and then tune the thresholds based on real data.

If you've just received your server and haven't chosen the distro and services yet, do choosing a Linux server distro and SSH configuration before monitoring; changing them later is a hassle. For physical servers where you want full hardware control, a dedicated server is the more common choice, but for most sites a managed Linux hosting is enough and takes a lot of this monitoring off your shoulders.

This is where they go wrong

The most common mistake I've seen is this: someone configures Alertmanager, but the alert sending path goes through the same server. That is, the alert is sent via local SMTP. The server's disk fills up, postfix can't write the queue, and no email goes out. The sign is also this: you see in the dashboard that the alert triggered, but there's no news in your inbox. Always use an external relay for sending alerts.

The second mistake is alerting on "the service is up" instead of "the service works correctly." nginx with a 502 code is also up. The health check should call a real endpoint that also touches the database, not just see that port 80 is open.

Frequently asked questions

Which tool is more suitable for monitoring a Linux server?

For a quick start, Netdata has the least effort and gives a live dashboard in a few minutes. If you want precise alerts and long-term history, the combination of Prometheus, node_exporter, and Alertmanager is a better choice, at the cost of having to write a few YAML files. For a single server, Netdata is enough; from three servers up, Prometheus is worth the time.

What should I set the CPU alert threshold to?

There is no fixed number. Set the threshold relative to your core count and your own load pattern, not based on a general number. Collect data for a week, see where the normal peaks are, and set the threshold above that. More important than the number is the persistence condition: the alert should activate after a few minutes of persistence, not on a momentary spike.

Why did I find out about the failure late even though I have monitoring?

It's almost always one of two reasons: either the alert was set on the wrong metric, or the alert sending path went through the same server that broke. The third reason isn't rare either: there were so many false alerts that the user filtered them. If you're ignoring your alerts, the problem isn't monitoring, it's threshold tuning.

Do I need both external and internal monitoring at the same time?

Yes, and the reason is simple: internal monitoring says what's happening, external monitoring says what the user sees. The former goes silent when the server dies; the latter reports at exactly that moment when the server dies. If you have only one choice, take external and add internal later.

Set up an external check on your site's homepage today. One HTTP request every 60 seconds, with a 10-second timeout and an alert that reaches your phone. This half hour of work is worth more than any beautiful dashboard.