Server monitoring and effective alerting

Why Server Monitoring Needs a Strategy, Not Just Tools

Most technical teams, when they first set up server monitoring, monitor everything: CPU, RAM, disk, network, services. The result? They receive 40 alerts on day one, 20 on day three, and by the end of the week, nobody is looking at the alerts anymore. This phenomenon is called "Alert Fatigue," and it's exactly what causes a real outage to go unnoticed for hours.

Effective server monitoring means knowing what to monitor, what threshold makes sense, and when someone should actually be paged. This article is a practical roadmap: from selecting key metrics to setting up alerts that are worth responding to.

What to Monitor? The Critical Metrics That Really Matter

Not all metrics are created equal. Monitoring 200 metrics only creates noise. Focusing on 5–7 key metrics that are directly tied to user experience and service health yields much better results.

Basic Infrastructure Metrics

• CPU: Measure the load average relative to the number of cores. If the load average on a 4-core server reaches 4, the CPU is completely saturated. But high CPU isn't always bad; if your service does video processing or heavy computation, 80% usage is normal.
• RAM: Usage percentage matters, but more important is swap usage. If the system starts swapping, it means RAM is truly scarce and performance drops significantly. Set the swap alert threshold at 10%, not 50%.
• Disk: Disk filling up is one of the most common causes of service outages. But instead of alerting at 80%, think about the growth rate. If the disk is 70% full and grows 2% daily, you have a problem in 15 days. Alerts should be based on prediction, not just the current state.
• Network: Monitor bandwidth usage and network interface errors (RX/TX errors). Network errors are often a sign of hardware or cable failure and appear before a complete outage.

Service-Level Metrics

Infrastructure is just a means to an end. What really matters is the health of your service. Be sure to add these metrics:

• Uptime: An external check from another location that sends a request to your site or API every 60 seconds. This is the only way to know if your service is truly up from the user's perspective.
• Latency: The average response time over the last 5 minutes. If it goes from 500 milliseconds to 2 seconds, something is broken, even if CPU and RAM look healthy.
• Application Errors: The number of 5xx errors in the web server or application error logs. This metric is usually the first sign of a bug or memory leak.
• Message Queues: If you use Redis or RabbitMQ, monitor the queue length. A continuously growing queue means the consumer has stopped working.

Setting Thresholds: How to Find the Right Number

Don't set thresholds based on guesswork. The right approach has three steps:

1. Collect a baseline period: Gather data for at least 2 weeks without any alerts. See how much each metric fluctuates under normal conditions.
2. Set the threshold at 2 to 3 times the standard deviation: If CPU normally fluctuates between 20% and 40%, set the alert threshold at 70–80%, not 50%. Early alerts only create noise.
3. Adjust thresholds over time: After each real incident, check whether the threshold alerted too early or too late. Thresholds should be living things, not set once and forgotten.

Practical Example: Setting Up a Disk Alert

Suppose you have a database server with a 500 GB disk. Here's the recommended approach:

# Staged Alerts
Warning:  disk usage > 75%  (notify via Telegram, no page)
Critical: disk usage > 85%  (SMS + email to admin)
Emergency: disk usage > 92% (automated phone call, only for on-call team)

# Growth-rate-based alert (for prediction)
If daily growth rate > 1.5% and remaining space < 20GB → immediate alert

Key point: Staged alerts let the team know how urgent each alert is. If all alerts are the same, the team quickly becomes desensitized.

Smart Alerting: Less, But More Effective

The goal of alerting isn't to report every minor issue. The goal is to report issues that require human action. Here are a few key techniques for that:

1. Alert Aggregation and Compression

If 5 services on one server go down, you shouldn't send 5 separate alerts. Send one alert that says "Server X is unreachable and 5 services are affected." Tools like Alertmanager in Prometheus do this automatically.

2. State-Based Alerts, Not Event-Based

State-based alerting means only alerting when the state changes, not every time a check runs. Example:

# Bad: alerts every 5 minutes until the problem is resolved
if cpu > 90% then alert

# Good: only alert when the state changes from OK to CRITICAL
state = OK
if cpu > 90% for 10 minutes then
    if state != CRITICAL then
        alert("CPU became critical")
        state = CRITICAL
    end
end

This alone reduces alert volume by 90%.

3. Consider the Duration

A 5-second CPU spike is usually harmless. A 15-minute spike is a real problem. Set a duration for each alert:

• CPU > 90%: sustained for at least 10 minutes
• Disk > 85%: sustained for at least 30 minutes (since disk growth is gradual)
• Service Down: immediately (0 minutes duration)

4. Layer Your Notification Channels

Not all alerts should go to all channels. A suggested structure:

1. Info alerts: Only to the team's Telegram or Slack channel. No immediate action needed.
2. Warning alerts: Email + message to the on-call admin. Should be reviewed within 1 hour.
3. Critical alerts: SMS + automated phone call. Action required within 15 minutes.

Recommended Tools and a Practical Configuration Example

There are many open-source tools for server monitoring. The combination of Prometheus + Alertmanager + Grafana is the industry standard and works for both small and large teams. If you have a simpler infrastructure, Netdata or Zabbix are lighter options.

Here's an example alerting rule in Prometheus that implements the concepts above:

groups:
  - name: server-alerts
    rules:
      - alert: HighCPULoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU above 90% on {{ $labels.instance }}"

      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "Root disk will fill up within 24 hours"

Notice that the second rule uses the predict_linear function, which predicts the growth rate — this is the "prediction-based alerting" we discussed in the thresholds section.

How to Eliminate Alert Fatigue

If your team is ignoring alerts, the problem isn't the team — it's the alerting system. Three golden principles:

• Every alert must have a specific action: If you send a "High CPU" alert, it must be clear who should do what, and by when. If there's no specific action, remove that alert.
• Every alert must be testable: At least once a month, simulate a real failure scenario (e.g., stopping a service) and verify that the alert fires correctly and reaches the right person.
• Periodic alert review: Hold a 30-minute meeting every month to review all alerts from the past month. Remove or fix any alert that didn't lead to action.

Summary: Server Monitoring Means Peace of Mind, Not Anxiety

Good server monitoring is a system that stays quiet when everything is normal and, when a real problem occurs, notifies exactly the right person, at the right time, with enough information. By monitoring the right metrics, setting data-driven thresholds, and designing staged alerts, you can prevent costly outages and save your team from alert fatigue.

If you're looking for an infrastructure where you can easily run these tools, ServerNet provides a suitable cloud platform for setting up server monitoring. But more important than any tool is the process we've described in this article — implement it and see the results.