Why audit logs matter and how to keep them

What is an audit log and why do you need it?

An audit log is a record of important events on a server, database, or application that shows you who, when, from where, and what was done. These logs differ from regular system logs (like syslog) because they are purposefully designed to answer three questions: "Did something bad happen?", "Who is responsible?", and "How can I prevent it from happening again?".

Many server administrators only look at error logs and think their security is solid. But in a real attack, the attacker usually deletes the logs first to leave no trace. If your audit log is stored on the same server with root access, it's practically useless. In this article, you'll learn which events to record, where to store the logs, and how to ensure no one can alter them.

Which events should we record in the audit log?

Recording everything makes logs so voluminous that you can't find the important event. The golden rule is: record everything necessary to reconstruct an incident, nothing more. Below is a list of essential events.

Authentication and access events

• Successful and failed logins to the system (SSH, console, admin panel)
• Password changes by users or administrators
• Creation, deletion, or modification of user and group permissions
• Use of sudo or su to elevate privileges
• Database connections using high-risk accounts (such as database root)

Example: On Linux, use the following command to log all sudo commands to a separate file:

# /etc/sudoers.d/audit
Defaults logfile=/var/log/sudo-audit.log
Defaults log_input, log_output

With this configuration, every command executed with sudo is logged along with its output. This is highly effective in detecting abuse of administrative access.

Configuration changes and critical files

• Changes to files like /etc/passwd, /etc/shadow, and /etc/ssh/sshd_config
• Installation or removal of software packages
• Changes to firewall rules (iptables, nftables, ufw)
• Changes to cron jobs or systemd timers

To monitor changes to sensitive files, use the auditd tool:

auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config_change
auditctl -w /etc/passwd -p wa -k user_db_change

The -p wa parameter means recording write and attribute change events. The -k label also allows you to search events with ausearch -k sshd_config_change.

Network and service events

• Opening new ports on the server
• Unusual connections to sensitive ports (like 3306 for MySQL)
• Starting or stopping critical services (nginx, apache, mysql)
• Attempts to connect to suspicious IP addresses (such as Tor networks or sanctioned IPs)

To record new connections, you can use ss with cron, but a more professional approach is using auditd to monitor sockets:

auditctl -a always,exit -F arch=b64 -S bind -S connect -k network_connections

How to keep the audit log tamper-proof?

Recording events is only half the job. If an attacker can delete the logs, it's as if nothing happened. Below, we explain three layers of protection.

1. Storage outside the server (Centralized Logging)

Never store the audit log only on the same server where the events occur. An attacker who gains root can run rm -rf /var/log and delete everything. The standard solution is to forward logs to a centralized server.

The simplest method is using rsyslog. On the central server (e.g., with IP 192.168.1.10), add the following line to /etc/rsyslog.conf:

# On the central server
module(load="imtcp")
input(type="imtcp" port="514")
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

And on client servers:

# On the client server
*.* @@192.168.1.10:514

Important note: The central server should only accept SSH access from specific IPs and should also forward its own logs elsewhere to keep the security chain unbroken.

2. Digital signing and tamper prevention (WORM Storage)

Sending logs to another server doesn't stop an attacker; if they gain access to that server too, they can modify the logs. A stronger solution is using WORM (Write Once Read Many) storage. In this method, the log file is written only once, and no one (even root) can modify it.

On Linux, you can use the chattr +a attribute, which only allows appending:

chattr +a /var/log/audit/audit.log

But this method isn't perfect either; an attacker can delete the file and recreate it. For full protection, you should place the file on a separate partition with the mount -o remount,ro option or use tools like auditd with the max_log_file_action = keep_logs feature.

A more professional approach is sending logs to a cloud service with Object Lock capability. Services like S3 Object Lock or Azure Immutable Blob Storage allow you to set a lock duration (e.g., 1 year). During this period, no one can delete or modify the logs. If you're using cloud infrastructure, definitely explore this option.

3. Log rotation and long-term retention

The audit log should be retained for at least as long as your security review period. Common standards like PCI-DSS require 1 year of retention and 3 months of online access. Use logrotate for automatic log rotation:

# /etc/logrotate.d/audit
/var/log/audit/audit.log {
    weekly
    rotate 52
    compress
    delaycompress
    notifempty
    missingok
}

This configuration rotates logs weekly and keeps 52 copies (one year). You can also transfer the compressed files to the central server.

Common mistakes in audit log management

Below, we review several common mistakes that usually compromise log security.

Mistake 1: Storing logs on the same system partition

If /var/log is on the root partition, disk filling can bring down the entire system. An attacker can also disrupt services by intentionally filling the disk. Solution: allocate a separate partition for /var/log and monitor its capacity.

Mistake 2: Not recording failed attempts

Many administrators only record successful events. But failed attempts (like logging in with a wrong password) are early indicators of brute-force attacks. In auditd, make sure USER_LOGIN events with a failed result are also recorded:

auditctl -a always,exit -F arch=b64 -S execve -F success!=1 -k failed_commands

Mistake 3: Ignoring timestamps

If the server clock isn't accurate, reconstructing the sequence of events during an attack becomes impossible. Make sure NTP is enabled and the clock is synchronized with a reliable source:

timedatectl set-ntp true
timedatectl status

How to use the audit log in incident response?

When an incident occurs, the first step is to copy the logs before taking any action. If you reboot the server or stop services, evidence may be lost. Recommended steps:

1. Create a disk image or copy log files to separate media
2. Search for events related to the attacker's account: ausearch -ua username
3. Review network connections at the time of the incident: ausearch -k network_connections -ts recent
4. Compare central server logs with local logs to detect tampering

Important note: If local and central logs differ, it means the attacker has modified the local log. This itself is a sign of an attack.

Supporting tools for audit log management

For high log volumes, consider the following tools:

• auditd: The standard Linux tool for security auditing
• rsyslog or syslog-ng: For centralized log forwarding
• SIEM (like Wazuh or Elastic Stack): For automated analysis and alerting
• logrotate: For managing file size and rotation

If your infrastructure is on cloud servers, you can use cloud log management services. ServerNet also provides the ability to forward logs to external destinations on its hosting platforms, which is suitable for implementing a centralized architecture.

Summary

An audit log isn't just a text file; it's a critical defensive tool. To be truly effective, it must have three characteristics: comprehensive (records important events), centralized (stored outside the main server), and immutable (no one can tamper with it). By implementing the methods described in this article, you can be confident that in the event of an attack, you'll have sufficient evidence to identify the attacker and prevent a recurrence.

Start today: first, identify the list of critical events on your server, then set up log forwarding to a central server, and finally configure a rotation and long-term retention plan. These three simple steps will significantly enhance your security.