Server Time Zone: Why the Wrong Clock Breaks Everything

Your JWT token is rejected with the error Token used before issued. Or a freshly installed SSL certificate shows as "expired" in the browser, while its expiration date is months away. Or in Kibana you see two consecutive logs where the second log has an older timestamp than the first. These three symptoms share one root cause: your system clock is not synchronized with the outside world. The problem usually starts with the server time zone, not the hardware clock.

The first thing you should do is figure out where you currently stand. One command, three lines of output:

timedatectl
               Local time: Sat 2025-01-11 14:22:07 +0330
           Universal time: Sat 2025-01-11 10:52:07 UTC
                 RTC time: Sat 2025-01-11 10:52:07
                Time zone: Asia/Tehran (+0330, +0330)
System clock synchronized: no
              NTP service: inactive

The last two lines say it all. If System clock synchronized: no, your clock is drifting freely. On a typical physical server, a clock without synchronization drifts by a few hundred milliseconds to a few seconds per day, forward or backward. On a virtual machine this number is worse, because the guest gets its time from the hypervisor, and if the hypervisor is busy, the drift can reach tens of seconds over a few days.

Why an incorrect clock invalidates API signatures

Most signed APIs (AWS Signature V4, HMAC signatures in payment gateways, OAuth with exp and iat) have a time tolerance window. In AWS this window is 15 minutes. If your server clock is 16 minutes behind, every request is rejected with RequestTimeTooSkewed. If it is 16 minutes ahead, the same thing happens but with a different message.

With JWT the story is worse. The issuing server signs the token with iat (issued-at time) and the consuming server compares it against its own clock. If the consumer's clock is behind, the token "came from the future" and is rejected. This error usually appears in logs as Token used before issued or NotBefore claim is in the future, and many people go looking at the JWT library, while the problem is in chronyd.

SSL certificate and clock: the real error you see

Certificate validation depends entirely on the clock. If the server clock is behind the certificate's notBefore, the client gets certificate is not yet valid. If it is ahead of notAfter, it gets certificate has expired. Both errors look the same in the browser, and the user thinks the certificate is broken.

Here is where people make a mistake: the admin reissues the certificate, reinstalls it, and still sees the error. Because the certificate was never the problem. A quick way to diagnose:

openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates
date -u

If your server's UTC date is outside the range from notBefore to notAfter, no certificate will work.

Set the server time zone correctly, not just the clock

Don't confuse two concepts. The system clock (UTC) must be synchronized with NTP. The time zone is only a display and has no effect on internal calculations. That is, when you run date and see +0330, the internal clock is still UTC, and that is correct.

To set the time zone:

timedatectl set-timezone Asia/Tehran
timedatectl set-ntp true
timedatectl status

On older servers that don't have systemd, the equivalent is:

ln -sf /usr/share/zoneinfo/Asia/Tehran /etc/localtime
echo "Asia/Tehran" > /etc/timezone
hwclock --systohc

The last line is important and many people skip it. hwclock --systohc synchronizes the hardware clock (RTC) with the system clock. If you don't do this, after a server restart the clock reverts to its previous value and you think the settings were wiped.

chrony or systemd-timesyncd?

On most modern distributions, systemd-timesyncd is enabled by default. For a typical server it is enough. But if your server is virtual, has a heavy load, or the clock jumps frequently, choose chronyd. The practical difference: chronyd can gently "skew" the clock (slew) instead of jumping suddenly, and it performs better on high-latency networks.

If the choice were mine: on any server that works with APIs or has centralized logging, I would install chronyd. The cost is that there is one more service to manage and you have to understand the /etc/chrony/chrony.conf config. On a simple web server that only runs PHP, I would not touch systemd-timesyncd.

Logs without a correct clock are worthless

When you have two servers and one's clock is 40 seconds behind, the order of events in your centralized logs lies. A request that started on server A and finished on server B is displayed in reverse in Kibana or Loki. Debugging a distributed bug with unsynchronized logs is nearly impossible.

To understand where each service writes its logs and how you can compare timestamps, Linux log locations: where does each service write? is a good reference. If you manage services with systemd, systemd unit file: sections, restart, and dependencies shows how to configure startup order and dependency on the time service.

A common mistake in containers

A container doesn't have its own clock; it gets the clock from the host kernel. So if the host clock is wrong, all containers are wrong. But there is a trap here: some images have their own /etc/localtime inside them and don't mount it from the host. The result is that date inside the container shows UTC and outside shows +0330. This contradiction makes the application log and the host log not line up.

The solution is simple, but it must be done deliberately:

docker run -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro ...

Or in docker-compose.yml:

environment:
  - TZ=Asia/Tehran

The TZ variable only affects programs that read it. It has no effect on the system clock itself. Many people don't know this difference and think setting TZ solves the problem.

How to tell whether the clock is really synchronized

Three commands, three levels of confidence:

1. timedatectl — overall status and whether NTP is enabled.
2. chronyc tracking — the current offset relative to the reference server, in milliseconds.
3. chronyc sources -v — the list of NTP servers and the connection status to each.

The output of chronyc tracking looks something like this:

Reference ID    : A29FC87B (time.cloudflare.com)
System time     : 0.000034 seconds fast of NTP time
Last offset     : +0.000021 seconds
RMS offset      : 0.000089 seconds

If System time is more than a few hundred milliseconds, it means synchronization is not yet complete or your NTP server is unreachable. A value below 50 milliseconds is enough for most use cases. For financial transactions or distributed systems, aim for below 10 milliseconds.

A point we run into a lot in Iran: access to public NTP servers like pool.ntp.org is sometimes filtered or slow from some data centers. If chronyc sources shows that no server is reachable, use an internal NTP server or the ntp.irancell.ir servers. The NTP port is 123/udp, and if your firewall has it closed, no synchronization will happen. For a quick review of ports and which ones should be open, see Reference of common service ports and ports that should not be open.

Apply the settings on your own servers

If you work on a dedicated server, you have full access to time and firewall settings and can even set up an internal NTP server. On a dedicated server this is easier because you have full control over the kernel and network. On a cloud server the same settings apply, but remember that some hypervisors pull the guest clock from the host, and you must disable or configure synchronization inside the guest so that two time sources don't fight each other.

If you have just set up a server and want to configure it correctly from the ground up, Choosing a server Linux distribution: a practical, no-nonsense guide helps you understand which distribution has which time tool by default. And if you work on a managed host and don't have root access, Linux hosting from ServerNet is an option where the basic time settings are already done and you just need to make sure your application reads the correct time zone.

One last point I see a lot in practice: after every clock change, restart the services that depend on time. Databases, queues, and some applications cache the time in memory and keep the old value until restarted. A simple systemctl restart prevents hours of pointless debugging.

Frequently asked questions

Why does the server clock revert after a restart?

Because the hardware clock (RTC) has not been synchronized with the system clock. After every change, run the hwclock --systohc command so the value is stored in the RTC. If you don't do this, the server reads the old value from the hardware on the next boot.

Is setting TZ=Asia/Tehran enough for a container?

No. The TZ variable only affects programs that read it and has no effect on the system clock. The container clock comes from the host kernel. If you want the /etc/localtime and /etc/timezone files to be correct as well, you must mount them into the container as read-only.

How do I find out how far the server clock is from the real time?

The chronyc tracking command shows the current offset in seconds. If chronyd is not installed, you can use ntpdate -q pool.ntp.org, which only reports the difference and does not change the clock. A value below 50 milliseconds is acceptable for most use cases.

Should I set the server time zone to UTC or Asia/Tehran?

For a server, I recommend UTC. Logs, databases, and communication between services all work more simply and unambiguously with UTC. Apply the local time zone at the display layer (frontend or reporting). If you are forced to use local, at least make sure all servers in a cluster have the same time zone.