A service that comes up with systemctl start but silently dies a few minutes later almost always has its problem in the unit file, not in the application itself. The telltale sign is this: systemctl status shows inactive (dead) and journalctl -u myservice has no clear error. That means the process exited with a zero exit code or a signal, and systemd decided—according to a policy you never set—not to bring it back up. This article takes that unit file apart, section by section.
Where the unit file lives and which version wins
systemd reads three paths in order of priority: /etc/systemd/system/ has the highest priority, then /run/systemd/system/, and finally /usr/lib/systemd/system/. If a file with the same name exists in two paths, the one with higher priority wins and the lower version is completely ignored. This means you can leave the vendor file untouched and write your own version in /etc; a package update will no longer wipe your changes.
After every change, two commands are required and one is not enough:
systemctl daemon-reload
systemctl restart myserviceIf you don't run daemon-reload, systemd keeps the old version in memory and you think your change had no effect. This is where people go wrong: they edit the file, run restart, nothing happens, and then they spend hours hunting for a bug in the application code. The sign is that systemctl cat myservice shows something different from the file you just saved.
The three sections of a unit file and the keys that actually get used
Every unit file is made up of the [Unit], [Service], and [Install] sections. A fourth section, [Socket], exists for socket-activated services, which is a separate discussion.
The [Unit] section: identity and ordering
Description=is the text shown insystemctl status. Keep it short and free of non-ASCII characters.After=network-online.targetmeans this service should run after the network is ready. Note thatAfteris only ordering, not a requirement.Requires=means if that unit doesn't come up, this one won't run either.Wants=is the weaker version: it tries, but its failure doesn't block execution.BindsTo=is stricter thanRequires; if the other unit stops, this service stops too.
Take the difference between After and Requires seriously. Many people write only After=mysql.service and expect their service not to run when MySQL is down. It won't work. After only says "if both were queued, that one goes first." For a real dependency you also need Requires or Wants.
The [Service] section: the heart of the matter
The most important key in this section is Type=, and choosing it wrong is the source of most confusion:
| Type | When to use it | Sign it's wrong |
|---|---|---|
simple | Default; the process stays in the foreground | Service goes active immediately but doesn't work |
forking | The program daemonizes itself | systemd loses the main process and the service goes dead |
notify | The program signals readiness via sd_notify | Service stays activating forever |
oneshot | A one-time task, like a setup script | Service goes dead after finishing, which is normal |
The execution keys you'll almost always need:
[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=65535
Environment=NODE_ENV=production
EnvironmentFile=-/etc/myapp/envTwo notes on this block. First, EnvironmentFile with a minus sign means don't error out if the file is missing; without it, a missing file causes the service to fail entirely. Second, take LimitNOFILE seriously: the systemd default is usually 1024, and network applications under real load die with a Too many open files error. If server load has spiked and you don't know why, this is one of the places to check; the guide on diagnosing the cause of high server load covers the investigation path step by step.
The [Install] section: when to enable it
WantedBy=multi-user.target means this service should be enabled on a normal boot. Without this line, systemctl enable works but warns, and the service won't come up at boot. For timers you write WantedBy=timers.target.
Restart policy: Restart= and what actually happens
The commonly used values of Restart= are: no (default), on-failure, always, on-abnormal, and on-abort. The difference between on-failure and always comes down to one thing: if the process exits with a zero exit code, on-failure considers it a success and won't run it again, but always will. For a service that must always stay alive, always is the right choice.
But there's a trap here that few people mention: Restart=always without limits throws a broken service into an infinite loop. systemd prevents this by default with StartLimitBurst and StartLimitIntervalSec; the common default is 5 starts in 10 seconds. If it exceeds this, the service enters the failed state and stops trying. The exact message in the journal is:
start request repeated too quickly for myservice.serviceWhen you see this, systemctl reset-failed myservice clears the state, but it doesn't fix the underlying problem. First find the cause of the exit, then tune the policy. RestartSec=5 isn't trivial either; too small a value multiplies the pressure on resources during a failure.
Dependencies between services: ordering, requirements, and stopping
There are three independent axes, and the common mistake is conflating them:
- Start ordering:
After=andBefore=. This only determines the turn. - Requirement:
Requires=,Wants=,BindsTo=. This determines whether it runs at all. - Stop propagation:
PartOf=andPropagatesReloadTo=. This determines whether, when the parent service restarts, this one restarts too.
The correct pattern for a service that needs a database combines all three:
[Unit]
Description=My App
After=network-online.target mysql.service
Wants=network-online.target
Requires=mysql.service
PartOf=mysql.service
[Service]
Type=simple
ExecStart=/opt/myapp/bin/server
Restart=always
RestartSec=5With PartOf=mysql.service, every time MySQL restarts, the application restarts automatically too and no dead connections are left behind. Many people leave this line out, and after every database restart the application throws errors on stale connections until someone brings it up manually.
A practical warning: Requires is not bidirectional. If you manually stop MySQL, your service stops too, but if your service crashes, MySQL stays untouched. If you want bidirectional behavior, look at BindsTo, at the cost that a transient error in the other service brings down the whole chain.
Mistakes seen often in practice
ExecStart with a relative path. systemd wants an absolute path; ExecStart=./server is rejected with Exec format error or No such file or directory, even if the file exists. Always write the full path.
A missing User=. Without it, the service runs as root. This isn't just a security issue; files the service creates are owned by root and the application user can't read them later. If you're running several services side by side on a dedicated server or cloud server, this is one of the most common causes of strange permission errors.
Ignoring ExecReload. If you don't define it, systemctl reload is effectively a full restart and active connections get dropped. For services like Nginx or HAProxy that reload without downtime, this difference matters.
And a note on containerized services: if your application runs inside Docker, you usually don't need a manual unit file and Docker's own Restart=always is enough. The guide on setting up Docker on a VPS explains the difference between these two layers; combining both without reason only makes troubleshooting harder.
Frequently asked questions
Why don't changes take effect after editing the unit file?
Because systemd caches the file version in memory and keeps running the old version until you run systemctl daemon-reload. After the reload, restart the service too; reload alone won't bring the running process up with the new settings.
What's the difference between After and Requires in a unit file?
After only determines start ordering, and if the other unit doesn't run, your service still comes up. Requires creates a real dependency; if that unit fails, your service won't run either. For a full dependency you need both together.
Why does a service with Restart=always enter the failed state?
Because systemd prevents the restart loop. If the service starts more than StartLimitBurst times within the StartLimitIntervalSec window, it stops and the message start request repeated too quickly appears in the journal. systemctl reset-failed clears the state, but the cause of the exit must be fixed separately.
Where should I create the unit file so a package update doesn't wipe it?
In /etc/systemd/system/. Files in /usr/lib/systemd/system/ belong to installed packages and may be overwritten on update. If you want to change only part of an existing unit, instead of copying the whole thing, use systemctl edit myservice so a drop-in is created in /etc/systemd/system/myservice.service.d/.
The next step is clear: run systemctl cat on the service that's giving you trouble right now, compare the [Service] section with what's covered here, and fix Type= and Restart= first. Most services that "die on their own" simply have these two keys wrong.