uRadical blog - What a Dev Needs to Know of Systemd

If you've deployed an application to a Linux server in the past decade, you've almost certainly encountered systemd. Yet for many developers, it remains a mysterious black box — something the ops team handles. This post aims to change that by giving you a practical understanding of systemd: what it is, why it exists, what the fuss was about, and most importantly, how to use it effectively.

A Brief History: From SysVinit to systemd

To understand systemd, you need to understand what came before it.

The SysVinit Era (1980s–2010)

Linux inherited its initialization system from Unix System V (hence "SysVinit"). This venerable system used shell scripts located in /etc/init.d/ to start and stop services. It was simple and familiar, but it had significant limitations:

Sequential startup: Services started one after another, making boot times painfully slow on modern hardware
No dependency management: If service A needed service B, you had to manually ensure B started first
No process supervision: If a daemon crashed, it stayed dead unless you had something else watching it
No standardisation: Every distribution had slightly different conventions for init scripts

The Upstart Interlude (2006–2015)

Canonical developed Upstart for Ubuntu as an event-driven replacement for SysVinit. It introduced asynchronous service startup and could automatically restart crashed services. However, it was largely seen as a transitional solution rather than a fundamental rethink of how init should work.

Enter systemd (2010–present)

In 2010, Lennart Poettering and Kay Sievers at Red Hat began working on systemd. Their original design document outlined an ambitious vision: a modern init system that could leverage Linux-specific features to dramatically improve boot times and service management.

Fedora adopted systemd in 2011. Arch Linux followed in 2012. After a contentious vote in 2014, Debian adopted it in 2015, effectively making it the standard across most major distributions. Today, systemd is the default on RHEL, CentOS, Ubuntu, Debian, Fedora, openSUSE, Arch, and virtually every mainstream Linux distribution.

The Controversy (and Its Resolution)

No discussion of systemd would be complete without addressing the controversy it sparked. For several years, it was one of the most divisive topics in the Linux community.

The Criticisms

Violation of the Unix Philosophy: Critics argued that systemd violated the Unix principle of "do one thing and do it well." Unlike the simple shell scripts of SysVinit, systemd grew to encompass logging (journald), network management (networkd), DNS resolution (resolved), and much more.

Binary Logging: The journal uses a binary format rather than plain text files. To traditionalists accustomed to grep-ing through /var/log/, this felt like a betrayal of Unix principles.

Scope Creep: What started as an init system absorbed udev, consoled, and dozens of other components. Critics saw this as an attempt to become an indispensable layer between the kernel and userspace.

Tight Integration: Because systemd provides so many integrated services, many applications and desktop environments (notably GNOME) began depending on it, making it difficult to avoid even for those who wanted to.

The Resolution

The controversy has largely subsided, though not because everyone came to love systemd. Rather, several factors contributed to acceptance:

1. Practical benefits: Faster boot times, better dependency handling, and unified service management across distributions were tangible improvements

2. Maturation: Early bugs and rough edges were smoothed over through years of development

3. Documentation: The systemd documentation improved dramatically, making it easier to learn

4. Inevitability: With major distributions committed, fighting systemd became impractical for most users

As one Debian developer famously put it: "We didn't choose systemd because we loved it; we chose it because it worked better."

For developers today, systemd is simply a fact of life on Linux. Understanding it is no longer optional.

Why systemd Over the Alternatives?

Here's what systemd brings to the table compared to SysVinit:

Feature	SysVinit	systemd
Service startup	Sequential	Parallel
Dependency handling	Manual via scripts	Automatic, declarative
Process supervision	None built-in	Automatic monitoring and restart
Resource control	None	cgroups integration
Logging	Separate syslog	Integrated journal
Service file format	Shell scripts	Declarative INI-style
Boot time	Slower	Significantly faster
Socket activation	No	Yes

The parallel startup alone can reduce boot times by 50% or more on systems with many services.

Core Concepts and Terminology

Before diving into commands, let's establish the vocabulary.

Units

A unit is anything that systemd manages. Units are defined by configuration files called unit files. There are several types:

.service – Processes (daemons, applications)
.socket – IPC or network sockets for socket activation
.timer – Scheduled tasks (replaces cron)
.mount – Filesystem mount points
.target – Groups of units (similar to runlevels)
.path – Path-based activation (file/directory monitoring)
.device – Kernel device exposure
.slice – Resource management hierarchy

Targets

Targets replace the concept of runlevels. Common targets include:

multi-user.target – Multi-user system (equivalent to runlevel 3)
graphical.target – Multi-user with GUI (equivalent to runlevel 5)
rescue.target – Single-user rescue mode
network-online.target – Network is fully configured

Unit File Locations

Unit files live in several directories, in order of priority:

1. /etc/systemd/system/ — Local configuration (highest priority)

2. /run/systemd/system/ — Runtime units

3. /usr/lib/systemd/system/ — Distribution-provided units

User-specific units go in ~/.config/systemd/user/.

Essential systemctl Commands

systemctl is your primary interface to systemd. Here are the commands you'll use most often:

Service Management

# Start a service
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service
sudo systemctl restart nginx

# Reload configuration without restart (if supported)
sudo systemctl reload nginx

# Check service status
systemctl status nginx

# Enable service to start at boot
sudo systemctl enable nginx

# Disable service from starting at boot
sudo systemctl disable nginx

# Enable AND start immediately
sudo systemctl enable --now nginx

# Check if service is enabled
systemctl is-enabled nginx

# Check if service is active (running)
systemctl is-active nginx

Viewing Information

# List all running services
systemctl list-units --type=service

# List all services (including inactive)
systemctl list-units --type=service --all

# List enabled/disabled status of all services
systemctl list-unit-files --type=service

# Show dependencies of a unit
systemctl list-dependencies nginx

# Show what depends on a unit
systemctl list-dependencies --reverse nginx

# Show full unit file contents
systemctl cat nginx

# Show all properties of a unit
systemctl show nginx

System State

# View current default target
systemctl get-default

# Set default target
sudo systemctl set-default multi-user.target

# Reboot
sudo systemctl reboot

# Power off
sudo systemctl poweroff

# Analyze boot time
systemd-analyze

# Show boot time per service
systemd-analyze blame

# Show critical chain (what delayed boot)
systemd-analyze critical-chain

Working with the Journal (journalctl)

systemd includes its own logging system called the journal. Access it with journalctl:

# View all logs
journalctl

# View logs for a specific service
journalctl -u nginx

# Follow logs in real-time (like tail -f)
journalctl -f

# Follow logs for a specific service
journalctl -f -u nginx

# Logs since last boot
journalctl -b

# Logs from previous boot
journalctl -b -1

# Logs since a specific time
journalctl --since "2024-01-15 10:00:00"

# Logs in the last hour
journalctl --since "1 hour ago"

# Only error messages
journalctl -p err

# Kernel messages only
journalctl -k

# Show output in JSON
journalctl -o json-pretty

# Check disk usage
journalctl --disk-usage

# Vacuum old logs (keep only last 2 weeks)
sudo journalctl --vacuum-time=2weeks

The journal's binary format enables powerful filtering that would be cumbersome with plain text logs.

Writing Service Unit Files

Understanding how to write service files is essential for deploying your own applications.

Basic Structure

A service file has three main sections:

[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The [Unit] Section

This section describes the unit and its relationships:

Description= – Human-readable description
Documentation= – Links to documentation
After= – Start this unit after the listed units
Before= – Start this unit before the listed units
Wants= – Weak dependency (continue if dependency fails)
Requires= – Strong dependency (fail if dependency fails)
BindsTo= – Like Requires, but also stops when dependency stops

The [Service] Section

This section configures how the service runs:

Type determines how systemd tracks the service:

simple (default) – The process started by ExecStart is the main process
forking – The process forks and exits; systemd waits for the parent to exit
oneshot – The process exits after completing its task
notify – The service sends a notification when ready (using sd_notify)
dbus – The service acquires a D-Bus name when ready

Common directives:

[Service]
# What to run
ExecStart=/path/to/binary --flags
ExecStartPre=/path/to/pre-script    # Run before ExecStart
ExecStartPost=/path/to/post-script  # Run after ExecStart
ExecStop=/path/to/stop-script       # Custom stop command
ExecReload=/bin/kill -HUP $MAINPID  # How to reload config

# Process settings
User=appuser
Group=appgroup
WorkingDirectory=/opt/app

# Environment
Environment=NODE_ENV=production
Environment=PORT=3000
EnvironmentFile=/etc/myapp/env

# Restart behaviour
Restart=always              # always, on-failure, on-success, on-abnormal, on-abort, on-watchdog
RestartSec=5                # Wait 5 seconds before restart
StartLimitBurst=5           # Max 5 restarts
StartLimitIntervalSec=60    # Within 60 seconds

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp

The [Install] Section

This section controls how systemctl enable creates symlinks:

[Install]
WantedBy=multi-user.target  # Enable as part of this target
Alias=myapp.service         # Alternative name

Real-World Examples

Here's a service file for a Node.js application:

[Unit]
Description=My Node.js API
Documentation=https://github.com/myorg/myapp
After=network.target

[Service]
Type=simple
User=nodeapp
WorkingDirectory=/opt/nodeapp
ExecStart=/usr/bin/node /opt/nodeapp/server.js
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
Environment=NODE_ENV=production
Environment=PORT=3000

[Install]
WantedBy=multi-user.target

And for a Go application that daemonizes:

[Unit]
Description=My Go Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=goservice
ExecStart=/usr/local/bin/mygoservice
Restart=always
RestartSec=5

# Security
NoNewPrivileges=true
ProtectSystem=full
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Installing Your Service

1. Create the service file:

sudo vim /etc/systemd/system/myapp.service

2. Reload systemd to recognize the new file:

sudo systemctl daemon-reload

3. Enable and start:

sudo systemctl enable --now myapp

4. Check status:

systemctl status myapp
journalctl -u myapp -f

Timers: Modern Cron Replacement

systemd timers provide a more powerful alternative to cron jobs.

Why Timers Over Cron?

Dependency management (run after another unit completes)
Logging to the journal (no separate log configuration)
Easy to debug (systemctl status, journalctl)
Can catch up on missed runs (Persistent=true)
Resource control via cgroups
Better calendar expressions

Creating a Timer

You need two files: a .service file and a .timer file.

backup.service:

[Unit]
Description=Daily Backup Job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

backup.timer:

[Unit]
Description=Run backup daily at 2am

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Timer Options

Calendar-based (like cron):

OnCalendar=Mon..Fri *-*-* 09:00:00  # Weekdays at 9am
OnCalendar=*-*-01 00:00:00          # First of every month
OnCalendar=hourly                    # Every hour
OnCalendar=daily                     # Every day at midnight
OnCalendar=weekly                    # Every Monday at midnight

Monotonic (relative to events):

OnBootSec=5min           # 5 minutes after boot
OnStartupSec=10min       # 10 minutes after systemd starts
OnUnitActiveSec=1h       # 1 hour after the service last started
OnUnitInactiveSec=30min  # 30 minutes after the service last stopped

Useful options:

Persistent=true      # Run immediately if a scheduled run was missed
RandomizedDelaySec=5min  # Add random delay (useful for distributed systems)
AccuracySec=1min     # Timer precision (default 1min)

Managing Timers

# List all timers
systemctl list-timers

# Enable and start a timer
sudo systemctl enable --now backup.timer

# Check when a timer will next fire
systemctl status backup.timer

# View timer logs
journalctl -u backup.timer
journalctl -u backup.service

# Test the OnCalendar expression
systemd-analyze calendar "Mon..Fri *-*-* 09:00:00"

User Services

systemd can also manage services for individual users without root privileges.

# Create user service directory
mkdir -p ~/.config/systemd/user/

# Create your service file there
vim ~/.config/systemd/user/myuserapp.service

# Reload user daemon
systemctl --user daemon-reload

# Enable and start
systemctl --user enable --now myuserapp

# View status
systemctl --user status myuserapp

# View logs
journalctl --user -u myuserapp

Important: User services only run while the user is logged in, unless you enable "lingering":

# Allow user services to run at boot without login
sudo loginctl enable-linger username

Debugging and Troubleshooting

Common Issues

Service fails to start:

# Check status and recent logs
systemctl status myapp
journalctl -u myapp --no-pager -n 50

# Verify syntax
systemd-analyze verify /etc/systemd/system/myapp.service

Service starts but immediately stops:

Check if the process is forking when Type=simple
Ensure ExecStart uses absolute paths
Verify the user has necessary permissions

Changes not taking effect:

# Always reload after editing unit files
sudo systemctl daemon-reload
sudo systemctl restart myapp

Circular dependencies:

# Check for dependency issues
systemctl list-dependencies myapp
systemd-analyze verify myapp.service

Useful Debugging Commands

# Show failed units
systemctl --failed

# Reset failed state
sudo systemctl reset-failed myapp

# View all logs since last boot
journalctl -b

# View unit file changes vs vendor default
systemd-delta

# Check boot issues
systemd-analyze critical-chain

# Plot boot sequence
systemd-analyze plot > boot.svg

Quick Reference: Common Patterns

Run a Script at Boot

[Unit]
Description=Run once at boot
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/startup-script.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Web Application with Socket Activation

[Unit]
Description=Web App
Requires=webapp.socket
After=webapp.socket

[Service]
Type=simple
ExecStart=/opt/webapp/server
User=webapp

[Install]
WantedBy=multi-user.target

Watchdog for Critical Services

[Service]
Type=notify
WatchdogSec=30s
Restart=on-failure

Resource Limits

[Service]
MemoryMax=512M
CPUQuota=50%
TasksMax=100

Conclusion

systemd may have had a controversial arrival, but it has become an indispensable part of the modern Linux ecosystem. As a developer, understanding systemd basics lets you:

Deploy applications properly with automatic restart and logging
Debug production issues faster using the journal
Replace fragile cron jobs with robust timers
Understand what's happening at boot time

You don't need to become a systemd expert, but knowing your way around systemctl and journalctl, and being able to write a basic service file, will make your life significantly easier when working with Linux servers.

The best way to learn is to start using it: create a service file for one of your applications, set up a timer for a maintenance script, and dig into the journal when something goes wrong. The documentation is extensive, and the patterns are consistent once you understand the basics.