Running Your Scripts as Real Services with Systemd

[Unit]
Description=My App
After=network.target

[Service]
Type=simple
User=appuser
ExecStart=/usr/bin/node /opt/app/server.js
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

[Unit]
Description=Cleanup temp files

[Service]
Type=oneshot
ExecStart=/opt/scripts/cleanup.sh

[Unit]
Description=Run cleanup daily

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Systemd is PID 1 on most Linux distributions. It starts services, tracks their state, restarts them when they crash, and logs their output. You interact with it through systemctl and journalctl.

Custom services work the same way as built-in ones. You write a unit file — a plain text config that tells systemd what binary to run, which user to run it as, and what to do when it dies — and drop it into /etc/systemd/system/.

Your First Service File

Let's say you've a Python script at /home/anurag/mybot/bot.py that you want to run as a service. The service file:

[Unit]
Description=My Discord Bot
After=network.target

[Service]
Type=simple
User=anurag
WorkingDirectory=/home/anurag/mybot
ExecStart=/usr/bin/python3 /home/anurag/mybot/bot.py
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Each section does something specific:

The [Unit] Section

• Description — Human-readable name. Shows up in systemctl status.
• After — Wait for the network to be up before starting. Otherwise, your script might try to connect to the internet before the network is ready.

The [Service] Section

• Type=simple — Assumes the process stays in the foreground. systemd considers it "started" the moment it forks. Type=notify is the correct choice for any service that supports it — the process explicitly tells systemd when it's ready. Type=simple is a guess; systemd doesn't actually know if your process started successfully.
• Type=forking — For processes that daemonize themselves (fork and exit the parent). systemd waits for the parent to exit, then tracks the child.
• User — Run as this user instead of root. Never run application services as root unless they need privileged ports or resources.
• WorkingDirectory — Sets the working directory before executing. Required when your process reads relative paths.
• ExecStart — The command to execute. Must be an absolute path. Shell features like pipes and redirects don't work here (use a wrapper script for those).
• Restart=on-failure — Restart only on non-zero exit codes or signals. Restart=always restarts regardless of exit status.
• RestartSec=10 — Delay between restart attempts. Without this, a broken service restarts as fast as systemd can spawn it.

The [Install] Section

• WantedBy=multi-user.target — Start this service when the system reaches "multi-user mode" (normal operation). This is what makes it start on boot.

The Commands You'll Use Over and Over

Once you've created the service file, here's how to use it:

# Reload systemd so it sees your new file
sudo systemctl daemon-reload

# Start the service
sudo systemctl start mybot

# Check if it's running
sudo systemctl status mybot

# Stop the service
sudo systemctl stop mybot

# Restart (stop then start)
sudo systemctl restart mybot

# Enable start on boot
sudo systemctl enable mybot

# Disable start on boot
sudo systemctl disable mybot

# View logs for this service
sudo journalctl -u mybot

# Follow logs in real-time
sudo journalctl -u mybot -f

The daemon-reload step trips people up constantly. If you edit a service file and nothing changes, you forgot to reload. systemd caches the old config until you explicitly tell it to re-read.

Reading the Status Output

systemctl status output, annotated:

● mybot.service - My Discord Bot
 Loaded: loaded (/etc/systemd/system/mybot.service; enabled; vendor preset: enabled)
 Active: active (running) since Thu 2025-01-09 10:30:00 UTC; 2h ago
 Main PID: 12345 (python3)
 Tasks: 2 (limit: 4680)
 Memory: 45.2M
 CPU: 1min 23.456s
 CGroup: /system.slice/mybot.service
 └─12345 /usr/bin/python3 /home/anurag/mybot/bot.py

Jan 09 10:30:00 myserver systemd[1]: Started My Discord Bot.
Jan 09 10:30:01 myserver python3[12345]: Bot connected to Discord
Jan 09 10:30:01 myserver python3[12345]: Logged in as MyBot#1234

Fields worth reading:

• Active: active (running) — Process is alive. failed means it exited non-zero. inactive means it's not running and wasn't expected to be.
• enabled / disabled — Whether it starts on boot. This is the symlink in /etc/systemd/system/multi-user.target.wants/.
• Main PID — The tracked process. If this is wrong, your Type= is wrong.
• The last few journal lines appear at the bottom — usually enough to spot the problem without running journalctl separately.

When Things Go Wrong

Common failure modes and what causes them:

Error: "Failed to start" with exit code 203

This usually means the executable in ExecStart can't be found. Check that:

• The path is absolutely correct (use full paths)
• The file is executable (chmod +x script.sh)
• If it's a script, it has the right shebang line at the top (#!/usr/bin/python3)

Error: Exit code 1 / Service keeps restarting

Your script is crashing. Check the logs:

sudo journalctl -u mybot -n 50

The -n 50 shows the last 50 lines. Look for Python tracebacks or error messages.

Script works manually but not as a service

This is almost always an environment problem. Your shell session has PATH, environment variables, and a home directory. systemd services get a minimal environment — no .bashrc, no user PATH, no virtualenv activation.

Fixes:

• Use full paths in the script itself, not just in the service file
• Set WorkingDirectory properly
• If you need environment variables, add Environment=VARNAME=value to the [Service] section

A More Complete Example

A production Node.js service file with logging and environment variables:

[Unit]
Description=My Node.js Web Application
Documentation=https://github.com/myuser/myapp
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node /var/www/myapp/server.js
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=myapp
Environment=NODE_ENV=production
Environment=PORT=3000

[Install]
WantedBy=multi-user.target

Additional directives used here:

• Group — Sets the group ID in addition to the user.
• Restart=always — Restart on any exit, including clean ones. For services that should never be down. Pair with RestartSec and StartLimitBurst to avoid the restart trap described above.
• StandardOutput/StandardError — Routes stdout/stderr to the journal. Default behavior on most distros, but explicit is better.
• SyslogIdentifier — Tags journal entries. Without this, entries are tagged with the binary name, which can be ambiguous.
• Environment — Sets environment variables for the process. For secrets, use EnvironmentFile instead.

Environment Files: For Secrets and Config

Don't put secrets in the service file — it's world-readable by default and visible in systemctl show. Use an environment file:

DATABASE_URL=postgres://user:password@localhost/mydb
API_KEY=super_secret_key_here
NODE_ENV=production

Then reference it in your service file:

EnvironmentFile=/etc/myapp/env

Make sure the file is readable only by root (or the service user):

sudo chmod 600 /etc/myapp/env
sudo chown root:root /etc/myapp/env

Running Multiple Instances

Template units let you run the same service with different parameters. Name the file with an @ symbol ([email protected]), and the %i specifier gets replaced with the instance name:

[Unit]
Description=My App Instance %i

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/myapp/app.py --config /etc/myapp/%i.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

The %i gets replaced with whatever you put after the @:

sudo systemctl start myapp@production
sudo systemctl start myapp@staging

# Now you've two instances running with different configs

Timers: Better Than Cron (Sometimes)

systemd timers are an alternative to cron. A timer unit triggers a service unit on a schedule. Two files required:

[Unit]
Description=Daily backup script

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

[Unit]
Description=Run backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer (not the service):

sudo systemctl enable backup.timer
sudo systemctl start backup.timer
sudo systemctl list-timers # See when it will run next

The advantage over cron: Persistent=true means if the system was off at 3 AM, systemd runs the backup at next boot. Cron just misses the run.

When Something Breaks

systemctl status myservice, journalctl -u myservice, systemd-analyze blame. Those three commands solve 90% of service problems. status shows you whether it's running and the last few log lines. journalctl -u gives you the full log history for that unit. systemd-analyze blame tells you what's slow at boot — helpful when a service is timing out because it's waiting on a dependency that takes 30 seconds to start.

Quick Reference

Template for a basic service file — copy and edit:

[Unit]
Description=DESCRIPTION HERE
After=network.target

[Service]
Type=simple
User=USERNAME
WorkingDirectory=/path/to/directory
ExecStart=/path/to/executable
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Commands:

sudo systemctl daemon-reload # After editing files
sudo systemctl start NAME # Start now
sudo systemctl stop NAME # Stop now
sudo systemctl restart NAME # Restart
sudo systemctl enable NAME # Start on boot
sudo systemctl disable NAME # Don't start on boot
sudo systemctl status NAME # Check status
sudo journalctl -u NAME # View logs
sudo journalctl -u NAME -f # Follow logs live