Systemd: The Biggest Fallacies

Over this past year, I've seen a lot of frequently-used but logically invalid arguments for using systemd. This blog post is meant to serve as a repository of common but invalid arguments for using systemd that I and others have had to refute multiple times. This is meant to be a living document--I'll update it with more fallacies as I encounter them, and I will direct people here who make these mistakes.

Please be informed that this post is not meant to be a criticism of systemd or its authors. For the record, I personally believe that the question "is systemd good or bad?" can only be answered in terms of a particular user's requirements--systemd is good for the people who need it, but perhaps less desirable for people who do not.

Fallacy #1: "Systemd is multiple binaries, therefore it is not monolithic"

In his blog, Lennart Poettering claims that systemd is not monolithic by pointing out that it is made of upwards of 69 separate binaries. This is a non sequitur, because "modular" and "monolithic" are not mutually exclusive terms.

A piece of software is modular if it is decomposable into distinct functional units such that each unit addresses a specific concern. Systemd, the Linux kernel, and X.org are all examples of modular software. Systemd addresses its concerns with its binaries, the Linux kernel with loadable kernel modules, and X.org with its drivers and extensions.

Now, a piece of software is monolithic if its components (if it has any at all) are tightly coupled--that is, components logically depend on one another to the point where using them in different contexts requires re-implementing the missing ones. Examples include Linux and X.org--in Linux's case, you can't use a kernel module without the kernel, the kernel can't run without the requisite kernel modules to interface with the hardware, and you can't use a Linux kernel module with other kernels or as a stand-alone program. Similarly, you can't use an X video driver without the X server, you can't use the X server without at least one video driver, and you can't use X's video drivers with other graphics managers or as stand-alone programs.

Under these definitions, systemd qualifies as both modular and monolithic. You cannot run journald without systemd, and cannot run systemd without journald (at least, not without losing logging for systemd-supervised programs). None of the *ctl programs work without systemd, nor do its collection of systemd-*d daemons. It used to be possible to run logind separately, but not anymore. According to the systemd developers, udev will likely be next to hard-depend on systemd. The point is, despite the fact that systemd is comprised of multiple binaries, the hierarchical logical coupling between them means that it is more accurate to think of them as extensions to systemd-PID-1 that just happen to run in separate address spaces. They are not truly independent, composable programs.

By contrast, examples of modular, non-monolithic software include GNU coreutils, the s6 supervisor system, DJB's daemontools, and the GNU compiler collection. Not only are these applications broken into modules, but you can use the modules largely independently of one another in different contexts, composing them together to accomplish more complex tasks than the individual modules themselves can handle.

Now, "modular vs non-modular" and "monolithic vs non-monolithic" are continuous trade-offs. Systemd is less monolithic and more modular than, say, the "ls" program. But it is more monolithic than any of the non-monolithic examples above, due to the inter-binary logical coupling they exhibit.

Fallacy #1.1: "Systemd's components have well-defined interfaces, so you can just replace the parts you don't like."

This line of thinking commits an error of omission. Not all of its interfaces are stable, so this isn't a sustainable option unless you can commit to it full-time. Of particular importance is the interface to the main service daemon, which isn't stable. Also, the authors themselves concede that parts of systemd cannot be re-implemented independently.

Fallacy #1.2: "The Linux kernel is monolithic, therefore it is okay for systemd to be monolithic"

This is also a non sequitur. Whether or not systemd should be monolithic has absolutely nothing to do with the Linux kernel.

Fallacy #2: "Lots of people use systemd, therefore you should too"

This is a classic example of the bandwagon fallacy. Recall that lots of people and distros used sysvinit at one point, but that did not make it the most superior init system either.

Fallacy #2.1: "Systemd earned widespread adoption through its technical merits"

This is an example of a self-serving bias. It remains to be seen how many people use systemd because they explicitly want systemd, versus how many people use it because their favorite upstream programs depend on it, their Linux distribution switched to it, or their workplace mandated its adoption. The latter cases--installing systemd as a means to use other programs--do not count as adoption through technical merit.

Why do distributions adopt systemd, if not for its technical merits? A big motivating factor is that other application suites (e.g. GNOME) increasingly depend on it to work. If the goal of the distribution is to provide support for these end-user application suites, then pulling in systemd as a dependency is only a means to that end: they're not adopting systemd because it's technically superior; they're adopting it because it's less costly than maintaining patches to keep systemd decoupled.

Speaking of adoption, how does one measure "widespread?" Android is probably the most widely-used Linux-powered OS by far, and it doesn't use systemd. Perhaps the most widely-deployed consumer GNU/Linux distribution by sales is ChromeOS, but ChromeOS uses Upstart, and that probably won't change. So, arguably systemd isn't that widespread across the Linux landscape in the first place.

Adoption in GNU/Linux distributions isn't universal either. Gentoo and Slackware--distributions for technically-inclined users--have not adopted it as the default init system. Also, have a look at Debian's popcon numbers for systemd-sysv vs sysvinit-core. At the time of this writing there are roughly 3x as many installs as systemd-sysv (which conflicts with sysvinit-core) than there are of sysvinit-core, which is about the same disparity as that between the meta-packages for Xfce 4 and GNOME 3. So, despite being available to all major GNU/Linux distributions, it has by no means achieved universal uptake.

Fallacy #3: "People who don't like systemd just don't like change"

This is an ad hominem fallacy. Just because systemd is popular these days doesn't mean it will solve everyone's problems better than the status quo. The people who do not use systemd often have very good technical reasons for avoiding it, and there are plenty of alternative init systems that are better suited for the task.

It is tempting to think that there is no technical reason to keep using an existing init system, because systemd is backwards-compatible with initscripts. However, this line of thinking makes a lot of implicit assumptions about the needs of the user, which may not hold true. For example:

It assumes that the user is comfortable adding all of systemd's code to his/her trusted computing base.
It assumes that the user isn't already doing advanced things with namespaces and cgroups which systemd would break.
It assumes that the user is unhappy with his/her current service management setup, which systemd would replace.
It assumes that the user has sufficient RAM, CPU, storage, and power to add systemd to his/her running computer. This is not always the case for low-power embedded Linux, where "init" might just be the embedded application itself (NB: I'm referring to really low-powered devices such as this, not smartphones or equivalent hardware).
It assumes that the user has a sufficiently recent Linux kernel. This is also not always the case with more exotic hardware, where proprietary kernel modules can keep you locked into an old version.
It assumes that the user will have a completely bug-free experience and will not have to sink unjustifiable amounts of time into setting systemd up in their environment.
It assumes that the user will easily be able to migrate away from systemd if it turns out to be a bad match down the road. If the user is setting a computer up that he/she won't be able to easily administer (like an Internet-of-Things device), it will need to be as bug and quirk free as possible on the first try. A provably-correct init system would be ideal, but a tried-and-true well-understood init system would be preferable to an unproven one.

Dismissing other peoples' init system requirements as illegitimate is a great way to get written off as a troll. So is trying to tell them they're wrong about their own requirements.

Fallacy #4: "Unit files are better than shell scripts"

This is a faulty generalization that ignores a lot of important realities. Now, I don't think anyone disagrees that less code is better when you can get away with it. However, the devil is in the details, and there are subtle but important engineering details regarding service management that often get overlooked in conversations about unit files.

Fallacy #4.1: "Unit files reduce complexity"

This is misleading vividness, and a mischaracterization of the problem. While it is true that unit files look simpler than init scripts and can reduce the complexity of defining services, they don't reduce the complexity of the init system, nor do they reduce the complexity of debugging the init system when it misbehaves (see Fallacy #4.2). In fact, all the complexity that used to be in shell scripts under sysvinit (as well as a few daemons and tools) has simply moved into systemd, which currently has over 275,000 lines of code. Moreover, it's 245,000 lines of C according to this analysis, of which 81,000 are required to have a minimal working instance (this is charitably excluding systemd-udevd, which adds an additional 13,000 lines). By way of comparison, the unmodified initscripts on my Debian laptop weigh in at just over 10,000 lines of shell script (according to sloccount(1)).

It's not clear that systemd makes booting and service management less error-prone than other systems, but the overall complexity of systemd is certainly higher than sysvinit.

Fallacy #4.2: "Shell scripts are buggy"

This is a red herring often used to argue against using shell scripts. Bugs aren't intrinsic to shell scripts--buggy software is buggy, no matter what language it's written in.

For those of you who have never had to debug a C program (the language systemd is written in), take my word for it that C is a lot harder to debug than a shell script, since a lot more can go wrong in C. This is because the programmer must not only deal with variables and flow control, but also memory management, thread management, thread synchronization, file descriptor management, low-level IPC, CPU-specific memory models, C's undefined behaviors (see Appendix J), and a bunch of other things that aren't applicable to reasoning about the behavior of shell scripts. Moreover, if something goes wrong during boot, the C debugging tools aren't guaranteed to be available--they could be installed on media that gets mounted late in the boot process (such as on /usr or /usr/local). This problem does not exist when using shell scripts to boot the system.

While it is true that the programs a shell script invokes are (usually) also written in C, this does not invalidate the aforementioned claim. In practice, if the shell script fails to execute, the bug is almost always in the script, not the underlying shell or program it invokes. If systemd fails to act on a valid parameter of a unit file, however, the user must debug systemd's C code to find out why.

Fallacy #4.3: "Systemd is better because it gets rid of shell scripts!"

This is another red herring, and more of an opinion than a technical argument. However, shell scripts were never necessary to boot your system in the first place. The kernel and plenty of init systems (including sysvinit) can be configured to start an arbitrary program, written in arbitrary languages. For example, Pardus Linux's init system uses Python programs instead of shell scripts.

Fallacy #4.4: "Systemd is better because it reduces shell script code duplication!"

This is also a red herring, which distracts from the fact that nothing stops the user from consolidating all of the init scripts' common code into a file that individual scripts can source. Take a look at what the BSDs do with /etc/rc.subr, for example--service definitions with BSD's rc are often shorter than unit files!

Fallacy #5: "Systemd enables you to use $LINUX_SPECIFIC_FEATURE"

This is yet another red herring. Nothing was stopping the user from leveraging those Linux-specific features before systemd. By definition, all these features had to predate systemd before systemd could be written to use them.

Fallacy #6: "Systemd is open source, so if you don't like it, you can fork it!"

Some people are working on that, despite the fact that systemd is a large moving target.

However, this statement is an example of black-and-white thinking that usually misses the point. A non-systemd user is often more interested in having systemd not encroach on his/her systems. This fallacious argument tells them to fork systemd...to not have to use systemd.

Fallacy #6.1: "Those systemd detractors should stop whining and make their own alternative!"

This also misses the point, and is a mixture of black-and-white thinking and ad hominem. Most non-systemd users are perfectly happy to leave systemd alone and let it do its thing for the people who want to use it. Whatever init system they are currently using is perfectly fine for their needs.

The problem is that the systemd project doesn't want to leave them alone. Lennart Poettering's vision for systemd is for it to subsume the whole base OS. He's also advocated for GNOME's hard-dependency on logind, and is of course responsible for logind having a hard-dependency on systemd-as-PID-1. There are plenty of init systems, but only one of them seems to be actively trying to exclude the others by making unrelated software require it to function.

A lot of non-systemd users don't need a systemd clone, since they're happy with what they have already. So, there's no point in telling them to go work on cloning systemd.

Fallacy #7: "You should use systemd, since it gives you socket activation!"

This is propaganda. Systemd isn't needed for socket activation, since the kernel already gives it to you for free.

For Internet socket activation, all the user has to do is start the daemon, have it bind on the port, and let it sit there idly. If the kernel needs RAM, it will swap the (idle) daemon to disk. If the kernel receives an incoming connection, it will swap the daemon back into RAM so it can handle it. Systemd isn't involved at all here, but it has the same effect. As an added bonus, the kernel's virtual memory subsystem ensures the daemon's runtime state gets preserved across swaps--something systemd does not allow with its form of socket activation.

Fallacy #7.1: "You should use systemd, since it lets you assign targets to be activated on receiving the connection!"

This is also propaganda. Xinetd has done this for decades. It can be configured to run whatever would be invoked by the unit files, or run "systemctl ... enable" to achieve the same end.

Fallacy #7.2: "You should use systemd, since it supports more than network sockets! It supports pipes and UNIX sockets too!"

More propaganda. This tiny shell script achieves the same effect with pipes:

$ mkfifo /path/to/pipe
$ sh -c 'while true; do cat /path/to/pipe >/dev/null; /path/to/program; done' &

When a program writes to /path/to/pipe (i.e. echo "" > /path/to/pipe), /path/to/program will be executed. This script can be extended to do more complex things of course, like have the "while" loop take different actions depending on what you write to /path/to/pipe, but the point is, pipe activation isn't hard to come by. A similar script can be written for using UNIX sockets with netcat and socat. If lifecycle management is a concern--for example, ensuring that the pipe or UNIX socket always gets unlinked once the program is done with it--I encourage you to take a look at runfs.

Fallacy #7.3: "You need to use systemd for socket activation, since otherwise you'll encounter race conditions with parallel start-up!"

This is a moral panic. First, this has been a non-issue for decades, and continues to be a non-issue for modern event-driven systems (for example, Android gets along fine without systemd). Second, both parallel start-up and guaranteed absence of race conditions can be had using basic information from procfs.

The problem is formulated as follows. Suppose processes B depends on processes A[0], A[1], ..., A[n] to be started and listening before B can start. We want to start B in parallel with A[0]...A[n], but ensure that B does not try to connect to any A[i] until all of A[0]...A[n] are listening.

The solution prior to systemd is to simply start A[0], A[1], ..., A[n], and wait a few seconds to let them reach a listening state before starting B. The intuition is that A[0]...A[n] will reach listening state before B connects, since even one second is a very long time for a process A[i] to go from double-fork daemonizing to listening on a socket. While it is true that B could race ahead of A[0]...A[n] and try to connect before one of them is listening, the probability of this happening has proven extremely small over the decades.

Now, suppose for the sake of argument that B does manage to race ahead of some process A[i]. What happens? B's connection attempt will fail with a "connection refused" error, even though all processes A[0]...A[n] are listening. B would fail to start, unless it is programmed to handle this case (e.g. to retry connecting until a retry threshold or timeout has been exceeded).

Ideally, B is smart enough to do this on its own. If not, the socket activation solution proposed by systemd is to modify processes A[0]...A[n] so that they receive a socket descriptor from systemd, which systemd used to buffer up messages from B while each A[i] was starting up (systemd's "socket activation"). This lets B start in parallel with A[0]...A[n] while preventing it from making forward progress until A[0]...A[n] are all listening.

This solution is costly, however--it requires modifying O(n) programs, it will only work with systemd, and it will solve a problem that is rare in practice. A simpler strategy is to leverage information in procfs. We can write a script that first checks the socket descriptors in /proc/A[0]/fd/, /proc/A[1]/fd/, ..., /proc/A[n]/fd/ to see if they are in a listening state by matching their inode numbers against open sockets in /proc/net/$PROTOCOL. Once all sockets are in a listening state, the script starts B. There is no need for systemd to get involved, there is no need to modify A[0]...A[n], and the solution works wherever procfs is available (and even if it isn't, only the script--not the daemons--needs to be modified to port the "socket activation" functionality).

Fallacy #8: "$PROGRAM doesn't depend on systemd, it just depends on something that implements systemd's interface"

When talking about one of the components or interfaces that cannot be re-implemented independently (such as systemd-logind or the service bus API), this argument is a distinction without a difference. It's like saying "Internet Explorer doesn't depend on Microsoft Windows, it just depends on something that implements Microsoft Windows's interface!" While it's certainly possible to re-implement the requisite parts of the API, doing so effectively re-implements most of the platform as well. So, if $PROGRAM depends on one of these components or interfaces, then $PROGRAM either requires systemd or a mostly-complete re-implementation, making it systemd in all but name.

Fallacy #9: "Systemd is the KISS principle in action"

This is a misattribution. The KISS ("Keep It Simple, Stupid") principle refers to a system's design, not its user interface. KISS is the principle of avoiding unnecessary complexity. But, as we have established in Fallacy #4.1, the minimally-usable systemd is more complex than sysvinit and tries to do more.

Now, the systemd developers believe that the extra complexity in even the minimal systemd is necessary to meet their requirements. All I am pointing out is that these requirements are not universal, and for many, the complexity is unjustified.

Fallacy #10: "Binary log files aren't a problem in practice"

This can be either faulty generalization or false induction, and is often made in the context of "I've used binary logging, therefore binary logging must be okay for everyone."

The question we're interested in answering is "Are the log format and logging facility always able to record the set of events and failures that the user requires, in the order in which they occur?" If the answer to this question is "yes," then the choice of formatting is arbitrary, since the implementation details won't matter. Otherwise, if there are cases where the logging facility can behave unexpectedly (such as a bug in the logger, a kernel panic, or hardware failure--things that happen somewhat regularly), the implementation details do matter, since the user will need to manually intervene to determine the sequence of events and failures leading up to the bad behavior. In this case, logging structured binary records is almost always a worse design choice than logging unstructured human-readable text, since it's almost always easier to recover a plain-text log from unrecoverable logging failures (particularly corruption).

Log corruption is perhaps the hardest problem for a logging facility to address, since information is lost. However, human-readable text remains at least partially readable when corrupted, since the rules, grammar, and semantics of written language serve as naturally-occurring error-correcting measures. For example, a user can tell at a glance whether or not a string of text is truncated, or is missing one or more words, or contains data that shouldn't be there (i.e. non-printable characters), or doesn't fit any expected log message pattern, and so on, simply because logging only human-readable text lets the user deduce what the logger intended to write. However, the more binary data gets included with log records, the harder it becomes for the user to leverage written language to deduce intent (e.g. is a string of arbitrary bytes a sign of corruption, or is it part of the record, or is it a bit of both?).

At the time of this writing, journald can corrupt the log simply by crashing, putting the log into a state where it is impossible to tell whether or not it was corrupted or tampered with (the developers have known about this for years, and they don't think this is a bug). Now, it is possible to manually parse a corrupt journald log if journalctl can't help, but it requires more effort than parsing a corrupted plain-text log since the user must be aware of the journald log format in order to extract the human-readable strings from the binary records. So, when it comes to dealing with failure modes that involve log corruption (in journald's case, these are all the failures that can lead to unclean shutdown--exactly the failures a logging facility should capture), a plain-text log will almost always be of more immediate use than journald's log.

Now, journald's logging could be more robust. It could improve resilience in the face of corruption by including forward error correction codes, by keeping its binary data in a separate file (or encoding it as human-readable text), and by implementing ACID semantics on log writes. However, it does not do any of these things at the time of this writing, which makes it all the more fragile in the face of failure.

Fallacy #10.1: "Journald's binary format lets its perform authenticity and integrity checks"

This is propaganda. Most modern syslog implementations do this too, so this doesn't compel journald's adoption.

Fallacy #10.2: "Journald binary format lets it have an index that makes log access faster"

This is also propaganda. While no one will contest that indexing will speed up log access (at least in theory), there's also no technical reason why journald can't keep the index separate from the logfile, or can't write the index into the logfile as human-readable text.

Fallacy #10.3: "All the problems you have with journald can be avoided by having it simply pass everything to syslog"

This assumes a faulty premise that there is a need for journald at all in this case. If all the user needs is syslog, then why require journald at all?

Fallacy #11: "People who don't like systemd really just don't like Lennart Poettering, and that's not a valid excuse to avoid it"

First, this is putting words in the mouths of non-systemd users (ad hominem). Second, there are plenty of valid non-technical reasons to use or avoid software. Not trusting the lead developer to have your interests at heart is one of them.

The GNU/Linux ecosystem is a bazaar, not a cathedral. Software development in this space necessarily has a social dimension, because far-flung developers working on a common goal have to communicate to get things done. As such, developers cultivate reputations for their management style, coding practices, responsiveness to bug reports, and so on, and these rightfully influence the degree to which other developers trust them to do a good job on their projects (for example, Debian switched from GNU libc to EGLIBC for a time, citing friction with the GNU libc maintainer as a major reason).

Now, among some circles, Lennart Poettering has developed a negative reputation for too often breaking compatibility and dismissing constructive criticism of his projects (including bug reports). This is not an attack on his character; it is a critique of his project management style. For those users who are negatively affected by this, it makes complete sense to avoid systemd, since it does not appear to them that the project lead has their interests at heart. This is no different than a user leaving Windows for Linux, because Microsoft no longer appears to have their interests at heart.

Disliking how Lennart does his job and manages his projects is not the same as disliking him personally, but too often, it seems to me that both proponents and detractors conflate the two. It is prudent for proponents to determine which is being criticized before throwing around ad hominem accusations, and it is prudent for detractors to emphasize the target of their criticisms. My point is, skepticism towards systemd due to its management style is not ad hominem, and skepticism towards any software project due to its management style can be quite reasonable.

Fallacy #11.1: "You should judge systemd on its technical merits alone, otherwise your criticisms are wrong"

This is simply naive. Software does not exist in a vacuum--there are people, organizations, intentions, and reputations behind every line of code, and there are costs to buying into software and switching away from it even when it's free (as in beer and as in freedom). As such, it's perfectly reasonable to embrace or avoid software for reasons besides technical merits. For example, Free Software aficionados don't use non-free software, even if there exist technically superior non-free offerings, since non-free software violates one or more of the Four Freedoms.

However, suppose for the sake of argument that systemd does exist in a vacuum, and we want to judge systemd on its technical merits alone. This means that we cannot permit any biases or prejudices we may harbor for or against systemd, its authors, its users, etc. to influence our evaluation. We only want to know whether or not systemd meets a user's requirements.

To judge on technical merits alone, there first needs to be a formal specification that precisely and unambiguously defines what systemd does and does not do. Second, there needs to be a formal proof that systemd's implementation meets this specification. Both are necessary to evaluate systemd on its technical merits alone; otherwise, there's no way conclusively show that systemd meets the requirements.

Sadly, doing these two things is a huge undertaking that's not feasible in practice (this isn't specific to systemd; this applies to most non-trivial pieces of software). So, we're left with less-than-perfect heuristics for the time being, like doing black-box tests, or reading the source code, or considering the intentions and trustworthiness of the developers. But that's okay, since more often than not they still yield useful insights into whether or not software meets your requirements. There's plenty of space for argument over what are the best heuristics, but my points are that (1) technical merits aren't always enough to decide whether or not to use a piece of software, and (2) judging on technical merits alone is infeasible in practice anyway.

Fallacy #12: "The Linux kernel is complex, and you use the Linux kernel, therefore you are okay with systemd being complex too"

This is a non sequitur similar to Fallacy 1.2. Just because someone runs the Linux kernel does not mean that he/she wants more complexity than it offers. Suggesting otherwise is a false dilemma: accept all the complexity of Linux+systemd, or go without either. Complex vs simple is not a binary choice, and tolerance of software complexity depends on a wide variety of factors, including the context in which it is used, the degree of testing it has received, the degree of familiarity with its inner workings, etc.

Besides, a lot of Linux's extra complexity can be factored out at build time, which some users put in the effort to do.

Fallacy #13: "Systemd has at least 574 contributors from many different backgrounds, therefore it is not pushed or controlled by a small group"

This is naive at best, and an outright non sequitur at worst. While it is true that the systemd project has many contributors, the vast majority of systemd code came from only a few people.

I checked out the latest systemd code using the links here, and ran gitstats on it (I charitably included udev in this analysis, even though udev existed before systemd). Here's what I found:

The systemd project has 688,539 lines total (1,468,444 added, 779,905 removed). Note that this includes everything, not just the C code that compiles into systemd's binaries.
The systemd project has 17,503 commits.
The top ten contributors by number of commits represent 82.73% of all commits.
Of the top 10 contributors by commit count, eight of them are also in the top ten list of contributors by lines of code added.
The top 10 contributors wrote 90.8% of systemd. By lines of code added, the top 10 are responsible for contributing a net total of 625,711 lines.

So, it is in fact more accurate to say that systemd is controlled by a handful of people. The fact that a few hundred people each sent them a small amount of code does not mitigate this handful's influence.

Conclusion

The point of this article is to serve as a repository of commonly used but invalid arguments for using systemd. I will keep adding more fallacies as I encounter them, and update the text above to improve clarity. Again, this post is not meant to be a criticism of systemd; only a refutation of common but invalid arguments in favor of its adoption.

Comment Policy

I'm not a fan of censorship, so I don't delete comments (but Blogger.com might employ anti-spam policies). However, I will move off-topic comments to a separate web page to keep the conversation here relevant. You're free to have whatever off-topic conversations you want elsewhere :)