The role of operating systems in security¶

Operating systems and hypervisors—as the lowest layers of the software stack—are responsible for managing the resources of a computing device and for creating the foundation on which secure applications can be built. From a security perspective, their most important task is to isolate security domains and to mediate (check) all operations that may require crossing such isolation boundaries. In the ideal case, the operating system protects each process from all other processes. For example, peripheral device processes must not be able to access memory allocated to a designated primary process P, learn anything about P’s activities except what P chooses to reveal, nor restrict P from using the resources allocated to it, such as CPU time. Some operating systems can even regulate information flows so that data classified as secret can never leak into processes without appropriate permissions, nor can processes modify classified data without appropriate rights. These two principles are, incidentally, the core principles of multilevel security (MLS). The details of isolation are examined in a later module. Here we continue by considering how the type of operating system affects security.

There are many ways to design an operating system. Early operating systems ran in the same and only security domain as applications, with no isolation whatsoever. Many embedded systems still operate in this way today. An application can damage the file system, the network stack, drivers, or any other part of the system.

Monolithic system. In modern general-purpose operating systems, most—though not all—of the OS code typically resides in a single security domain, strictly isolated from applications. Each application is also isolated from all other applications. This is the structure of Windows, Linux, macOS, and many descendants of the original UNIX. Because almost all OS components operate in a single security domain, the model is very efficient. Components interact simply via function calls and shared memory. The model is also secure as long as every component behaves correctly. If attackers manage to compromise even a single component, such as a driver, all protections are effectively nullified. Device drivers and other operating system extensions (e.g., Linux kernel modules) are therefore particularly important from a security perspective. These extensions, often written by third parties and more prone to defects than the core OS code, can completely compromise system security.

In such systems today, the boundary between the kernel and other security domains has become more blurred than before, because processes outside the kernel can bypass it, for example in high-speed networking, or implement as user processes such components that are not critical for performance. Examples include File System in User Space (FUSE) on UNIX systems and the User Mode Driver Framework (UMDF) in Windows. Nevertheless, the majority of operating system functionality still forms a single monolithic security domain.

A multi-server operating system represents a very different design choice. In this model, all components are isolated into their own processes, and a very minimal microkernel (essentially a scheduler) lies at the core. This structure is not as efficient as the monolithic model, because all interactions between OS components necessarily involve inter-process communication (IPC). Furthermore, a multi-server system behaves like a distributed system, and the problems of such systems can become very complex. The advantage of the structure, however, is that a vulnerable driver, for example, cannot as easily compromise the rest of the system. Moreover, although conceptually a multi-server system resembles a distributed system, much of the complexity of true distributed systems arises from unreliable communication, which does not exist within multi-server systems. The general view is that microkernel-based multi-server models have security and reliability advantages over monolithic and single-domain models, but the price of security is somewhat higher overhead.

A “library operating system” is a fourth option. In this model, a very minimal kernel is supplemented for each application by its own set of libraries that provide exactly the operating system services the application requires. This structure allows applications to omit all functionality they would not use anyway. Library operating systems were first proposed in the 1990s (e.g., in MIT’s Exokernel and Cambridge’s Nemesis projects). After several years of relative inactivity, they are becoming popular again—especially in virtualized environments, where they are commonly referred to as unikernels. From a security standpoint, unikernels do not provide as strong isolation of OS components as microkernel-based multi-server systems. On the other hand, the (library) operating system code can be much smaller and simpler. In addition, a library cannot break isolation: it is only part of the trusted computing base of each individual application, and nothing else.

The case of virtualized environments can be compared to operating systems as well. A virtual machine, its applications, and a stripped-down operating system may together form a single security domain. More commonly, however, there is a hypervisor at the lowest level, with one or more operating systems such as Linux or Windows running inside the virtual machines it creates. Such a hypervisor provides operating systems with the illusion that they are running on hardware they exclusively control. At the other end of the spectrum, the entire system may be divided into separate, relatively small virtual machines. In fact, some operating systems, such as Qubes OS, fully integrate virtualization and operating system concepts by allowing individual user processes to be isolated into their own virtual machines. Finally, as noted earlier, unikernels are also popular in virtualized environments alongside hypervisors.

One disadvantage of virtual machines is that each operating system image consumes storage space and increases redundancy. Each system believes it is the sole master of the hardware, even though resources are in reality shared. In addition, each virtual machine’s operating system requires separate administration: updates, configuration, testing, and so on. For these reasons, a popular alternative is virtualization at the operating system level. In this approach, multiple environments—called containers—run on top of a single shared operating system. Containers are isolated from each other as much as possible, with their own kernel namespaces, resource limits, and so forth, but they ultimately share the underlying OS kernel and often use the same executable code and libraries.

Compared to virtual machines, containers are lighter-weight. However, if management aspects are ignored, virtual machines are often considered more secure than containers, because virtual machines compartmentalize resources more strictly and share only a hypervisor as a thin layer between hardware and software. On the other hand, some believe containers are more secure than virtual machines because they are so lightweight that applications can be decomposed into “microservices” with carefully defined interfaces within containers. In addition, the total amount of code and functionality that must be secured decreases. An early approach to containers (i.e., “OS-level virtualization”) can already be found in the chroot command, which was added to UNIX version 7 in 1979. In 2000, FreeBSD’s Jails went much further in OS-level virtualization. Today there are many container implementations, of which Docker is among the best known.

Small operating systems. Finally, one more class of operating systems must be mentioned: those suitable for small and resource-constrained devices, especially in the Internet of Things (IoT). Although there are differing opinions on what IoT means and the range of devices spans from smartphones to smart dust, the general understanding is that IoT includes at least the least-resourced devices. For such devices, even highly stripped-down general-purpose operating systems may be too large, when the OS is expected to fit into only a few kilobytes. As an extreme example, popular IoT operating systems such as RIOT can be less than 10 kilobytes in size. They run on systems ranging from 8-bit microcontrollers to general-purpose 32-bit processors, which may still have advanced features such as a memory management unit. They may lack features expected of operating systems like Windows or Linux, including application isolation. Instead, they may support functions important in many embedded systems, such as real-time operation or low-power networking.