Process Isolation

Most multi-tasking operating systems limit the effects that one process can have upon other processes in the system. Memory access is protected, files can be opened for exclusive access by one process, etc. When processes end (cleanly or otherwise), the resources they held are reclaimed by the system.

ProcessIsolation is commonly achieved today by a complex scheme of hardware support, process resource-tables represented in a kernel address space, and running code with different privileges at a rather coarse-grained scale. This approach is expensive, having the following costs:

expensive and challenging process IO, requiring a great deal of marshaling, copying, and serialization of messages
processes are treated as homogeneous entities even though they are not; no support for type-safety, protocol, deadlock analysis.
context switches for many kernel-level operations add to their overhead; compare ExoKernel
switches between processes are very expensive, requiring one to wipe-out or replace the TLB
efficiency costs for long-running processes; memory fragments, resource clutter builds over time, etc.
coarse-grained security determined by privileges in 'kernel' layer
opportunity cost: since process internals are generally opaque to OS, it is not generally possible to optimize a process, move objects in memory to eliminate fragmentation, move a process to a different machine, etc. To achieve these things, a VirtualMachine with yet another ProcessModel is required.
compensation costs: the expensive and challenging process IO naturally encourages heavy-weight processes above lightweight ones, reinvention and duplication above reuse, and work-arounds such as multi-threading (which involves even more reinvention of process). This has a considerable memory and security cost in a multi-tasking operating system. The security cost is due to highly privileged processes duplicating code that can be performed with fewer privileges, and the the memory cost is due to literal, physical duplication of code, runtime frameworks, etc. in memory.

The hardware solution to ProcessIsolation was favored because the OS designers believed there was a great need to support native-code compilation from unsafe programming languages, such as C/C++.

A viable alternative is to achieve ProcessIsolation through software.

The software approach is explored by ErosOs, its successor CoyoteOs?, Microsoft's Singularity. Doing so involves handing the OS some sort of verifiable code (e.g. VM bytecode, proof-bearing assembly, or even a middle or high-level language) when representing processes. Further compilation may be performed by the OS itself (and optionally cached for reuse).

Achieving ProcessIsolation through software offers considerable opportunity for automatic optimizations, massively cheaper process IO (typed message passing without context switch, without marshaling, without serialization, without copying, possibly without gatekeeper given sufficient StaticTyping), fine-grained security via the ObjectCapabilityModel, better support long-running processes (including persistent processes), supports adding cross-cutting features (e.g. location transparency, persistence, a general multi-process debugging framework, multi-process deadlock analysis), and potentially allows for cross-process optimizations. Not all of these are obtained automatically, of course; they still require language design decisions aimed to support the ProcessModel.

In addition, programmers will be less apt to compensate by making heavyweight processes or introducing such error-prone concurrency solutions as multi-threading. If programmers have access to lightweight processes and cheap simple process IO that is easy to manage and configure (e.g. simple MessagePassing to a channel or process), they are apt to favor it. Since compensation is ultimately rework, this avoids rework.

The software approach is sometimes implemented for a ProcessModel within a VirtualMachine. ErlangLanguage is an example, albeit not one that achieves ObjectCapabilityModel and various other features useful for ProcessIsolation.

SeeAlso FaultIsolation, SingleLanguageOperatingSystem, MultitaskingVirtualMachine