Networking | Programming | Security | Linux | Computer Science | About

Operating System Integrity

Assume you have an operating system that can enforce all your access control policies. Unauthorized access to resources is impossible as long as the operating system works as intended. Attackers may then direct their attention to the operating system itself and try to disable security controls by modifying the operating system. You now definitely face an integrity problem, no matter what your original concerns were. The operating system is not only the arbiter of access requests, it is also itself an object of access control. The new security policy is:

Users must not be able to modify the operating system.

The threat model in operating system security assumes that the attacker has access to the operating system command line, but not to the physical hardware. When securing an operating system, two competing requirements have to be addressed:

Two important concepts commonly used to achieve these goals are modes of operation and controlled invocation (also called restricted privilege). These concepts can be used in any layer of a computing system, be it application software, operating system, or hardware. However, these mechanisms can be disabled if the attacker gets access to a lower layer.

Modes of Operation

The first prerequisite for an operating system to be able to protect itself from users is the ability to distinguish between computations ‘on behalf of’ the operating system and computations ‘on behalf of’ a user.

Mode of operation – The mode of operation defines which actions, e.g. machine instructions, function calls, connections to network ports, may be performed on a system.

In dual-mode operation a system can work in

A status flag in the processor’s control register can record the current mode at hardware level. For example, the Intel 80x86 processor has two status bits, thereby supporting four modes. The Unix operating system distinguishes between supervisor (root) and user mode.

Why are such modes useful? For example, the operating system could grant write access to memory locations only if the processor is in supervisor mode to stop users from writing directly to memory and corrupting the logical file structure.

Controlled Invocation

When a user wants to execute an operation requiring supervisor mode, e.g. a write to a memory location, the processor has to switch between modes – but how should this switch be performed? Simply changing the status bit to supervisor mode would give all privileges associated with this mode to the user without any control of what the user actually does. Therefore, it is desirable that the system only performs a predefined set of operations in supervisor mode and then returns to user mode before handing control back to the user. We refer to this process as controlled invocation.

Controlled invocation – Invocation of a function that executes privileged instructions to provide a limited, well-defined functionality and then returns to user mode.

Published on Fri 02 January 2009 by Ralph Holdsworth in Security with tag(s): operating system integrity