DSCI525:

Trusted System Design, Analysis and

Development

Kernel Implementation Strategies

[Link] Ryutov

This content is protected and may not be shared, uploaded, or distributed.

 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Presentation5

©Tatyana Ryutov
 Presentation6

©Tatyana Ryutov
 Your Review Topics
• From my understanding python has no true private methods (class
variables are directly accessible through name mangling, and
decorators like static methods are not enforced). Does that mean that
even though Python supports OOP it does not conform to Parnas
original vision of information hiding? because its information-hiding
features are not enforced like in Java or C++.
• Please review relationship between SK and RM.
• And how are separation kernels different from GEMSOS? Are
separation kernels used today, and if so, why are they more
commercially viable than GEMSOS?
• Where is the LDT stored?
• Can we talk more about different system designs and how to improve
them with principles?
• L4.Q3
• The kernel layers for make known was confusing to me. Can you
please explain again?

©Tatyana Ryutov
 Review: L4.Q3
• Why is the GEMSOS design decision “Gates can only be
called from PL1” important (from security point of view)?

Student’s question: hat are examples of non-TCB

subjects that use rings 2-7 for GEMSOS?
©Tatyana Ryutov
 Review: BLP “get-read” Model Rule
Example
• BLP model defines “elements” of rule
– Objects considered to be segments
• “Elements” of rule are essentially parameters
– Process-id
– Segment-id
– “read”
• Check that conditions are met
– Current access class of Process-id dom Segment-id
– If met, SDW is added to descriptor segment
• Mode of access is the requested “read”
– Abstractly, means alter current access (set b)
• Mapping between elements of the model and parameters
of kernel calls are not trivial
– Elements of the models are not one to one map to parameters of
the call
6
©Tatyana Ryutov
 Review: GEMSOS Make Known API
function
kp_makeknown_segment(
mentor : unsigned; PLSN of the mentor of the segment
entry : unsigned; to be made known
seg_num : unsigned; Entry number of the segment
seg_mode : seg_acc_type;
sync_mode : sync_acc_type;
PLSN to assign
ring_num : byte;
var limit : unsigned; requested mode of access
output

var class : access_label

the ring number of the
) : unsigned; subject for which the
segment is made known
exception

©Tatyana Ryutov
 Review: GEMSOS Kernel Layers
Example
(Applications, Kernel Gate Library)
Gate Layer
Process Manager (PM) Process
Upper Device Manager (UDM) Local
Segment Manager (SM)
Upper Traffic Controller (UTC)
Memory Manager (MM)
Inner Device Manager (IDM)
Secondary Storage Manager
Non-Discretionary Security Manager (NDSM) Kernel
Kernel Device Layer (KD) Global
Inner Traffic Controller (ITC)
Core Manger (CM)
Intersegment Linkage Layer (SG)
System Library (SL)
(Hardware)

©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
Gate Layer
Process
Local

• Subjects in PL1 enter the kernel by generating a specific signal kernel trap
• Gate Layer - a trap handler that manages the interface to the kernel
– Receive call and completes entry to Ring 0
– Copy parameters from outer ring to Ring 0
– Invoke the makeknown_segment kernel entry point
– Marshall parameters
• Entry point call to next layer
– Process Manager (PM)

©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
Gate Layer
Process Manager (PM) Process
Local

• Process Manager (PM) provides the management of the

application process structures
– Obtain current access class of BLP “Process-id”
• Explicit required “element” of BPL rule
– Access class is input parameter to lower layer call
• Make call to next layer - Segment Manager (SM)
– Manages all non-kernel access to segments
– Skip the Upper Device Manager (UDM) layer
• Provides the interface to I/O devices for non-kernel subjects

©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
Gate Layer
Process Manager (PM) Process
Upper Device Manager (UDM) Local
Segment Manager (SM)

• Segment Manager (SM) manages all non-kernel access to segments

– This layer provides a per-process Known Segment Table (KST) and maps
process-local segment names into unique, system-wide names
– Use PLSN to index into KST to get Alias Table handle
– Use Alias Table handle and entry number as parameters
• Call Secondary Storage Manager
– Skip Upper Traffic Controller (UTC)
– Skip Memory Manager (MM)
– Skip Inner Device Manager (IDM)

©Tatyana Ryutov
 Kernel Layers for Make Known
(Applications, Kernel Gate Library)
Gate Layer
Process Manager (PM) Process
Upper Device Manager (UDM) Local
Segment Manager (SM)
Upper Traffic Controller (UTC)
Memory Manager (MM)
Inner Device Manager (IDM)
Secondary Storage Manager
Non-Discretionary Security Manager (NDSM) Kernel
Kernel Device Layer (KD) Global
Inner Traffic Controller (ITC)
Core Manger (CM)
Intersegment Linkage Layer (SG)
System Library (SL)
(Hardware)

©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
... Process
Segment Manager (SM) Local
... Kernel
Secondary Storage Manager Global
...
(Hardware)
• Secondary Storage Manager (SSM)
– Provide the abstractions of segments and the segment naming hierarchy
– Provide the interface to internal I/O to the disk devices used for permanent
storage of segments
– Alias Table handle and entry number for target segment
– Access target “Segment-id” access class
• Is an explicit required “element” of BPL rule
• Return target access class to segment manager
• Return target RAM address from alias table

©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
... Process
Segment Manager (SM) Local
... Kernel
Non-Discretionary Security Manager (NDSM) Global
...
(Hardware)

• SM receives target access class from SSM

– Check access class of Process-id dom Segment-id
• Example is for BLP “get-read” Model Rule
– Call Non-Discretionary Security Manager (NDSM)
• NDSM Layer
– Responsible for the interpretation of access labels in terms of
the specific security policy implemented
– Tests dominance relation between access classes
– Returns dominance result to segment manager
©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
Gate Layer
Process Manager (PM) Process
Upper Device Manager (UDM) Local
Segment Manager (SM)

• SM receives dominance result from NDSM

– Use requested PLSN specified for segment-id
– Confirm that entry is free in LDT
• Create a new SDW in LDT
– Use RAM address returned by SSM
– Use kernel call requested access mode (i.e., read)
• Create new pathname in KST

15
©Tatyana Ryutov
 Review: Kernel Layers for Make Known
(Applications, Kernel Gate Library)
get access class Gate Layer
of process Process Manager (PM) Process
Upper Device Manager (UDM) Local
get alias handle Segment Manager (SM)
Upper Traffic Controller (UTC)
Memory Manager (MM)
get access class Inner Device Manager (IDM)
of segment Secondary Storage Manager
Non-Discretionary Security Manager (NDSM) Kernel
Kernel Device Layer (KD) Global
check BLP rules Inner Traffic Controller (ITC)
Core Manger (CM)
Intersegment Linkage Layer (SG)
System Library (SL)
(Hardware)

©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Virtualization and Sharing
• MAC demands SK able to isolate [GAS]
– Prevent actions of one subject influencing another
– Prevent even if subjects want to communicate
• Most real-world system share physical resources
– Memory
– I/O devices
– I/O media
– Communications lines
• Fixed allocation is expensive and inconvenient
– Resources must be shared among processes
– Dynamically assigned to processes as need arises

©Tatyana Ryutov
 Virtualization Principles
• Easiest way to control sharing is to virtualize resources
– Process accesses via virtual name or address
– Kernel maps virtual into a physical name and controls sharing
• GEMSOS kernel virtualization
– Segmentation & PLSN
• To isolate prevent knowing physical identifiers
– Dynamic resource allocation can give covert channel
– General solution is to only reveal virtual information
• Two special cases not needing virtualizing:
1. Static allocation to set of processes, not reallocated
2. Allocation by direct action of an individual (human)

TS TS S S

TS TS S S

TS TS S S
©Tatyana Ryutov
 Constructing Controlled Sharing
• Information sharing is real goal for multi-user system
– Major purpose of policy is to specify allowed sharing
– Physical resource sharing is NOT the primary goal
• In general cannot eliminate inter-process sharing
– Equivalent to static allocation to each process
– Necessary that SK manage resource sharing
• Footnote: Gasser wrongly says can not eliminate
channels
– Per [FER], GEMSOS can have no storage channels
– But in fairness to Gasser, no other SK claims that

20
©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Users Need Trusted Path
• Often users must interact directly with SK
– Kernel and individual mutually authenticate
– Core concern is Trojan horse in untrusted code
• User login is quintessential example of the need
– In TCSEC is explicit requirement as low as Class B2
– Trojan horse can trick user to give away password
• User for a MAC policy must know access class
– Needs access class of output information received
– Needs to constrain access class on input information

©Tatyana Ryutov
 Administrator Needs Trusted Path
• Administrator needs trusted facility management
– In worse case, interface with entire trusted SW
• Often need bounded path for specific function
– Change DAC security attributes of a user
• Set/reset password
• Grant authorization for group membership or role
– Change and manage MAC security attributes
• Set/remove user clearance
• Downgrade object classification – perhaps by copying
• Do resource allocation, e.g., change network connections
high to low

23
©Tatyana Ryutov
 Constructing Trusted Path Input
• Individual needs a direct communication link
– To the kernel itself
– To trusted software running on the kernel
• Classic “secure attention key” to initiate
– Kernel is assured can only be originated by user
– Untrusted software cannot intercept, mask or fake
• Can use a primitive physical initiation
– A separate electrical connection, e.g., manual switch

©Tatyana Ryutov
 Constructing Trusted Path Output
• Challenge: how do you know that what you see
on the screen is not manipulated by TH?
• Separate user display hardwired to kernel
– Complete separate terminal device
– Special lights controlled by kernel
– Special reserved area of the screen
• Serially reusable display for a trusted path input
– Kernel must make sure TH cannot interfere

25
©Tatyana Ryutov
 GEMSOS Kernel Trusted Path Support
• GEMSOS kernel does not support direct user input hence,
there is no need for trusted path
• However, GTNP supports virtual machines (VM)
– If one builds virtual machine on top, it has to support user input
• Example: change access class of single-level device
– Requires trusted path
• GTNP provides trusted path mechanisms:
– All serial I/O devices allow detection of <BREAK> key
– User-definable key sequence for secure attention key
– VM component layers “can easily intercept activation”
• Can hit “secure attention key” any time to not forget where
you are

©Tatyana Ryutov
 Example: Building Trusted Path on x86
Computers
• A Trusted Path contains three components: the trusted path program endpoint (PE), the
device endpoint (DE), and the communication path between these two endpoints

micro-kernel that
runs at a higher
privilege
level than the OS

indicates the correct PE-

to-DE trusted path state

Research prototype at Carnegie Mellon University

©Tatyana Ryutov Building Verifiable Trusted Path on Commodity x86 Computers, Zongwei Zhou, et. al
 Smartphone: Trusted Path Example
• Leverage ARM TrustZone: normal and secure world
– Secure monitor mode switches the two worlds
• TrustUI: design a security-oriented split driver model
– Implemented in Samsung Exynos development board with ARM Cortex-A9 processor
• Secure world exclusively uses a LED indicator to indicate the security status
– LED & color randomization

Building Trusted Path on Untrusted Device Drivers for Mobile Devices, Wenhao Li et. al
[Link] in Resources
©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Trusted Functions
• In addition to trusted path [GAS]
– Like trusted path can challenge ideal layering
• Functions trusted to maintain system security
• Often carried out in autonomous processes
– Are not a kernel layer
• Not in inner hardware-protected domain
• Run outside the kernel and use kernel services
– Are clearly part of trusted software, i.e., in the TCB
• Two design approaches:
– Trusted function “privileged” to bypass SK constraints
– “Trusted subject” per BLP – limited *-property

30
©Tatyana Ryutov
 “Privileged” Trusted Functions
• Examples:
– internal backup allowed to look at kernel DBs
• “Kernel has a special kind of interface”
– Needs to identify and distinguish privileged processes
• E.g., created with LDT that has “gate” for special API
– May be privileged to modify kernel databases
• E.g., change segment’s access class
– May not be constrained by SK
• Kernel’s special API not mapped to FSPM rules
– Difficult to come up with rigorous definition
• Logically should be considered part of the kernel
– Architecturally run as processes outside SK

©Tatyana Ryutov
 “Privileged” Interface Challenges
• Can become catch-all for the difficult to verify
– No good reason to subject to less scrutiny that SK
– Difficult to apply formal verification tools
• Experience is substantial increased trusted code
• Weak argument it is just “implementation detail”
– Interface to kernel is not like for internal kernel layers
– Cannot use assurance tools like layering
• There is no one consistently enforced policy
– Is not a decomposition of reference monitor policy
– Hard to make convincing argument policy is enforced

©Tatyana Ryutov
 BLP Trusted Subjects
• Trusted subjects - trusted not to violate security even without enforcing
confinement
– Essentially, a “write down” violating *-property
• E.g., lowering object’s security level
– Solution: define set of trusted subjects that sanitize or remove sensitive
information before security level is lowered

Trusted
Secret-
Confidential

©Tatyana Ryutov
 FSPM-compliant “Trusted Subjects”
• Identical kernel API for trusted and untrusted
– Rigorously decomposed from system security policy
– Only trusted aspect is that *-property is not applied
• Trusted subjects explicit part of FSPM
– Can be precisely constrained to range of access classes
• Dramatically reduces verification kernel
– Primarily verify info flow between access classes, e.g.,
email server that receives massages at different access
classes

34
©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Trojan Horse (TH) Context for Policy
• Definition from TCSEC:
A computer program with an apparently or actually
useful function that contains additional (hidden)
functions that surreptitiously exploit the legitimate
authorizations of the invoking process to the detriment
of security. For example, making a “blind copy” of a
sensitive file for the creator of the Trojan Horse.
• “Far easier to exploit” than most flaws [GAS]
– “Fundamentally impossible to prevent” in nearly all
• Need “complete change in philosophy”
– Can be successful in “a flawless, bug-free system”

36
©Tatyana Ryutov
 Limiting the Trojan Horse
• Extremely difficult to keep TH out of your system
– No “practical scheme for detecting a Trojan horse”
– Not effective to “scrutinize vendor software”
• Mandatory controls are essential [GAS]
– TH “can defeat any type of DAC mechanism”
• Impossible to eliminate TH or prevent it running
– Even in a system enforcing MAC
• MAC prevents TH from giving away information
– Only thwart crossing MAC access class boundaries
– TH can still bypass discretionary controls

37
©Tatyana Ryutov
 Example: TH and DAC

Access Control Bob: read, write

Uses shared program

Word Reads
Processor Sensitive file
Sensitive

Bob

Bob, Joe: read, write

TH Copies
Sensitive file
to Joe’s file Sensitive
Joe Inserts Trojan Horse
Into shared program

©Tatyana Ryutov
 Example: TH and MAC

Access Control
secret
secret

Word Reads
Processor Sensitive file

Bob

unclassified unclassified
TH Tries to copy
Secret file
to Joe’s file
Joe

©Tatyana Ryutov
 Balanced Assurance for DAC
• “Nobody questions the usefulness of DAC”
• Great deal of controversy about DAC in SK
– At tension with “as small and simple as possible”
– “Cost to implement anything in a kernel is quite high”
• Are much lower cost ways to implement DAC
– Already a useful part of most commodity OSs
– The price includes lower assurance
• MAC is almost always implemented in the kernel
– Add DAC in expanded TCB perimeter, i.e., “subsets”
• Such approaches called “Balanced assurance”
– Cost of assurance in balance with protection provided

40
©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Kernel Implementation Strategies
• In general a SK resembles an OS [GAS]
• Most differences follow from OS priorities
– SK gives the high priority to the three principles in comparison to
development cost, functionality, performance, backward compatibility,
flexibility, etc.
• Verifiability has the most direct impact on kernel
– SK is too primitive and inconvenient to use directly for an interface to
applications, need an OS layer on top of SK
• Three classes of SK implementation strategies
1. New operating system
2. Compatible operating system (emulation)
3. Identical operating system (virtual machine)

©Tatyana Ryutov
 Review: System Calls
• System call is an interface between a user app and a service
provided by the OS
– Provides a safe mechanism to request specific kernel operations
– The API of the OS from the user program’s point of view, accessible
via a library of code, e.g., libc

©Tatyana Ryutov
 Kernel Implementation Strategies [GAS]

ISOS - existing insecure operating system

RMC - reasonably modern computer
SOS - secure operating system
©Tatyana Ryutov
 New Operating System
• Designing system from scratch
– Include not just the kernel, but OS and applications
– Use a hardware base for verifiability and performance
• Distinctive is lack of backward compatibility
– Define your own OS interface
• Simple mapping of OS features to SK features
– Interface compatible with the security policy enforced by the kernel
• Example:
– System low process writes readable data for others, e.g.,
configuration information
– System high process can read all data, e.g., for audit
• Example is an “appliance” vs. general purpose
– Class A1 SCOMP and Boeing LAN designed for that

©Tatyana Ryutov
 Why are we looking at these old
systems?
• Goal: demonstrate the design and implementation
decisions behind the development of security
kernels
– What worked and what did not, so that you have
informed decisions for future
• Scomp and Gemsos security kernels are as close
to secure operating systems as possible
– These efforts led to the development of the assurance
methodologies that we have today
– Challenges: performance, practical utility, and
maintenance complexity

©Tatyana Ryutov
 Secure Communications Processor
(SCOMP)
• Kernel-based system
– Designed to implement Multics MLS requirements
– Formally verified hardware and software
– Was used in applications such as military mail guards (now “data
diodes”)
• Original idea was to build an emulator to allow execution on an
ordinary OS (UNIX)
– Issues:
• converting between incompatible representations
• inefficient HW support
• Ended up with new HW and API (SCOMP Kernel Interface
Package SKIP)
– Honeywell Level 6 minicomputer augmented with a custom Security
Protection Module (SPM)
– All authorization was checked in hardware
– SCOMP had a security kernel and 4 hardware rings
©Tatyana Ryutov
 SCOMP Design Choices
• The designers threw some compatibility away in the
name of security
– SCOMP was not UNIX
• Hard problem: interfacing groups with different
security levels
• MLS policy was a first-class task
– Replace procedural/admin controls with MLS policy

©Tatyana Ryutov
 SCOMP Architecture

©Tatyana Ryutov
 Clever IO
• SCOMP uses descriptors for IO, similar to memory
descriptors
– apps must ask SK for a hardware descriptor
• Because mediation happens in hardware, the
drivers do not need to be in Ring 0
– Decreasing the size of the security kernel
– I/O extensibility (re-verification of the kernel not needed)
• Modern hardware (e.g., x86), adopted the four-ring
architecture of SCOMP, but not the I/O mediation

©Tatyana Ryutov
 SCOMP Architecture Notes
• Access to resources is mediated using MLS policy:
– App requests HW descriptor
– If granted, SK builds the descriptor (object + permissions)
and returns a reference to the descriptor
• Ability to run drivers securely in user space
– by mediating access between the CPU and I/O bus
• Isolation provided by the ring mechanism
– Rings and transitions are implemented in hardware
– Access bracket mechanism controls access
• Verification was part of the process
– the basis for the A1 evaluation level

©Tatyana Ryutov
 SCOMP Hardware
• Based on Multics with two key changes:
– Only four rings, all in hardware
• Argument addressing mode prevents confused deputy problem
– A specific type of privilege escalation: access caller-supplied
arguments at the caller’s privilege level
• Kernel prevents itself from accessing data that the caller could
not access
– Hardware includes a security protection module
(SPM)
• Provides a tamperproof service to mediate all memory
and I/O accesses

©Tatyana Ryutov
 SCOMP: Security Protection Module
• SCOMP uses a security protection module (SMP) which interfaces with the
Virtual Memory Interface Unit
• Mediates all accesses to I/O controllers and memory by mediating the I/O bus
• Translates virtual addresses to physical segment addresses for authorization

Honeywell Level 6/DPS 6, a bus-structured 16-bit minicomputer

©Tatyana Ryutov
 SCOMP: Trusted Operating Program
(STOP)
• Three components:
– A security kernel (ring 0)
– A set of trusted software
• Provides administrator, operator, and user services
necessary to interface with the security kernel
– A kernel interface package for user applications
• File system mechanism, process control, and device I/O
• These elements provide users with an efficient
interface for the development of applications

©Tatyana Ryutov
 SCOMP: Trusted Operating Program
Security Kernel
• Resource management:
– Memory management, process scheduling, interrupt
management, audit and reference monitor
– 10K lines of code mostly in Pascal
• Supports segments, devices, and processes
• Access control similar to Multics, but ring brackets
allow for owner/group/others
– BLP sensitivity levels and categories, ring bracket policies
and discretionary policies
• 38 gates to enter ring 0

©Tatyana Ryutov
 SCOMP: Trusted Software
• Trusted software runs services that do not require ring 0
privilege, but provide functions that must be trusted to enforce
control on user applications properly:
1. Trusted not to violate system secrecy or integrity goals, e.g., secure
loader
2. Trusted to maintain security policy correctly: e.g., user authentication
• Three kinds of trusted software:
1. Trusted user processes: login, DAC management, MLS level selection,
process management
2. Trusted operation services: system management, logging, startup,
shutdown, set time, etc.
3. Trusted maintenance services: modify system data, install new
program versions, etc.
• Invoked through a trusted path directly by users

©Tatyana Ryutov
 SCOMP: Kernel Interface Package
(SKIP)
• Interface for user applications to access trusted functions
• Operate on user-level objects
– Files via a hierarchical file system with compatibility
– Process management
– Concurrent I/O through an event mechanism
– SKIP functions run in ring 2, (invoked via gates) and the
libraries run with user applications in ring 3
• For example, calls to modify file system state, such as renaming a file, are
invoked from a user application using the SKIP library which invokes a
SKIP gate before being processed in ring 2
• Thus, file system operations are protected from user applications

©Tatyana Ryutov
 SCOMP: Applications
• The Scomp hardware and software is general purpose, but it
defines a completely new application interface, so new
application software needs to be constructed
• Unix??
– Emulation was found to be too slow and insecure
• Mail guard
– Makes sure that secrets are not leaked in communications to less
secret subjects
• Secure Office Management System (by Honeywell)
– Word processor, email, spreadsheet, database, and printing support
– Challenge: other vendors developed office software for ordinary
operating systems with greater features

©Tatyana Ryutov
 SCOMP Evaluation: Complete
Mediation
• How does the RM interface ensure that all security
operations are mediated correctly?
– All mediation done in hardware
– Access control via segments (memory and I/O)
• Does the RM interface mediate security-sensitive
operations on all system resources?
– File system in ring 2 controls access to files
• Files are defined at a higher level
• Initial access to file data depends on access to I/O

©Tatyana Ryutov
 SCOMP Evaluation: Tamperproof
• How does the system protect the RM from
modification?
– Protection rings
– The security kernel runs in ring 0, and only 38 gates
permit access to the kernel from other protection rings
• Does the protection system protect the TCB?
– Also protection rings, just like Multics
– Ring brackets provide control over operations
– Controlled ring-crossing to allow less privileged
software to access an inner ring for a service function

©Tatyana Ryutov
 SCOMP: Verification
• Unlike Multics, the designers of SCOMP wanted
verifiable security, TCSEC A1 evaluation
• SCOMP was small
– Security Kernel: about 10k lines, mostly in Pascal
– Trusted software: about 11k lines
– SCOMP also has a “secure attention” key, which allowed a
user to be sure that they were accessing the OS not
something “in the middle”

61
©Tatyana Ryutov
 SCOMP Verification: Verifiable
• What is the basis for the correctness of the TCB?
– Verified with formal analysis tools:
• FTLS written in the SRI language SPECIAL
• Trusted software was verified using the Gypsy methodology
• Does the protection system enforce the system's
security policy (MLS secrecy and DAC integrity)?
– Also verified for correctness
• Clarity and consistency of the documentation
developed during product implementation

©Tatyana Ryutov
 Outline
• Review
• Additional SK Construction Techniques
– Virtualization and sharing
– Trusted path
– Trusted functions
– Kernel security policies
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Compatible OS (Emulation)
• Constraint: compatibility with existing applications
• Emulate insecure operating system (ISOS)
– Freedom to define interface with kernel (not visible to apps)
– KSOS project for Unix
• Existing UNIX apps to run without modification/recompilation
– With luck retain some code from original ISOS
• Hard to modularize not layered OS
• Typically emulator runs in each process on top of SK
– Renders OS calls into kernel calls
• Need hardware with at least three domains
– Protection rings are an efficient implementation

64
©Tatyana Ryutov
 Operating System Emulator
• Emulator - program running in an intermediate domain of each process emulates
operating system calls, using combinations of kernel calls
USERS
A B C D E F

APPLCATIONS USER DOMAIN

ISOS EMULATOR SYSTEM DOMAIN

VERIFIABLE SECURITY KERNEL

KERNEL DOMAIN

Segmented Hardware
©Tatyana Ryutov
 Emulation: Issues
• Incompatibilities between the security policy that the kernel
must enforce and the interface to be emulated
– Some ISOS interface functions not secure for policy
• E.g., chmod
– Some functions exceptionally difficult to emulate
• MAC controls in kernel limit emulator
– Emulator cannot reveal global dynamic status data
• E.g., ps command in Linux
– Cannot pass file descriptors on process creation
• E.g., Unix fork/exec process creation model
– Lead to complex per access class data structures
– Best you can do is single OS image per access class
• Inter-process communication is hard problem
– MAC prevents system-wide synchronization objects (e.g., locks)
– Can lead to many multiple instantiations of database

©Tatyana Ryutov
 Kernelized Secure Operating System
(KSOS)
• Intended to provide a provably secure OS for larger
minicomputers (1978)
• Developed at Ford Aerospace
– PDP 11/70 architecture (DEC)
• Goals:
– Compatible with the UNIX OS
• Emulation of the UNIX system call interface
• V6 UNIX kernel, at 10K lines of code and 48 system calls,
was much too big(!) to be formally specified or verified
– Support MLS
– Performance comparable to that of UNIX

67
©Tatyana Ryutov
 KSOS: System Structure
• KSOS is composed of three components:
– Security Kernel (trusted)
– UNIX Emulator (untrusted)
• Is a part of each UNIX process which on the PDP-11/7O
– Non-Kernel System Software (trusted)
• Because the Emulator is "untrusted" and is not intended to be verified, it cannot be
used by trusted software: rather, such software must use the kernel directly

NKSR: NON-KERNEL SECURITY RELATED SOFTWARE, e.g., login “trusted process”

©Tatyana Ryutov
 KSOS: Security Kernel
• Kernel objects:
– processes, process segments, kernel files (to support
directories), devices, file subtypes
• Supports MAC confidentiality and integrity, and DAC
– UNIIX style DAC: 9 bits, r/w/e u/g/o
• 32 kernel calls
• Implemented in a higher-level type safe language
Modula-2 (the type safety of Pascal)

©Tatyana Ryutov
 UNIX Emulator
• UNIX Emulator
– Translates UNIIX system calls to SK
– Creates the UNIX file system from the more primitive file
system provided by the kernel
– Manages UNIX directories
• Kernel enforces MAC confidentiality, integrity and DAC
• KSOS Applications
– Applications that utilize the full KSOS system
• Based on UNIX
– Applications that use the kernel directly without Emulator
• E.g., MLS applications

70
©Tatyana Ryutov
 KSOS: SK Verification
• SK design was documented and specified in the
SPECIAL specification language
– A formal, nonprocedural language describing the behavior
of systems in the manner suggested by Parnas
• SK specification was examined using the Boyer-
Moore theorem prover
– Proof that the kernel specification contained no explicit
violations of the BLP model
• This verification allowed KSOS to be considered a
candidate for an “A1” rating

71
©Tatyana Ryutov
 KSOS: Problems
• Poor performance of UNIX emulator
– KSOS kernel and the KSOS Unix emulator were each
larger than contemporary Unix systems
– Subsequent security kernel efforts, such as the SCOMP,
avoided performance issues through a hardware-level
implementation
• Verification of the "trusted processes”
• Mandatory integrity model was overly restrictive

72
©Tatyana Ryutov
 Outline
• Student presentations
• Review
• Kernel implementation strategies
– New operating system
– Case study SCOMP
– Compatible operating system (emulation)
– Case study: KSOS
– Identical operating system (virtual machine)
– Case study: VAX VMM

©Tatyana Ryutov
 Virtual Machine (VM)
• Virtualization uses software to create an abstraction layer over computer
hardware
• Idea: abstract HW of a single computer into several different execution
environments
– This indirection layer creates virtual system (virtual machine, or VM) on
which OSs or applications can run
• Components:
– Host – underlying hardware system (or OS)
– Virtual machine manager (VMM) or hypervisor – manages the resources of
the underlying hardware and provides an abstraction of one or more VMs
• Runs in the most privileged level (kernel mode)
– Guest – software running within a VM (i.e., guest OSs and guest
applications)
• All guest software (including the guest OS) runs in user mode
• Single physical machine can run multiple OSs concurrently, each in its
own VM

©Tatyana Ryutov
 Physical vs. Virtual Machine

hypervisor

• Extra level of indirection (software abstraction):

– Enforces isolation
– Multiplexes physical HW across VMs
©Tatyana Ryutov
 Hypervisor Types
• Native hypervisor (type 1) runs on bare HW
– E.g., VMware vSphere with ESX/ESXi, Microsoft Hyper-V, Citrix
Hypervisor (Xen Server), KVM
• Hosted hypervisor (type 2) runs on top of another OS
– E.g., VMware Workstation, VMware Player, VirtualBox
• A combination of the native and the hosted approaches
– Processor virtualization + dedicated hosted OS

VM VM
app app app app user
VM VM mode
app app app app user guest OS guest OS
mode
guest OS guest OS
hypervisor (VMM)
kernel kernel
hypervisor (VMM) host OS mode
mode
hardware hardware

Type 1 Type 2

©Tatyana Ryutov
 Identical OS (Virtual Machine)
• Most restrictive case, but may be most realistic
– Minimum or no changes to OS code
– No changes to applications – object code compatible
• Simulates air gap: provides isolation (but not sharing)
• Kernel is virtual machine monitor (VMM)
– Creates multiple virtual machines (VMs)
• Kernel’s idea of a “subject” is a VM
– Each VM has a unique identifier and a set of security attributes
– Kernel cannot perform any finer-grained access control over
processes inside VM
• Principal “objects” are virtual disks (physical disc partitions),
not files

©Tatyana Ryutov
 Kernel-based VMM Approach

USERS
A B C D E

User
Processes Virtual Machines
(Subjects)

ISOS ISOS ISOS

SECURITY KERNEL
Virtual Machine Monitor (VMM) KERNEL DOMAIN

Virtualizable Hardware

Type1 hypervisor
©Tatyana Ryutov
 VMM Configuration Alternatives
• Designer has two basic choices for a VM
1. Each VM is a surrogate for a single users
2. Each VM is used by group of several users
• Single user per surrogate virtual machine (VM)
– Kernel can enforce policy on a per user basis
– User may have multiple processes – same rights
• Allow single VM to serve group of several users
– Each user considered members of a group
– Per group access control, e.g., RBAC
• Primary issues in basis for choice
– Security policy – access control per user or per group
– Performance – a classic issue for use of VMM

©Tatyana Ryutov
 Implications of the BLP
Security Model on VMM
• Pure isolation hypervisor divides a machine into
partitions, and permits no sharing
– Similar to a collection of system-high separate computers
– Only useful for customers of the largest and most
expensive servers
– Is it more secure than running on different machines?
• Sharing hypervisor permits significant resource
sharing between partitions
– Requires a secure file store

80
©Tatyana Ryutov
 Case Study: VAX/SVS VMM
• Goal: build a high-assurance VMM on RM concept
• VMM is security kernel
– Type 1 VMM, runs directly on the hardware
– Can run untrusted VMs
– Invoked on trap to execute privileged instruction
• Only VMM can access hardware directly
• Was designed “from scratch” and ignored compatibility

©Tatyana Ryutov
 VAX VMM: Design Choices/Constraints
• Subjects – users and VMs
– Users access the security kernel via a trusted path mechanism
• Objects – devices, volumes, virtualized re-sources, and
security kernel files
• Access control
– MAC - single-level virtual machines
– DAC (ACLs) on objects inside each VM
• BLP secrecy and Biba integrity
• Layered design
– Strict levels of abstraction
• Advanced software engineering
– Language the most critical issue: PL/I & Pascal

82
©Tatyana Ryutov
 VAX VMM: Layers
• Project was the commitment to a
layered implementation
– Almost pure sequence of single layers
• 16 layers with no upward dependencies
• Did not enforce “no op” calls through
intervening layers
• Testing emphasized layer entry points
• Team members were partitioned
throughout the layers

83
©Tatyana Ryutov
 VAX VMM: Kernel Assurance
• Principle reason for SK was assurance
– First Class A1 with performance and compatibility
• Formal methods – significant quality factor
– Formally specifying the security kernel detected several kernel bugs
– Separation of duties between the software engineer and the verifier
• Covert channel analysis and countermeasures
– TCSEC A1 requirement
– Many resource allocation mechanisms were static or human assisted
resource allocation
– Timing channels are hard (future lecture)
• Configuration control – rigorous and thorough
– Reviewed items were physically protected against modification
• Trusted distribution – security seal and crypto checksums
– HW and SW

©Tatyana Ryutov
 VAX VMM: Conclusions
• VAX VMM was a technical success
– Met all Class A1 security requirements
– Ran on commercial hardware (DEC VAX)
– Provided high degree of OS and apps compatibility
– Acceptable level of performance
– Met requirements of commercial software vendor
• This functional, high security system was discarded just as it
became ready for commercial deployment
• Management decision to cancel before done
– Worsening business climate for DEC
– Export controls would disrupt may potential sales
– Introduction of new devices would require considerable effort
• Rigorous reviews for any new code
– No GUI, lack of good networking

©Tatyana Ryutov
 What technology in Scomp, KSOS, and
VAX VMM are still in use today?
– Distant cousin of SCOMP: XTS Guard 5 (BAE systems)
• cross domain solution that enables secure sharing between networks
• EAL 4+
• [Link]
– MLS OS based on SCOMP
• runs on Intel x86 hardware
• conformance to the Linux API
• Secure Trusted Operating Program (STOP) [Link]
us/product/stop
• EAL 4+ FIPS 140-2 rated
– Ideas
• TCB is well-defined in rings, and protected by gates
– Intel x.86
» Rings implemented in hardware
» Gates
• I/O Device Drivers in Scomp can be run in user-space
– XEN split driver
• MLS
– SELinux, Solaris Trusted Extensions (EAL4+)
• Security kernel
– seL4
©Tatyana Ryutov
 Conclusions
• The experiences from developing Multics, SCOMP, and
KSOS substantially influenced the TSCEC content
• Unfortunately, significant advances in computer
security and SW methodologies have been mostly forgotten
• Developers rediscover the same old “new and novel”
solutions to the same old problems about every five years
• Solutions that can often be found in the older literature

87
©Tatyana Ryutov
 SK Construction Techniques Summary
• Virtualization and sharing
• Trusted path
• Trusted functions
• Security kernel policies
• Kernel implementation strategies
– New operating system
– Compatible operating system (emulation)
– Identical operating system (virtual machine)

88
©Tatyana Ryutov