Reference Monitor

• Reference monitor is an
important part of security kernel.
• It controls access to objects
• It separates subjects and
objects. It enforces that a
subject can access only those
objects which are allowed by
security policy
• It is collection of access
controls for all objects (like
access control matrix)
• Reference monitor must be
• tamperproof, that is, impossible to weaken or disable,
• unbypassable, that is, reference monitor is always invoked when
access to any object is required
• analyzable, that is, small enough to be subjected to analysis and
testing.
Trusted Systems
• A trusted system will perform certain activities as per the
users’ expectations.
• A software is said to be a trusted software if the code has been
rigorously developed and analyzed, and it does what it is
expected to do and nothing more.
• An OS is said to be trusted OS, if it correctly controls the
accesses of components or systems run from it
• Trusted programs can be used to access sensitive data.
• No user can access sensitive data directly without using trusted
programs.
• A trusted program has the following key characteristics:
• Functional correctness. The program does what it is supposed
to, and it works correctly.
• Enforcement of integrity. Even if presented erroneous
commands or commands from unauthorized users, the
program maintains the correctness of the data with which it has
contact.
• Limited privilege. The program is allowed to access secure data,
but the access is minimized and neither the access rights nor
the data are passed along to other untrusted programs or back to
an untrusted caller.
• Appropriate confidence level. The program has been examined
and rated at a degree of trust appropriate for the kind of data and
environment in which it is to be used.
Trusted systems have three features:
• A security policy: defines a list of security qualities it enforces
• appropriate measures and mechanisms by which it can enforce
that security adequately
• independent scrutiny or evaluation to ensure that the
mechanisms have been selected and implemented properly
so that the security policy is in fact enforced.
Trusted System Functions
• Trusted Computing Base (TCB)
• TCB has everything necessary for a system to enforce its
security policy

• TCB consists of the parts of the trusted operating system that

enforces security policy.
TCB Functions:
System elements on which security enforcement could depend:
• hardware, including processors, memory, registers, a clock, and
I/O devices
• security-critical processes - which are separated and protected
• primitive files, such as the security access control database and
identification and authentication data
• protected memory, so that the reference monitor can be
protected against tampering
• some interprocess communication, so that different parts of the
TCB can pass data to and activate other parts;
What constitutes TCB?
TCB monitors four basic interactions:
• Process activation. In a multiprogramming environment, activation and
deactivation of processes occur frequently. Changing from one process to
another requires a complete change of registers, relocation maps, file access
lists, process status information, and other pointers, much of which is
security sensitive information.
• Execution domain switching. Processes running in one domain often invoke
processes in other domains to obtain more or less sensitive data or services.
• Memory protection. Because each domain includes code and data stored in
memory, the TCB must monitor memory references to ensure secrecy and
integrity for each domain.
• I/O operation. In some systems, software is involved with each character
transferred in an I/O operation. This software connects a user program in the
outermost domain to an I/O device in the innermost (hardware) domain.
Thus, I/O operations can cross all domains.
TCB Design:
• OS is logically divided into TCB and Non-TCB parts.
• Security related parts of OS are in TCB and rest of OS are in Non-
TCB
• TCB code must run in some protected state in order to protect it
from interference or compromise by any code outside the TCB
• code outside the TCB can be changed, without affecting the
TCB’s ability to enforce security, to upgrade OS
TCB Implementation

• separate the security

functions of an operating
system and create a security
kernel as shown fig.
• security kernel forms an
interface layer
• security kernel monitors all
operating system hardware
accesses and performs all
protection functions
Object Reuse.
• Object reuse protection
• The object or resource used by one user should be allotted to another
user only after completely cleaning the data of previous users. Else the
current user may access the data of the previous user. Eg., secondary
memory, main memory, processor registers and other reusable storage
medium disk, tape …
Note: Object sanitization ensures no leakage of data if a subject
uses a memory object released by another subject.
• Very precise and expensive equipment can sometimes separate
the most recent data from the data previously recorded, from the
data before that, and so forth. This threat, called magnetic
remanence.
• Trusted path
• Trusted path must be used for communicating all sensitive data;
otherwise the hacker may spoof the user’s communication
• Note: A trusted path is an unforgeable connection by which the user can
communicate directly with the operating system, not with any
fraudulent intermediate application. E.g., In windows CTRL+ALT+DEL
opens login screen
• Audit log
• Information security officer or sys admin needs to audit all log files
periodically to see whether any malicious or suspicious event took place.
• The log files must be protected from modification or deletion. It should be
read and append only. Appending will be done only by authorised
process. System should shutdown when there is no space in log file to
record the events.
Rootkit
• In Unix, root is the the most powerful/privileged subject /user /entity
/identity.
• Root
• owns sensitive/critical system resources such as memory
• performs powerful actions such as creating users (all users including admins),
killing processes (any type),
• runs all primitive system tasks.
• Root has complete and unrestricted control of system.
• Hacker’s ultimate goal is to gain the root privileges.
• Rootkit: a malware Tool/script/package that obtains privileges of root
Phone rootkit
• A team of researchers at Rutgers university demonstrated the
installation of following rootkits in mobile phones. None of
these rootkits were detectable by owner.
• A rootkit that could turn on a phone’s microphone without the owner
knowing it happened
• A rootkit that could turn on a phone’s camera without the knowledge of
owner.
• A rootkit that could turn on a phone’s GPS and communicate location of
phone without the knowledge of owner.
• a rootkit that could turn on power-hungry capabilities—such as the
Bluetooth radio and GPS receiver—to quickly drain the battery
• Rootkits are normally undetectable. If they are discovered and
removed, they have the capability of reinfecting the system
again.
• A rootkit can remain undiscovered if
• It is getting executed before execution of other programs that might
block rootkit’s execution and
• It should not be detected as a file or process.
• If rootkit is introduced(executed) early enough, it can override
other normal system functions that would detect its presence.
• Normally, a rootkit evades (escapes from) detection . It operates
unchecked.
• Example: SONY XCP rootkit
Rootkit evades detection. Example
• Let mal_code.exe be a malware
stored in c:/winnt/apps as shown in
fig.
• When you run a file explorer
program on that directory,
mal_code.exe will appear in the
listing, and you might recognize and
eradicate the file.
• The malware can hide its presence
by intercepting the listing/output of
file explorer and removing its entry
from there, so that the user could
not see the file name
“mal_code.exe” in the directory
listing
• Usually, antivirus does not contain code to
• query the disk,
• determine the disk format,
• identify files and where they are stored,
• find the file names and properties from an index table, etc
• Instead, antivirus calls built-in functions through an application
programming interface (API) to get this information.
• For example, as shown in
Figure 5-19,
• antivirus calls the Windows API
functions: FindFirstFile() and
FindNextFile().
• These functions in turn call NT
Kernel “native mode” system
functions, such as
NTQueryDirectoryObject.
• The NtQueryDirectoryObject
function retrieves information
about a specified directory
object.
• To remain invisible,
the rootkit intercepts
these calls as shown in
fig so that if the result
from FindNextFile()
points to mal_code.exe,
the rootkit skips that
file and executes
FindNextFile() again to
find the next file after
mal_code.exe.
• Since rootkit is integrated with the operating system, it can
perform any function the operating system can, usually without
being detectable. For example, it can
• replace other parts of the operating system,
• rewrite pointers to routines that handle interrupts, or
• remove programs (such as malicious code checkers –
antivirus/antispyware) from the list of code to be invoked at system
startup.
• These actions are in addition to more familiar malicious effects,
such as deleting files, sending sensitive data to remote
systems, and forwarding harmful code to email contacts.
• Since rootkit is loaded automatically as part of operating system,
it remains undiscovered and perform any action without any
constraints
Not every rootkit is malicious
• Rootkits that parents can install on children’s computers, to
monitor the nature of email, messaging, and web searches.
• Law enforcement authorities also install rootkits on machines of
suspects so that agents can trace and even control what users of
the affected machines do.