UNIX Internals

Rohit Jnagal
[email protected]

Introduction
First Steps

Course Prerequisites
Prerequisites for the course:
C programming concepts and principles
Basic Computer Architecture concepts (refresher today!)
Basic algorithms and data structure concepts
Exposure to UNIX/Linux programming environment

Would be great if you also have:

Worked on a non-windows system
Paid attention in your earlier Operating System courses
Lots of Questions
Ability to stay awake through a 2 hours class

Operating System Refresher

Why an Operating System:
you dont want to know what goes into making pepperoni

Process & Resources (memory, disk, cpu, network)

Operating System Goals:
Create Resource Abstractions
Create Process Abstractions
Manage Resource Sharing
Process Isolation
Others ?

Operating System Refresher

Computer System Components:
Hardware
provides basic computing resources (CPU, Memory, I/O)
Operating System
controls and coordinates the use of hardware among various
application programs for various users.
Application Programs (Processes)
ways in which system used to solve problems of a user (compilers,
database systems, video games and making tea)
Users
(people, machines, other computers)

Operating Systems Refresher

Why learn Operating Systems:
To clear this course!
Easier to see how to use the system
To fine-tune performance of our workload
To add, modify or enhance a functionality in a transparent
way
To get new hardware up and running
To design OS-aware hardware and peripherals
OS is pure overload of real work. Applications are the ones
with real value.

Operating System History

Operating Systems through the ages:
Dark Ages ( 1945 55 )
Vacuum Tubes & Plugboards: the computing monstrosities
No OS
No Programming Languages
No Assembly Languages

Operating System History

Operating Systems through the ages:
1st Generation (1955 65 )
Mainframe Systems

Batching similar jobs

One job at a time
Automatic job sequencing
Resident Monitor
Initiate control in monitor
Control transfer to a job
When job completes, transfer back to
monitor

Severe I/O bottleneck

Operating System History

Operating Systems through the ages:
2nd Generation ( 1965 80 )
Integrated ICs Renaissance
Multi-programmed Batch Systems

Notion of computer architecture

Disk drives
On-line terminals
Several jobs in memory at the same
time
OS features

Job scheduling
Memory management ( Overlays)
CPU scheduling
Protection

Operating System History

Operating Systems through the ages:
2nd Generation (1965 80 )
Time-Sharing Systems
OS features:
Swapping
Virtual Memory
File System
CPU scheduling
Device Allocation
IPC
Interactive shell
Protection

Operating System History

A Long Time Ago, in a galaxy far, far away:
OS/360 Multiprogramming
MIT CTSS (Compatible Time Sharing)
MIT, Bell Labs, GE, MULTICS (Multiplexed Information and
Computing Service)
A game called Space Travel
And UNIX was born in 1969

Operating System History

Operating Systems through the ages:
3rd Generation (1980 - )
LSIs, VLSIs, Nano mobile an ant can use
Architecture Advances:
Smaller, faster Microprocessors
Caches
Larger Faster Storage
Offloading CPU work
Modern OS feature
GUI
Multimedia
Internet & Web
Networked, Distributed, etc

Operating System History

Current UNIX Systems:
HP HP-UX
HP ( Digital (Compaq) ) Tru64
Linux ( not exactly UNIX!)
FreeBSD, NetBSD
SUN Solaris
IBM AIX
Apple Mac OS X, etc
NSK (what about Non-Stop)

Operating System Classifications

Multi-user / Single user
Multiprogramming
Multi-threaded
User threads / thread libraries
System/Kernel threads
Monolithic Kernel / Micro-kernels
Functional Classification
Mainframe
Server
Desktop
Topographic Classification
Multiprocessor
Clustered Systems
Distributed Systems
Requirements/Constraints Classification
Embedded Systems
Real-Time Systems

UNIX Operating System

UNIX Structure
applications

Memory
Mgmt

MP
I/ O

Proc/
thread
mgmt

IPC

hardware
Misc...
LVM

File systems

shells

daemons

cmds

drivers

UNIX features
Files and Processes
File system has places ( / vs C:, D: )
Processes have life
Everything is a file
Processing environment
Hardware mode support
Interrupts and Exceptions
Processor Execution Levels
Process Management Subsystem
Memory Management Subsystem
Kernel/System Interfaces
So, whats different in a Non-UNIX system ?

Processor Modes
Supervisor or User Mode
Supervisor Mode
Can execute all instructions
Can reference all memory locations
Kernel execute in this mode

User Mode
Can execute subset of instructions
Can reference a subset of memory locations

To go from user to supervisor mode:

Interrupts/traps
Requests (system calls)

System Hierarchies
User program
User Level

Libraries

Systemcall interface

Kernel Level

Virtual File System

Phys File
System
Buffer Cache

LVM
device drivers
Hardware Level

lightweight system calls

Process Management
Subsystem

Memory
Management
Subsystem

IPC
Scheduling

I/ O
Hardware control

Primary Subsystems
Process Management
Manages all active processes and threads

Memory Management
Allocates memory for processes at run-time
Performs translations of physical memory locations to virtual memory
locations

File Subsystem
Virtual File System Layer
Abstracts and manages data for different file system types
Physical File System Layer
Manages access for specific file systems of various types

I/O Subsystem
Manages system I/O resources, drivers, and data structures
Drives system configuration process

Is my system UNIX or UNIX compliant ?

AT&T UNIX System V

Berkeley Software Distributions (BSD)
POSIX Standards
UNIX 2003 Standards
UNIX ABI
UNIX Kernel Interface / System Call Interface

HP-UX History
HPUX release History
Hewlett Packard UNIX
HPUX 1.0 in 1983
HPUX 10.0 (1995) : based on SVR4 UNIX standard
HPUX 11.0 (1997) : First 64-bit version
HPUX 11.11(2001) -> 11i series (World War 1 Armistice)
HPUX 11.20 (2002): First Itanium release
HPUX 11.23 11i v2 (2003): supports both PA and Itanium,
supports ccNUMA

HP-UX Specials
HP-UX features

First to use access control lists for file access permissions

First UNIX system with built in Volume Manager

Enhanced Reliability, security and partitioning

Clustering technology

Cell Based, ccNUMA architecture support

Virtual Server Environment (VSE)

Scaling
128 processors (expandable to 256)
1 Tb main memory
32 Tb maximum filesystem size
2 Tb maximum file size

Learning
check

Computer Architecture Refresher

Components:
CPU
Instruction Set Architecture
RISC Vs CISC
x86 CISC -> RISC +
micro-ops

IA32, IA64, PA-RISC, PowerPC, ARM

EPIC: Itanium

Pipelining
Fetch, Decode, Execute
Superpipelined, Superscalar

Registers
General Purpose Registers
Program Counters
Program Status Word (PSW)

Memory & Memory Hierarchy

Caches
RAM
Disks
Tapes
Peripherals

Computer Architecture Refresher

OS and Architecture :
Mutual Interaction
OS functionality limited by Architecture features
Multiprogramming on 8086

OS structure can be simplified with Architecture support

Interrupt, DMA, etc

Most proprietary OS are developed for a particular architecture

PA-RISC , IPF
x86
SPARC

Computer Architecture Refresher

Architecture support for OS:
Interrupt and Exceptions
How does the kernel notice an I/O has finished
Polling
Hardware Interrupt

Interrupts
Generated by Hardware Devices
Asynchronous

Exceptions
Generated by executing instructions
Synchronous
Trap (expected) or fault (unexpected)

Interrupt/Exception Handling

Computer Architecture Refresher

Architecture support for OS:
I/O Operations
Data Transfer Modes in I/O
Programmed I/O
DMA (Direct Memory Access)

Timers
How does OS take control of CPU from running programs?
Hardware timer
Timer Interrupt

Protected/Privileged Instructions
Direct I/O access
Memory State Management
Setting special mode bits
Halting a system

Computer Architecture Refresher

Architecture support for OS:
OS Protection
How does the processor know if a protected instruction should be
executed?
Rings or Modes of Operation
Access Rights

Crossing Protection Boundaries

User process call Kernel to do something privileged
OS defined sequence of System Calls
Modes, Contexts and Switches

Memory Protection
Protect user programs from each other
Protect itself from user programs
Memory Management Unit
Segmentation
Virtual Memory
Page table, Page protection, TLBs
Base and Limit Registers

Computer Architecture Refresher

Architecture support for OS:
Synchronization
Problems
Interrupt may occur at any time and may interfere with interrupted code
Synchronization of Concurrent Processes
Multi-processor synchronization

Solutions
Turn Off/On Interrupts
Use special atomic instructions
read-modify-write
test-and-set
Bus locks

Synchronization primitives spinlocks, semaphores, etc

Learning
check

System Calls
Methods to request system action on behalf of a user
Request
System Action
On behalf of a user
A Single program makes numerous system calls.
Program: Read from one file and write to another

System Calls
Passing Parameters
Three general methods:
Pass parameters in registers
Store them in memory and pass the address
Push the parameters on the stack
Usual rules apply: isolation, security, efficiency

System Calls
Types of System Calls
Process control
Create, execute, wait, end, abort, etc

File Management
Create, delete, open, close, read, write, copy, get/set attributes
ls, cat, more, grep etc.

Device Management
Read, write, attach, detach, get/set attributes

Information
Time, date, attributes

Communications
Send, receive, connect, accept, attributes

Making a System Call

Just use the stub!
Internally,
Stub pushes the syscall number on stack/register/u-area
Arguments go on stack/registers/u-area
A special trap instruction is called
Changes execution mode to kernel
Transfers control to syscall handler
using system call dispatch vector

Set return value and error status in register/locations

Restore hardware context
Returns to user mode and the stub

Learning
check