Enterprise Firewall

FortiOS Architecture

FortiOS 7.0
© Copyright Fortinet Inc. All rights reserved. LastLast
Modified:
Modified:
JulyJuly
26, 26,
20252025
Objectives
• Describe how FortiOS processes a packet
• Monitor process activity by using real-time debugs
• Describe how FortiOS uses memory
• Diagnose high memory and high CPU problems
• Diagnose conserve mode
• Optimize memory usage
• Troubleshoot unexpected reboots and frozen devices
• Use the crashlog for diagnostics
• Understand FortiOS workspace mode

© Fortinet Inc. All Rights Reserved. 2

Life of a Packet

3
Parallel Path Processing
• Parallel path processing (PPP) chooses from a group of parallel options to identify the
optimal path for processing a packet
• FortiGate can offload and accelerate many processes in hardware
• Content processors (CP8 or CP9) offload some UTM/NGFW processing and cryptographic operations
• Network processors (NP6 or NP7) offload traffic that does not require UTM or NGFW processing
• FortiGate hardware and software configuration affect the path that a packet takes

© Fortinet Inc. All Rights Reserved. 4

Life of a Packet–Initial Session Packets
Kernel
Admission
1. Forwarding
Network interface Quarantine control 2. Source NAT
CPU

IPsec VPN
FortiTelemetry encryption
NP6/7
ACL NP6/7

User
Host Protection Traffic shaping
authentication
Engine (HPE) (captive portal)

Kernel WAN
IP integrity 1. Destination NAT
optimization
header checking 2. Routing, RPF check, and
SD-WAN
3. Stateful inspection/policy
lookup/session management
IPsec VPN 4. Session helpers Network interface
decryption 5. User authentication
CPU
6. Device identification
7. SSL VPN
8. Local management traffic

UTM/NGFW
1. Flow-based inspection CP8/9
Optional/Configurable
2. Proxy-based inspection
3. Explicit web proxy CPU Mandatory
4. Botnet Check
© Fortinet Inc. All Rights Reserved. 5
Life of a Packet–Offloaded Non-UTM Sessions
Network interface

IP integrity
NP6/7
header checking
WAN
IPsec VPN optimization
decryption

Network interface
IPsec VPN
encryption

Traffic shaping

Optional/Configurable

Mandatory
© Fortinet Inc. All Rights Reserved. 6
Life of a Packet–NTurbo/Flow-based UTM Sessions
Network interface

IP integrity
NP6/7
header checking
IPS Engine WAN
CPU
optimization
IPsec VPN Flow-based
decryption UTM/NGFW

IPSA CP8/9
Network interface
IPsec VPN Single-pass rule
encryption matching

Traffic shaping

Optional/Configurable

Mandatory
© Fortinet Inc. All Rights Reserved. 7
Life of a Packet–Proxy-based UTM Sessions
Network interface

CPU
IPS Engine
Traffic shaping
Single-Pass IPS,
NP6/7
ACL Application Control,
Botnet, and SSL
CP8/9 Inspection WAN
optimization
IP integrity
header checking
CPU
Proxy
VoIP Inspection Network interface
IPsec VPN DLP
decryption Antispam
Web Filtering
Antivirus

IPsec VPN
NP6/7 encryption

Optional/Configurable

Mandatory
© Fortinet Inc. All Rights Reserved. 8
Memory Architecture

9
FortiOS Architecture
Configuration layer
CLI GUI API FortiManager

User space
Application Application Application Application
process process process process

Kernel
Device Device Device Device Device
driver driver driver driver driver

Hardware

© Fortinet Inc. All Rights Reserved. 10

FortiGate Memory Segmentation
• Kernel accesses the entire system memory directly
# diagnose hardware sysinfo memory
MemTotal: 3112828 kB
MemFree: 1680796 kB
Buffers: 506548 kB Free MemFree
Cached: 426892 kB
SwapCached: 0 kB
... MemTotal

Used memory

© Fortinet Inc. All Rights Reserved. 11

How the FortiGate Memory Is Used
• Kernel memory slabs
• System I/O cache
• Buffers
• Shared memory
• Process memory

© Fortinet Inc. All Rights Reserved. 12

Slabs
• Collection of objects with a common purpose and a fixed size
• Used by kernel
• Examples:

Slab Usage
tcp_session TCP session
ip_session Non-TCP session
ip_dst_cache Route cache
buffer_head Read/write data from disk, flash
inode_cache Information about files and directories
dentry_cache Cache for file system directory entries
arp_cache Cache for ARP

© Fortinet Inc. All Rights Reserved. 13

System I/O Cache
• Speeds up hard disk and flash disk writing and reading operations:
• Logging
• WAN optimization
• Explicit proxy
• Made of pages (4K size) of disk block (1K size)
• Two types of pages:
• Active
• Recently accessed
• Inactive
• Not used after some time
• Might be reclaimed by the kernel in case of shortage

© Fortinet Inc. All Rights Reserved. 14

System I/O Cache (Contd)
# diagnose hardware sysinfo memory
...
Cached: 1137808 kB
SwapCached: 0 kB
Active: 568600 kB
Inactive: 569208 kB
...

© Fortinet Inc. All Rights Reserved. 15

Shared Memory
• Allocated dynamically
• Allows the sharing of information among multiple processes

# diagnose hardware sysinfo shm

SHM FS total: 1001861120 955 MB
SHM FS free: 947621888 903 MB
SHM FS avail: 947621888 903 MB
SHM FS alloc: 54239232 51 MB

© Fortinet Inc. All Rights Reserved. 16

Process CPU Use
# diagnose sys top [refresh_time_sec] [number_of_lines]
Run Time: 1 days, 3 hours and 35 minutes
Press Shift+P to sort by
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 995T, 202F CPU usage. Press Shift+M
newcli 520 R 0.3 2.2 to sort by memory usage.
sshd 518 S 0.1 1.2
fsd 90 S 0.1 1.1
ipsengine 99 S 0.0 5.5
miglogd 41 S 0.0 4.8
pyfcgid 432 S 0.0 3.9
httpsd 102 S 0.0 3.3
pyfcgid 435 S 0.0 3.0
updated 75 S 0.0 2.4
cmdbsvr 25 S 0.0 2.3

Process ID State CPU Memory

© Fortinet Inc. All Rights Reserved. 17

Most Common Processes

Name Description
cmdbsrv Applies configuration changes
miglogd Logs collection and automation stitches
httpsd GUI access
sslvpnd SSL VPN
updated FortiGuard updates
wad WAN optimization, explicit proxy, proxy-based inspection for
HTTP and HTTPS, and FTP
scanunitd File scanning
iked IPsec

© Fortinet Inc. All Rights Reserved. 18

Most Common Processes (Contd)

Name Description
pppoed PPPoE protocol
hatalk, hasync HA protocol and synchronization
pptpd, l2tps PPTP and L2TP protocols
urlfilter FortiGuard web filtering
authd User authentication
fssod FSSO
proxyworker Proxy-based inspection for IMAP, POP, SMTP

© Fortinet Inc. All Rights Reserved. 19

Process States Review
• States:
• S: sleeping D
• R: running R
• D: do not disturb
• Z: zombie Z
• Normal states:
• S, R, and D (for a short time)
• Abnormal state:
• Z and D (if not for a short time)
S

© Fortinet Inc. All Rights Reserved. 20

General System Troubleshooting Commands

21
System Information
# get system status
Version: FortiGate-VM64 v7.0.1,build0157,210714 (GA)
Virus-DB: 87.00625(2021-07-14 14:20)
Extended DB: 87.00625(2021-07-14 14:20)
Extreme DB: 1.00000(2018-04-09 18:07)
AV AI/ML Model: 0.00000(2001-01-01 00:00)
IPS-DB: 6.00741(2015-12-01 02:30)
IPS-ETDB: 18.00120(2021-07-13 23:51)
APP-DB: 18.00118(2021-07-13 00:24)
INDUSTRIAL-DB: 18.00118(2021-07-13 00:24)
IPS Malicious URL Database: 3.00064(2021-07-08 07:49)
Serial-Number: FGVM010000077646
License Status: Valid
VM Resources: 1 CPU/1 allowed, 2007 MB RAM
Log hard disk: Available
Hostname: ISFW
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 0157
Release Version Information: GA
FortiOS x86-64: Yes
System time: Mon Jul 26 06:24:41 2021

© Fortinet Inc. All Rights Reserved. 22

Resource Use
# get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 2206648k total, 910028k used (41.2%), 1104748k free (50.1%), 191872k
freeable (8.7%)
Average network usage: 4 / 4 kbps in 1 minute, 6 / 6 kbps in 10 minutes, 16 / 20
kbps in 30 minutes
Average sessions: 22 sessions in 1 minute, 31 sessions in 10 minutes, 33 sessions in
30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per
second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 9 hours, 59 minutes

© Fortinet Inc. All Rights Reserved. 23

Real-Time Application Debug
• To enable most of the real-time debugs:
# diagnose debug application <application> <debug_level>
# diagnose debug enable
• Some applications (daemons) you can debug in real time:
• sslvpn SSL VPN
• ike IPsec VPN
• authd User authentication
• update FortiGuard updates
• Debug level:
• 0: Disable the specific debug
• Other values: outputs vary depending on the daemon
• -1: Enable all outputs

© Fortinet Inc. All Rights Reserved. 24

Real-Time Application Debug (Contd)
• Example, for IPsec real-time debug:
# diagnose debug application ike -1
# diagnose debug enable
• Enable timestamp:
# diagnose debug console timestamp enable
• Remember to disable the debug after troubleshooting:
# diagnose debug application ike 0
# diagnose debug disable
• Disable all application debugging:
# diagnose debug reset

© Fortinet Inc. All Rights Reserved. 25

Application Layer Test Commands
# diagnose test application ?
mm17 MM1/MM7 proxy.
smtp SMTP proxy.
ftpd FTP proxy.
pop3 POP3 proxy.
imap IMAP proxy.
nntp NNTP proxy.
forticldd FortiCloud daemon.
miglogd Miglog logging daemon.
urlfilter URL filter daemon.
ipsmonitor ips monitor
ipsengine ips sensor
ipldbd IP load balancing daemon.
....

© Fortinet Inc. All Rights Reserved. 26

Conserve Mode

27
Conserve Mode
• Triggered based on memory use
• Prevents using so much memory that FortiGate becomes unresponsive
• FortiGate leaves conserve mode as memory use goes below set threshold
• Three memory thresholds that you can configure on the CLI
• Extreme: threshold at which FortiGate starts dropping new sessions
• Red: threshold at which FortiGate enters conserve mode
• Green: threshold at which FortiGate exits conserve mode

© Fortinet Inc. All Rights Reserved. 28

Conserve Mode Thresholds
# config system global
Default
... threshold values
set memory-use-threshold-extreme 95
set memory-use-threshold-red 88
set memory-use-threshold-green 82
...
end

© Fortinet Inc. All Rights Reserved. 29

 Conserve Mode Logs

Log & Report > Events > System Events

• Crash log:
# diagnose debug crashlog read
...
2020-04-28 11:12:59 logdesc="Memory conserve mode
entered" service=kernel conserve=on total="1234 MB"
used="878 MB" red="876 MB" green="864 MB" msg="Kernel
enters memory conserve mode"
...

© Fortinet Inc. All Rights Reserved. 30

Proxy Inspection While in Conserve Mode
• Antivirus failopen governs FortiGate behavior for proxy-based inspection while in
conserve mode
config system global
set av-failopen {off | one-shot | pass}
set av-failopen-session {enable | disable}
end
• av-failopen-session - Enable or disable failopen
• Default setting is disable
• av-failopen - Configure how sessions failopen
• off – All new sessions that require content inspection are dropped, but existing sessions are still
processed
• pass – Stops inspecting new sessions. Inspection is automatically restarted when FortiGate exits
conserve mode
• one-shot – Similar to pass, but you must manually change the av-failopen setting to restart
inspection after FortiGate exits conserve mode

© Fortinet Inc. All Rights Reserved. 31

Flow Inspection While in Conserve Mode
• IPS failopen governs FortiGate behavior for flow-based inspection while in conserve
mode
config ips global
set fail-open {enable | disable}
end

© Fortinet Inc. All Rights Reserved. 32

Conserve Mode Diagnostics
# diagnose hardware sysinfo conserve

memory conserve mode: off

total RAM: 3039 MB
memory used: 817 MB 26% of total RAM
memory freeable: 582 MB 19% of total RAM
memory used + freeable threshold extreme: 2887 MB 95% of total RAM
memory used threshold red: 2675 MB 88% of total RAM
memory used threshold green: 2492 MB 82% of total RAM

© Fortinet Inc. All Rights Reserved. 33

Memory Tension Drops
• Kernel deletes oldest sessions if it cannot allocate more memory pages
• No direct link with conserve mode

# diagnose sys session stat

misc info: session_count=184 setup_rate=0 exp_count=0 clash=0
memory_tension_drop=0 ephemeral=0/196608 removeable=0
npu_session_count=61
nturbo_session_count=0
delete=0, flush=87, dev_down=16/120 ses_walkers=0
TCP sessions:
38 in ESTABLISHED state
1 in CLOSE_WAIT state

© Fortinet Inc. All Rights Reserved. 34

Ephemeral Drops
• A session is categorized as ephemeral when one of the following is true:
• A TCP session is not fully established
• A UDP with only a single packet is received
• These types of open sessions are common types of DoS attacks
• To protect memory use, FortiOS sets a limit on the total number of ephemeral sessions
(based on the model)
# diagnose sys session stat
misc info: session_count=184 setup_rate=0 exp_count=0 clash=0
memory_tension_drop=0 ephemeral=0/196608 removeable=0
npu_session_count=61
nturbo_session_count=0
delete=0, flush=87, dev_down=16/120 ses_walkers=0
TCP sessions:
38 in ESTABLISHED state
1 in CLOSE_WAIT state

© Fortinet Inc. All Rights Reserved. 35

Memory Use Optimization

36
Memory Use Optimization
• Disable features that are not required:
• Inspection of specific protocols (HTTP, FTP, SMTP, POP, IMAP)
• Logging to memory
• DHCP server
• Some IPS signatures
• Reduce the maximum file size to inspect (default 10 MB):
config firewall profile-protocol-options
edit <profile_name>
config [http|ftp|pop3|smtp|imap]…
set oversize-limit <MB>
end

© Fortinet Inc. All Rights Reserved. 37

Memory Use Optimization (Contd)
• Reduce the FortiGuard cache TTL (default 3600 and 1800 seconds):
config system fortiguard
set webfilter-cache-ttl 500
set antispam-cache-ttl 500
end

• Reduce DNS cache (default 1800 seconds):

config system dns
set dns-cache-ttl 300
end

© Fortinet Inc. All Rights Reserved. 38

Memory Use Optimization (Contd)
• Reduce the session time to live (TTL) • For each service:
• Globally: config system session-ttl
• For TCP (default to 3600 seconds): config port
config system session-ttl edit <id>
set default 300 set protocol <IP_protocol>
• For UDP (default to 180 seconds): set start-port <start_port>
config system global set end-port <end_port>
set udp-idle-timer 90 set timeout 300

© Fortinet Inc. All Rights Reserved. 39

Memory Use Optimization (Contd)
• Reduce the session TTL (default 3600 seconds)
• For each firewall policy:
config firewall policy
edit <id> Security Profiles > Application Control
set session-ttl 300
• Per application control

Configurable
on the CLI

Policy & Objects > Firewall Policy

© Fortinet Inc. All Rights Reserved. 40

Memory Use Optimization (Contd)
• Reduce TCP session timers:
config system global
set tcp-halfclose-timer 30 (default 120)
set tcp-halfopen-timer 8 (default 10)
set tcp-timewait-timer 1 (default 1)
end

SYN
tcp-halfopen-timer
SYN / ACK

ACK

FIN
tcp-halfclose-timer
FIN / ACK

tcp-timewait-timer

© Fortinet Inc. All Rights Reserved. 41

Workspace Mode

42
Workspace Mode
• Start workspace mode:
Start workspace transaction
• execute config-transaction start
• Configuration changes are made in a local
CLI process that is not viewable by other
processes
Make FortiOS configuration
• Abort configuration changes: changes
• execute config-transaction abort
• If changes are aborted, no changes are
made to the current configuration
Revert/edit FortiOS configuration
• Commit configuration changes: changes

• execute config-transaction commit

• After performing the commit, the changes
are available for all other processes and the Commit/abort workspace
kernel transaction

© Fortinet Inc. All Rights Reserved. 43

Diagnosing Workspace Mode
# diagnose sys config-transaction status
The CLI is running config transaction (id=1)
Transaction ID
Administrator

# diagnose sys config-transaction show txn-info

txn_id=1, expire=12 seconds, user='admin', userfrom='ssh(10.1.10.1)',
clicmd_fpath='/dev/cmdb/txn/4_Ede9G.conf
Changes are aborted if
config transaction id=1 will expire in 10 seconds they are not committed
config transaction id=1 has expired before the transaction
expires
# diagnose sys config-transaction show txn-cli-commands
config system global
set hostname "NewHostname"
end Changes pending to be
committed

© Fortinet Inc. All Rights Reserved. 44

Troubleshooting System Crashes

45
Console Logging
• Available only on some models
• Records console CLI output in a 4 MB log file on flash memory
• Useful for troubleshooting unexpected restarts and unresponsive devices
• Can be displayed on the CLI or downloaded from the GUI
• To enable or disable console logging (disabled by default):
# diagnose debug comlog < enable | disable >
• To read console logging:
# diagnose debug comlog read
• To clear console logging:
# diagnose debug comlog clear
• To display the console logging settings:
# diagnose debug comlog info

© Fortinet Inc. All Rights Reserved. 46

Troubleshooting Unexplained Restarts
• A crash dump message is usually generated through the console
• After an unexpected restart, check:
• Logs
• Console logs (available in some models)
• Crash log
• If the model does not support console logs, keep a laptop connected to the console port
and capture the crash dump message

© Fortinet Inc. All Rights Reserved. 47

Troubleshooting a Device That Freezes
• Keep a laptop connected to the console port

• In multi-CPU platforms, enable NMI watchdog:

# diagnose sys nmi-watchdog enable
• (Crashes the system if it has not scheduled any daemon in 10 minutes)

• After the device freezes, push the NMI button while the laptop is connected to generate
the crash dump
• Not all FortiGate models have an NMI button

© Fortinet Inc. All Rights Reserved. 48

Crashlog
# diagnose debug crashlog read
21:31:52 <03689> firmware FortiGate-VM64 v7.0.1,build0157b0157,210714(GA) (Release)
21:31:52 <03689> application sslvpnd
21:31:52 <03689> *** signal 11 (Segmentation fault) received *** Application name
21:31:52 <03689> Register dump:
21:31:52 <03689> RAX: fffffffffffffffc RBX: 0000000000da4850
21:31:52 <03689> RCX: ffffffffffffffff RDX: 0000000000000400
21:31:52 <03689> R08: 0000000000000000 R09: 0000000000000008 Termination signal
21:31:52 <03689> R10: 00000000000007d0 R11: 0000000000003246
21:31:52 <03689> R12: 000000000bd38fd0 R13: 0000000000000000
21:31:52 <03689> R14: 00007ffff7bc6420 R15: 0000000000000000
21:31:52 <03689> RSI: 000000000bdeb370 RDI: 000000000000000a
21:31:52 <03689> RBP: 00007ffff7bc6070 RSP: 00007ffff7bc6038
21:31:52 <03689> RIP: 00007fde3bbffde0 EFLAGS: 0000000000003246
21:31:52 <03689> CS: 0033 FS: 0000 GS: 0000
21:31:52 <03689> Trap: 0000000000000000 Error: 0000000000000000
21:31:52 <03689> OldMask: 0000000000000000
21:31:52 <03689> CR2: 0000000000000000
21:31:52 <03689> stack: 0x7ffff7bc6038 - 0x7ffff7bc7430
21:31:53 the killed daemon is /bin/sslvpnd: status=0x0

© Fortinet Inc. All Rights Reserved. 49

Termination Signals
• Any time a process closes, a crash log is generated
# diagnose sys kill <termination_signal> <process_id>

Signal number Description

4 Illegal instruction
6 Abort command from FortiOS
7 Bus error
9 Unconditional kill
11 Invalid memory reference
14 Alarm clock
15 Graceful kill

© Fortinet Inc. All Rights Reserved. 50

Crashlog Tips
• In most cases, entries in the crashlog are normal
• Consider a crashlog to be suspicious when:
• It happens at the same time as an abnormal FortiGate behavior
• For example, unexpected system restarts
• The crashed process is related to the FortiGate feature that failed
• For example a crash in the sslvpnd process when all SSL VPN connections went down

• The crashlog can provide information to Fortinet developers about the crash cause

© Fortinet Inc. All Rights Reserved. 51

Review
 Understand the life of a packet
 Review general system troubleshooting commands
 Enable real-time debugs
 Examine how FortiOS uses memory
 Review the most common FortiOS processes and process states
 Identify memory conserve mode
 Troubleshoot unexpected reboots and frozen devices
 Optimize memory use and examine the crash log
 Understand FortiOS workspace mode

© Fortinet Inc. All Rights Reserved. 52

Lab 2—FortiOS Architecture

53
Lab 2—FortiOS Architecture
• Run debug commands to gather information
about resource utilization on ISFW
• Check the crashlog

© Fortinet Inc. All Rights Reserved. 54