Intrusion Detection By Analyzing Flows

S e c t i o n 0 2 | M o d u l e 0 3
© Caendra Inc. 2019
All Rights Reserved
Table of Contents

Module 03 | Intrusion Detection By Analyzing Flows

3.1 Network Flows: Definition, Strengths & Limitations

3.2 Network Flow Analysis Toolkit

3.3 Practical Flow Analysis

IHRPv1 - Caendra Inc. © 2019 | p.2

Learning Objectives

By the end of this module, you should have a better

understanding of:

✓ Network flow tracking, including its strengths,

limitations and tools
✓ How to analyze network flows and leverage them for
situational awareness
✓ How to enrich network flows with other data sources

IHRPv1 - Caendra Inc. © 2019 | p.3

 3.1

Network Flows: Definition,

Strengths & Limitations

IHRPv1 - Caendra Inc. © 2019 | p.4

3.1 Network Flows: Definition, Strengths & Limitations

After an incident has occurred, the faster we identify the

attackers and the compromised assets, the quicker we will
be able to proceed to the containment and eradication
phases.

Collecting evidence inside a heterogeneous network and/or

analyzing every captured packet (if there are any) is not
only demanding, but time consuming as well.

IHRPv1 - Caendra Inc. © 2019 | p.5

3.1 Network Flows: Definition, Strengths & Limitations

Fortunately, most networking devices nowadays are able to

track and export network flows. Well-orchestrated network
flow tracking can provide incident responders with a bird’s-
eye view of all the communications that took (or currently
take) place.

Such a view can help incident responders discover not only

the attacking as well as the compromised machines but
also any abnormal network behavior.

IHRPv1 - Caendra Inc. © 2019 | p.6

3.1.1 Network Flows: Definition & NetFlow Overview

But what are network flows exactly?

A network flow is essentially a record of a communication

between two hosts. A network flow usually contains the source
IP address, the destination IP address and the TCP/UDP ports
used during a “conversation”. Other information, such as the
router or switch interface where the flow was spotted, the
number of bytes transferred etc. can also be contained in a
network flow, as you will see in just a bit.

IHRPv1 - Caendra Inc. © 2019 | p.7

3.1.1 Network Flows: Definition & NetFlow Overview

When it comes to network flow tracking, Cisco’s

NetFlow technology is the most commonly used
one, but other vendors have also approached the
network flow tracking topic by adopting jFlow,
cFlow, sFlow and IPFIX, which are similar in
terms of the provided functionality.

NetFlow reports on traffic statistics, such as

sender, receiver, packets, and bytes. It doesn’t
report the actual protocol payload though.

On your right, you can see the different NetFlow

versions.

http://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3

https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.html IHRPv1 - Caendra Inc. © 2019 | p.8

3.1.1 Network Flows: Definition & NetFlow Overview

Most firewalls, routers and switches can export NetFlow

logs to a collecting server.

As already discussed, unlike packet capture, with NetFlow

you can’t ensure that, for example, the traffic spotted over
port 80 is actually HTTP traffic, since it has no visibility into
the actual protocol payload.

IHRPv1 - Caendra Inc. © 2019 | p.9

3.1.1 Network Flows: Definition & NetFlow Overview

Some attributes NetFlow can export are:

• IP source address
• IP destination address
• Source port
• Destination port
• Layer 3 protocol type
• Class of service
• Router or switch interface
IHRPv1 - Caendra Inc. © 2019 | p.10
3.1.1 Network Flows: Definition & NetFlow Overview

NetFlow V5 is the most commonly used version and is

supported by numerous different router manufacturers.

The number of fields NetFlow V5 can export are limited

though.

IHRPv1 - Caendra Inc. © 2019 | p.11

3.1.1 Network Flows: Definition & NetFlow Overview

NetFlow V9, on the other hand, is a template-based flow

with no restriction on the number of fields to be exported.

Unlike V5 where the headers are hardcoded, V9 offers a

dynamic header. In order for the collector to decode the
datagrams, it must receive a template from the NetFlow
exporter which defines the headers. The collector cannot
decode the datagrams until the corresponding template is
received.

IHRPv1 - Caendra Inc. © 2019 | p.12

3.1.1 Network Flows: Definition & NetFlow Overview

IPFIX is an open-source implementation based on NetFlow

Version 9 and was created to meet the need for a universal
standard to export IP flow information from network devices.

It is on the IETF standards, and it is implemented by multiple

vendors.

Specifications of IPFIX are documented in RFC 7011, RFC 7015,

and RFC 5103.

IHRPv1 - Caendra Inc. © 2019 | p.13

3.1.1 Network Flows: Definition & NetFlow Overview

A NetFlow setup for monitoring usually consists of 3

components:

• Exporter: Router, Switch, Firewall, … etc.

• Collector: Software for receiving and storing NetFlow

• Analysis: Software for analyzing the flow

IHRPv1 - Caendra Inc. © 2019 | p.14
3.1.1 Network Flows: Definition & NetFlow Overview

NetFlow exports data in UDP datagrams.

The NetFlow RFC 3954 does not specify a NetFlow listening

port; however, Wireshark and most other solutions assume
NetFlow on port 2055 and IPFIX on port 4739.

IHRPv1 - Caendra Inc. © 2019 | p.15

3.1.1 Network Flows: Definition & NetFlow Overview

IPv4: What NetFlow is mostly interested in

Bit offset 0-3 4-7 8-13 14-15 16-18 19-31
Internet Explicit
Differentiated Services
0 Version Header Congestion Total Length
Code Point
Length Notification
Fragment
32 Identification Flags
Offset
64 Time to Live Protocol Header Checksum
96 Source IP Address
128 Destination IP address
160 Options
160 or
Data
192+

http://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3 IHRPv1 - Caendra Inc. © 2019 | p.16

3.1.1 Network Flows: Definition & NetFlow Overview

TCP: What NetFlow is mostly interested in

Octet 0 1 2 3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
Bit 0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
0 Source port Destination port

32 Sequence number

64 Acknowledgment number (if ACK is set)

C E U A P R S F
Data Reserve N
96 W C R C S S Y I Window size
offset d S
R E G K H T N N
128 Checksum Urgent pointer (if URG is set)
160
Options
…

http://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3 IHRPv1 - Caendra Inc. © 2019 | p.17

3.1.1 Network Flows: Definition & NetFlow Overview

UDP: What NetFlow is mostly interested in

Bit Offset 0-15 16-31

0 Source port Destination port
32 Length Checksum
64
Data
…

http://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3 IHRPv1 - Caendra Inc. © 2019 | p.18

3.1.1 Network Flows: Definition & NetFlow Overview

Network flows can be unidirectional or bidirectional, in

terms of representation. Suppose that the below
communication occurred. Let’s see this “conversation”
through the lens of network flows.

IHRPv1 - Caendra Inc. © 2019 | p.19

3.1.1 Network Flows: Definition & NetFlow Overview

Unidirectional Flow

Active Flows
Source Destinati
Flow Prot. srcPort dstPort
IP on IP
1 10.0.0.1 10.0.0.2 TCP 32000 23
2 10.0.0.2 10.0.0.1 TCP 23 32000
3 10.0.0.1 10.0.0.2 ICMP 0 0
4 10.0.0.2 10.0.0.1 ICMP 0 0

IHRPv1 - Caendra Inc. © 2019 | p.20

3.1.1 Network Flows: Definition & NetFlow Overview

Bidirectional Flow

Active Flows
Source Destinati
Flow Prot. srcPort dstPort
IP on IP
1 10.0.0.1 10.0.0.2 TCP 32000 23
2 10.0.0.1 10.0.0.2 ICMP 0 0

IHRPv1 - Caendra Inc. © 2019 | p.21

3.1.2 Network Flows: Strengths & Limitations

Let us now summarize the strengths and limitations of

network flow tracking, before we start covering the network
flow analysis toolkit.

IHRPv1 - Caendra Inc. © 2019 | p.22

3.1.2 Network Flows: Strengths & Limitations

Network Flow Tracking Strengths Network Flow Tracking Limitations

1. Capturing communication 1. Not a substitute of full packet

information in a storage-effective capture, since it does not contain
and time-effective manner the content of each message that
2. Still useful even in encrypted was sent
communications (when, where, 2. Analyzing network flows doesn’t
how much and at what time always pay dividends
information are still available) 3. Selecting/deploying the appropriate
3. Useful to baseline an entire flow collector needs careful
environment planning
4. Flows can be obtained from
multiple places within the network

IHRPv1 - Caendra Inc. © 2019 | p.23

 3.2

Network Flow
Analysis Toolkit

IHRPv1 - Caendra Inc. © 2019 | p.24

3.2 Network Flow Analysis Toolkit

When it comes to network flow analysis, three (3) tools

deserve our attention.

1. YAF
2. SiLK
3. FlowViewer

https://tools.netsa.cert.org/yaf/
https://tools.netsa.cert.org/silk/
https://ensight.eos.nasa.gov/FlowViewer/ IHRPv1 - Caendra Inc. © 2019 | p.25
3.2.1 Network Flow Analysis Toolkit: YAF

The first tool we will cover from the network flow analysis
arsenal is YAF.

YAF is Yet Another Flowmeter. According to its creators, YAF

processes packet data deriving from a PCAP traffic capture file or
a live capture (from an interface using PCAP) into bidirectional
flows and then exports those flows to IPFIX Collecting Processes
or in an IPFIX-based file format. YAF's output can be used with the
SiLK flow analysis tools, super_mediator, Pipeline 5, and any other
IPFIX compliant toolchain.

https://tools.netsa.cert.org/yaf/
https://tools.netsa.cert.org/silk/
IHRPv1 - Caendra Inc. © 2019 | p.26
3.2.1 Network Flow Analysis Toolkit: YAF

YAF’s optional features include:

• Application labeling
o Rules located in /usr/local/etc/yafApplabelRules.conf
• OS detection (supports passive OS fingerprinting via libP0f and DHCP)
o DHCP fingerprints located in /usr/local/etc/dhcp_fingerprints.conf
o Output viewed with yaf-file-mediator
• Deep packet inspection
o Enabled by specifying, --plugin-name=/usr/local/lib/yaf/dpacketplugin.la
o You can also specify a protocol to perform DPI. --plugin-opts="53 80 21"
o DPI rule configuration is located in /usr/local/etc/yafDPIRules.conf
o Output viewed with yaf-file-mediator

https://tools.netsa.cert.org/yaf/applabel.html
http://tools.netsa.cert.org/yaf/yafdhcp.html
https://tools.netsa.cert.org/confluence/display/tt/YAF+2.x+IPFIX+File+Mediator |
https://tools.netsa.cert.org/yaf/yafdpi.html IHRPv1 - Caendra Inc. © 2019
p.27
3.2.1 Network Flow Analysis Toolkit: YAF

PCAP -> IPFIX with YAF

>> yaf --in filename.pcap --out filename.yaf

• If you want application labeling add the below

o --applabel-rules= /usr/local/etc/yafApplabelRules.conf --max-payload 300
• If you want OS detection (via P0f) add the below
o --p0fprint --p0f-fingerprints /usr/local/etc/ --max-payload 300
• If you want OS detection (via DHCP fingerprints) add the below
o --plugin-name=/usr/local/lib/yaf/dhcp_fp_plugin.la
• To perform DPI on top on the conversion add the below
o --plugin-name=/usr/local/lib/yaf/dpacketplugin.la

IHRPv1 - Caendra Inc. © 2019 | p.28

3.2.2 Network Flow Analysis Toolkit: SiLK

The definitive tool when it comes to network flow analysis

is SiLK. According to its creators, the SiLK tool suite
supports the efficient collection, storage, and analysis of
network flow data, enabling network security analysts to
rapidly query large historical traffic data sets.

https://tools.netsa.cert.org/silk/ IHRPv1 - Caendra Inc. © 2019 | p.29

3.2.2 Network Flow Analysis Toolkit: SiLK

Just like Linux commands, SiLK commands are piped to

form a workflow. For effectively using SiLK, one should
become familiar with crawling with rwcut and walking
through flow files with rwcut and rwfilter.

https://tools.netsa.cert.org/silk/rwcut.html
https://tools.netsa.cert.org/silk/rwfilter.html
IHRPv1 - Caendra Inc. © 2019 | p.30
3.2.2 Network Flow Analysis Toolkit: SiLK

Let’s now cover some important SiLK commands starting with

rwfileinfo, that provides metadata information about a SiLK file.
>> rwfileinfo filename.rwf

https://tools.netsa.cert.org/silk/rwfileinfo.html IHRPv1 - Caendra Inc. © 2019 | p.31

3.2.2 Network Flow Analysis Toolkit: SiLK

You can view flow records in clear text with rwcut, as follows.
>> rwcut --num-rec=10 filename.rwf
>> rwcut --num-rec=10 --
fields=sip,dip,proto,sport,dport,stime filename.rwf

IHRPv1 - Caendra Inc. © 2019 | p.32

3.2.2 Network Flow Analysis Toolkit: SiLK

You can count how much traffic matched specific keys as well as what
protocols were running, as follows. (The two commands are identical)
>> rwtotal --proto --skip-zero filename.rwf
>> rwuniq --field=proto --values=records,bytes,packets --
sort-output filename.rwf

IHRPv1 - Caendra Inc. © 2019 | p.33

3.2.2 Network Flow Analysis Toolkit: SiLK

If you want to identify the most common servers and source

ports with at least 50 flows, you can do so as follows.
>> rwfilter --proto=6,17 --pass=stdout filename.rwf |
rwuniq --field=sip,sport --flows=50 --bytes --values=dip-
distinct| sort -nr -k 3,3 -t '|' | head

IHRPv1 - Caendra Inc. © 2019 | p.34

3.2.2 Network Flow Analysis Toolkit: SiLK

If you want to see how the distribution of bytes, packets and

bytes/packet looks like, you can do so with rwstats as follows.

>> rwstats --overall-stats filename.rwf

https://tools.netsa.cert.org/silk/rwstats.html IHRPv1 - Caendra Inc. © 2019 | p.35

3.2.2 Network Flow Analysis Toolkit: SiLK

If you want to identify the top 10 destination ports, you can do so

with rwstats as follows.

>> rwstats --fields=dport --count=10 filename.rwf

IHRPv1 - Caendra Inc. © 2019 | p.36

3.2.2 Network Flow Analysis Toolkit: SiLK

rwcount is useful for examining traffic across time. Consider the

--load-scheme switch during your investigations.

>> rwcount --bin-size=300 filename.rwf | more

https://tools.netsa.cert.org/silk/rwcount.html IHRPv1 - Caendra Inc. © 2019 | p.37

3.2.2 Network Flow Analysis Toolkit: SiLK

rwfilter is SiLK’s go to command for filtering, since there are

switches for every flow attribute. For example, to identify top
webservers you can execute the below.
>> rwfilter filename.rwf --sport=80,443,8080 --protocol=6 -
-packets=4- --ack-flag=1 --pass=stdout | rwstats --
fields=sip --percentage=1 --bytes

IHRPv1 - Caendra Inc. © 2019 | p.38

3.2.2 Network Flow Analysis Toolkit: SiLK

Scanning activity can be detected within a SiLK dataset, through

the rwscan command.

>> rwsort --fields=sip,proto,dip filename.rwf |rwscan --

scan-model=2 | more

https://tools.netsa.cert.org/silk/rwscan.html IHRPv1 - Caendra Inc. © 2019 | p.39

3.2.2 Network Flow Analysis Toolkit: SiLK

rwfilter can also detect scanning attempts. For example, one

could specify the following criteria: i) size less than 2048 bytes,
ii) 1 to 3 packets and iii) no RST or FIN flags.
>> rwfilter filename.rwf --bytes=0-2048 --packets=1-3 --
flags-all=/RF --pass=stdout|rwuniq --fields=sip --
values=dip-distinct,records| sort -k 3,3 -n -r -t '|'| head
-n 30

IHRPv1 - Caendra Inc. © 2019 | p.40

3.2.2 Network Flow Analysis Toolkit: SiLK

SiLK has a powerful feature called IP Sets. An IP set is a

binary representation of an arbitrary collection of IP
addresses. IP sets showcase their true potential when used
in conjunction with rwfilter.

IHRPv1 - Caendra Inc. © 2019 | p.41

3.2.2 Network Flow Analysis Toolkit: SiLK

That being said, one may want to associate a value with

each address in an IP set. For example, one may want to
associate the IP addresses that engage in web traffic with
the volume of flows, packets or bytes of web traffic that
each address carries. This can be done through another
powerful SiLK feature IP Bags.

Bags are essentially extended sets.

IHRPv1 - Caendra Inc. © 2019 | p.42

3.2.2 Network Flow Analysis Toolkit: SiLK

Let’s take for example the command flow below that can
detect a scanning attempt.
>> rwsort --fields=sip,proto,dip filename.rwf |rwscan --
scan-model=2 | more

One can turn the result of the above into an IP Bag, as follows.
>>rwsort --fields=sip,proto,dip filename.rwf |rwscan --
scan-model=2 --no-titles | cut -d '|' -f 1,5 | rwbagbuild -
-bag-input=stdin > rwscan-output.bag

IHRPv1 - Caendra Inc. © 2019 | p.43

3.2.2 Network Flow Analysis Toolkit: SiLK

The output can be viewed, as follows.

>> rwbagcat rwscan-output.bag | sort -t '|' -k 2,2 -rn |

head

IHRPv1 - Caendra Inc. © 2019 | p.44

3.2.2 Network Flow Analysis Toolkit: SiLK

We barely scratched the surface of SiLK’s capabilities. SiLK

is an extremely capable and customizable tool. You can
learn much more about it by referring to the below
resources.
▪ https://schd.ws/hosted_files/flocon2017/fa/flocon-
2016-silk-tutorial.pptx
▪ https://tools.netsa.cert.org/silk/analysis-handbook.pdf
▪ RVAsec: Jason Smith - Applied Detection and Analysis
Using Flow Data

https://www.youtube.com/watch?v=ndfcfHiszHY IHRPv1 - Caendra Inc. © 2019 | p.45

3.2.3 Network Flow Analysis Toolkit: FlowViewer

FlowViewer is a NetFlow analyzer. It is actually a front-end

for flow-tools. FlowViewer is great for reporting on
historical data.

https://ensight.eos.nasa.gov/FlowViewer/
https://github.com/adsr/flow-tools
IHRPv1 - Caendra Inc. © 2019 | p.46
3.2.3 Network Flow Analysis Toolkit: FlowViewer

FlowViewer reports are quite clear and straightforward. See

an example below.
In this case, we have
applied a filter for TCP
flags = 27.
NetFlow records
contain a field
reporting the
cumulative OR-ed TCP
flags seen on the flow.
If we “sum” (using OR)
all the flags involved in
a TCP connection (SYN
[2] + ACK [16] + PSH [8]
+ FIN [1]) we have 27.

IHRPv1 - Caendra Inc. © 2019 | p.47

 3.3

Practical Flow
Analysis

IHRPv1 - Caendra Inc. © 2019 | p.48

3.3 Practical Flow Analysis

Let’s now leverage network flows to detect some real-world

attacks. Whenever we are provided with additional data, we
will use them to enrich the network flows.

IHRPv1 - Caendra Inc. © 2019 | p.49

3.3.1 Practical Flow Analysis: Case 1

Case 1: Consider the provided 2015-03-09-traffic-analysis-

exercise.pcap file. Try to identify if there is anything
suspicious going on by analyzing network flows only.

Hint: Load the provided PCAP into CapLoader.

https://www.netresec.com/?page=CapLoader#trial IHRPv1 - Caendra Inc. © 2019 | p.50

 3.3.1 Practical Flow Analysis: Case 1
Detection
CapLoader provides incident responders with the capability to quickly
aggregate and view all the network flows found in a PCAP file. Specifically, each
row inside the Services tab represents a unique combination of Client-IP,
Server-IP, Server-port and Transport-protocol. Note that one row can be multiple
flows merged together.

IHRPv1 - Caendra Inc. © 2019 | p.51

 3.3.1 Practical Flow Analysis: Case 1
Detection

CapLoader also features periodic flow detection. This means that one can
detect malicious traffic based not on blacklists or IDS signatures but based on
the periodicity of flows. You can see the above in action, by sorting the rows
inside the Services tab based on the Periodicity column.

IHRPv1 - Caendra Inc. © 2019 | p.52

 3.3.1 Practical Flow Analysis: Case 1
Detection

Any Periodicity value greater than 20 can be considered periodic. To analyze the
two most periodic entries, you can select both of them, then right click on them
and finally, choose Open Selected Services in Flows Tab.

IHRPv1 - Caendra Inc. © 2019 | p.53

 3.3.1 Practical Flow Analysis: Case 1
Detection

By looking at the flows and their duration, it looks like we are dealing with a
beaconing malware. The key phrase here is “looks like”. Network flows cannot
provide us with solid evidence regarding an incident. They can speed up our
analysis though and also serve as a great indicator towards the right direction.

IHRPv1 - Caendra Inc. © 2019 | p.54

 3.3.1 Practical Flow Analysis: Case 1
Detection

If you are curious, the provided PCAP contains traffic from

a host infected by the Nuclear exploit kit, which is indeed
beaconing.

PCAP taken from: http://www.malware-traffic-

analysis.net/2015/03/09/

IHRPv1 - Caendra Inc. © 2019 | p.55

3.3.2 Practical Flow Analysis: Case 2

Case 2: The organization you work for has tasked you with
analyzing some collected NetFlows, to identify the
interactions of system 192.168.5.100, which was
compromised on August 16th 2016.

Take some time to study nfdump’s man page:

http://manpages.ubuntu.com/manpages/cosmic/en/man1
/nfdump.1.html

IHRPv1 - Caendra Inc. © 2019 | p.56

 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s start by using nfdump to see a protocol overview.

>> nfdump –o long –R . –A proto 'ip 192.168.5.100'

IHRPv1 - Caendra Inc. © 2019 | p.57

 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s start by using nfdump to see a protocol overview.

>> nfdump –o long –R . –A proto 'ip 192.168.5.100'

Nothing curious-looking so far.

IHRPv1 - Caendra Inc. © 2019 | p.58
 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s continue by checking ICMP traffic.

>> nfdump –o long –R . 'ip 192.168.5.100 and proto icmp'

IHRPv1 - Caendra Inc. © 2019 | p.59

 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s continue by checking ICMP traffic.

>> nfdump –o long –R . 'ip 192.168.5.100 and proto icmp'

We notice some pings (ICMP type 8, code 0) and two ICMP timestamp requests (ICMP type 13, code
0). We have already covered that ICMP timestamp requests can be used for malicious purposes.
IHRPv1 - Caendra Inc. © 2019 | p.60
 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s look into UDP as well.

>> nfdump –o long –R . –A proto,dstport –O bytes 'ip
192.168.5.100 and proto udp' | head -10

IHRPv1 - Caendra Inc. © 2019 | p.61

 3.3.2 Practical Flow Analysis: Case 2
Detection

Let’s look into UDP as well.

>> nfdump –o long –R . –A proto,dstport –O bytes 'ip
192.168.5.100 and proto udp' | head -10

We notice MDNS-related, NBNS-related and UPNP-related flows. They seem normal though
considering 192.168.5.100 is a Windows system.

IHRPv1 - Caendra Inc. © 2019 | p.62

 3.3.2 Practical Flow Analysis: Case 2
Detection

To look into TCP and also sort by the number of flows, we execute:
>> nfdump –o long –R . –A proto,dstport –O flows 'ip
192.168.5.100 and proto tcp' | head -10

IHRPv1 - Caendra Inc. © 2019 | p.63

 3.3.2 Practical Flow Analysis: Case 2
Detection

To look into TCP and also sort by the number of flows, we execute:
>> nfdump –o long –R . –A proto,dstport –O flows 'ip
192.168.5.100 and proto tcp' | head -10

Conversations to port 12345 stand out

IHRPv1 - Caendra Inc. © 2019 | p.64

 3.3.2 Practical Flow Analysis: Case 2
Detection
To identify with whom 192.168.5.100 exchanged data on port
12345, we execute:
>> nfdump –o long –R . –A proto,srcip,dstip,dstport 'src ip
192.168.5.100 and proto tcp and dst port 12345'

IHRPv1 - Caendra Inc. © 2019 | p.65

 3.3.2 Practical Flow Analysis: Case 2
Detection
To identify with whom 192.168.5.100 exchanged data on port
12345, we execute:
>> nfdump –o long –R . –A proto,srcip,dstip,dstport 'src ip
192.168.5.100 and proto tcp and dst port 12345'

We notice that 36.98.102.89 is the main destination of the traffic on port

12345. Assume that we have already checked the case of port 12345 being a
source port.
IHRPv1 - Caendra Inc. © 2019 | p.66
 3.3.2 Practical Flow Analysis: Case 2
Detection
Network flows provide us with a bird’s eye view of the whole
network. Let’s leverage that to see all 192.168.5.100 interactions
with other machines on the intranet.
>> nfdump –o long –R . –A proto,srcip,dstip,dstport –O
flows 'src ip 192.168.5.100 and proto tcp and dst net
192.168.5.0/24' | head -10

IHRPv1 - Caendra Inc. © 2019 | p.67

 3.3.2 Practical Flow Analysis: Case 2
Detection
Network flows provide us with a bird’s eye view of the whole
network. Let’s leverage that to see all 192.168.5.100 interactions
with other machines on the intranet.
>> nfdump –o long –R . –A proto,srcip,dstip,dstport –O
flows 'src ip 192.168.5.100 and proto tcp and dst net
192.168.5.0/24' | head -10

We notice traffic towards port 22 (SSH) of the 192.168.5.10 host. We should

note this host down for further investigation.
IHRPv1 - Caendra Inc. © 2019 | p.68
 3.3.2 Practical Flow Analysis: Case 2
Detection

If we try the same as previously and also include source ports, we

may gain a better understanding of what happened.
>> nfdump –o long –R . –A proto,srcip,srcport,dstip –O
flows 'src ip 192.168.5.100 and proto tcp and dst net
192.168.5.0/24' | head -10

IHRPv1 - Caendra Inc. © 2019 | p.69

 3.3.2 Practical Flow Analysis: Case 2
Detection

If we try the same as previously and also include source ports, we

may gain a better understanding of what happened.
>> nfdump –o long –R . –A proto,srcip,srcport,dstip –O
flows 'src ip 192.168.5.100 and proto tcp and dst net
192.168.5.0/24' | head -10

We notice that the majority of the traffic derives from source ports 62604,
41476 and 41477. Each of those source ports connects to one IP address.

IHRPv1 - Caendra Inc. © 2019 | p.70

 3.3.2 Practical Flow Analysis: Case 2
Detection

We can list all interactions from source port 62604, as follows.

>> nfdump –o long –R . 'src ip 192.168.5.100 and proto tcp
and src port 62604 and dst net 192.168.5.0/24' | head -20

IHRPv1 - Caendra Inc. © 2019 | p.71

 3.3.2 Practical Flow Analysis: Case 2
Detection

We can list all interactions from source port 62604, as follows.

>> nfdump –o long –R . 'src ip 192.168.5.100 and proto tcp
and src port 62604 and dst net 192.168.5.0/24' | head -20

We notice very short flows and packets having the SYN bit set. No doubt the
compromised host is being used to scan the intranet.

IHRPv1 - Caendra Inc. © 2019 | p.72

 3.3.2 Practical Flow Analysis: Case 2
Detection

Finally, we can also list connections from the intranet to the internet,
as follows.
>> nfdump –o long –R . –A proto,srcip,dstip –O flows 'src
ip 192.168.5.100 and proto tcp and ! dst net
192.168.5.0/24' | head -20

IHRPv1 - Caendra Inc. © 2019 | p.73

 3.3.2 Practical Flow Analysis: Case 2
Detection

By analyzing flows we have identified the following

regarding 192.168.5.100.
1. Conversations with the 36.98.102.89 host, port 12345
2. It scanned some intranet hosts
3. It connected to the SSH port of the 192.168.5.10 host

IHRPv1 - Caendra Inc. © 2019 | p.74

 3.3.2 Practical Flow Analysis: Case 2
Detection

By now, you should have an idea of how we can analyze

network flows. But what about enriching them?

Suppose we were also given Squid proxy logs. Let’s see

how we can leverage them to enrich the provided network
flows.

http://www.squid-cache.org/ IHRPv1 - Caendra Inc. © 2019 | p.75

 3.3.2 Practical Flow Analysis: Case 2
Detection

There are usually two important Squid proxy logs, cache.log,

which is the internal log of the caching proxy itself and
access.log, which contains all the passing URLs.

Let’s focus on access.log.

IHRPv1 - Caendra Inc. © 2019 | p.76

 3.3.2 Practical Flow Analysis: Case 2
Detection

For a better understanding let’s remove all “benign” entries

from access.log, as follows.
>> cat access.log | grep –v "ubuntu.com" | grep –v
"opensuse" | grep –v "openSUSE" | grep -v "novell.com"

IHRPv1 - Caendra Inc. © 2019 | p.77

 3.3.2 Practical Flow Analysis: Case 2
Detection

For a better understanding let’s remove all “benign” entries

from access.log, as follows.
>> cat access.log | grep –v "ubuntu.com" | grep –v
"opensuse" | grep –v "openSUSE" | grep -v "novell.com"

A file named binaries-only.zip was obviously downloaded by the intranet host 192.168.5.10 from the
54.229.228.176 remote server.
IHRPv1 - Caendra Inc. © 2019 | p.78
 3.3.2 Practical Flow Analysis: Case 2
Detection

Since we haven’t came across this destination when

analyzing the network flows. Let’s now see what network
flows have to say about this destination, as follows.
>> nfdump –o long –R . –A proto,srcip,srcport,dstip 'src ip
54.229.228.176 and proto tcp'

IHRPv1 - Caendra Inc. © 2019 | p.79

 3.3.2 Practical Flow Analysis: Case 2
Detection

Since we haven’t came across this destination when

analyzing the network flows. Let’s now see what network
flows have to say about this destination, as follows.
>> nfdump –o long –R . –A proto,srcip,srcport,dstip 'src ip
54.229.228.176 and proto tcp'

We notice that the 192.168.5.10 intranet host downloaded something from the 54.229.228.176 remote server, but we
already knew that from our Squid log analysis. What we didn’t know was that the 192.168.5.100 intranet host also
downloaded something from the same server. This is obvious from the number of bytes transferred. In addition, we
didn’t see the 192.168.5.100 host downloading something when analyzing the Squid logs. The proxy was somehow
bypassed, which is quite suspicious.
IHRPv1 - Caendra Inc. © 2019 | p.80
3.3.2 Practical Flow Analysis: Case 2

As you saw, the additional Squid proxy logs helped us move

our investigation deeper (a.k.a enriched the network flows).

Credits go to ENISA for the data set we just analyzed.

https://www.enisa.europa.eu/ IHRPv1 - Caendra Inc. © 2019 | p.81

3.3.3 Practical Flow Analysis: Case 3

Network flow tracking can facilitate the detection of

anomalous DNS activity. Examples:
• Categorize DNS packets: DNS requests, DNS responses
and unknown.
• Detect large fluctuations in DNS-related packet size per
hour
• HTTP flows not preceded by a DNS request

IHRPv1 - Caendra Inc. © 2019 | p.82

 3.3.3 Practical Flow Analysis: Case 3
Detection

Undoubtedly, if an incident responder comes across a flow

report like the below, he/she should be concerned.

IHRPv1 - Caendra Inc. © 2019 | p.83

 3.3.3 Practical Flow Analysis: Case 3
Detection

Undoubtedly, if an incident responder comes across a flow

report like the below, he/she should be concerned.

We notice that the 10.100.100.24 intranet host has issued an immense number of DNS requests.
Such a behavior is suspicious if and only if the 10.100.100.24 host is an end-user workstation and
not a system that performs name resolutions repeatedly by design.

IHRPv1 - Caendra Inc. © 2019 | p.84

3.3.4 Practical Flow Analysis: Case 4

Network flow tracking can facilitate the detection of

anomalous SMB activity. For example, consider the
following network flow reports.

IHRPv1 - Caendra Inc. © 2019 | p.85

 3.3.4 Practical Flow Analysis: Case 4
Detection
Flow report
regarding
inbound SMB
traffic to
10.200.100.0/24
VLAN

Flow report
regarding
outbound SMB
traffic from the
10.200.100.0/24
VLAN

IHRPv1 - Caendra Inc. © 2019 | p.86

 3.3.4 Practical Flow Analysis: Case 4
Detection
Flow report In a Windows-based
environment, traffic on port
regarding 445 (TCP) between
inbound SMB workstations or between
workstations and file servers
traffic to is common. In this case
10.200.100.0/24 though, we notice a machine
VLAN reaching port 445 (TCP) on
numerous other workstations.

We can’t be certain that

something is wrong, but note
that such behavior has been
spotted in the past by
malware, such as the
Flow report devastating conficker worm
regarding that used port 445 (TCP) to
spread.
outbound SMB
traffic from the In addition, the small number
of packets and their
10.200.100.0/24 “distribution” across the VLAN
VLAN is also curious-looking.

http://www.csl.sri.com/users/vinod/papers/Conficker/ IHRPv1 - Caendra Inc. © 2019 | p.87

3.3.5 Practical Flow Analysis: Case 5

Visualizing network flow data can oftentimes result in

quicker and more efficient intrusion detection. A great tool
for analyzing as well as visualizing network flow data is
iSiLK. iSiLK is essentially a graphical front-end for the SiLK
suite of tools.

https://tools.netsa.cert.org/isilk/index.html IHRPv1 - Caendra Inc. © 2019 | p.88

3.3.5 Practical Flow Analysis: Case 5

Let’s take for example, the SiLK command flow we used to

identify top webservers.
>> rwfilter filename.rwf --sport=80,443,8080 --protocol=6 -
-packets=4- --ack-flag=1 --pass=stdout | rwstats --
fields=sip --percentage=1 --bytes

IHRPv1 - Caendra Inc. © 2019 | p.89

3.3.5 Practical Flow Analysis: Case 5

Let’s do the same through iSiLK.

• http://tools.netsa.cert.org/isilk/isilk-admin-guide.pdf
• http://tools.netsa.cert.org/isilk/isilk-user-guide.pdf

IHRPv1 - Caendra Inc. © 2019 | p.90

3.3.5 Practical Flow Analysis: Case 5

First, execute the below to start iSiLK.

>> python isilk.py

Then, click on File -> Import Remote Data File

IHRPv1 - Caendra Inc. © 2019 | p.91

3.3.5 Practical Flow Analysis: Case 5

Choose the file you want to import and click OK. Example:

You should see something similar to the below.

IHRPv1 - Caendra Inc. © 2019 | p.92

3.3.5 Practical Flow Analysis: Case 5

To replicate the first SiLK command, click on Filter.

IHRPv1 - Caendra Inc. © 2019 | p.93

3.3.5 Practical Flow Analysis: Case 5

Then, specify the wanted source ports.

IHRPv1 - Caendra Inc. © 2019 | p.94

3.3.5 Practical Flow Analysis: Case 5

Also, specify the other filter options as follows and click Run Analysis.

IHRPv1 - Caendra Inc. © 2019 | p.95

3.3.5 Practical Flow Analysis: Case 5

You will see something similar to the below. Click on it…

IHRPv1 - Caendra Inc. © 2019 | p.96

3.3.5 Practical Flow Analysis: Case 5

The rwfilter’s result will appear.

IHRPv1 - Caendra Inc. © 2019 | p.97

3.3.5 Practical Flow Analysis: Case 5

Now, to replicate the second part of the command flow,

click on Stats.

IHRPv1 - Caendra Inc. © 2019 | p.98

3.3.5 Practical Flow Analysis: Case 5

Finally, specify the running rwstats and click Run Analysis.

IHRPv1 - Caendra Inc. © 2019 | p.99

3.3.5 Practical Flow Analysis: Case 5

If you now click on the remote file that was created, the
final results will appear.

IHRPv1 - Caendra Inc. © 2019 | p.100

3.3.5 Practical Flow Analysis: Case 5

The results are the same as the ones create by SiLK.

IHRPv1 - Caendra Inc. © 2019 | p.101

3.3.5 Practical Flow Analysis: Case 5

iSiLK has many network visualization features and is great

for learning SiLK. Spend some time to get comfortable with
it.

IHRPv1 - Caendra Inc. © 2019 | p.102

References

IHRPv1 - Caendra Inc. © 2019 | p.103

 References
NetFlow
https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.html

NetFlow for Cybersecurity

http://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3

YAF
https://tools.netsa.cert.org/yaf/

SiLK
https://tools.netsa.cert.org/silk/

IHRPv1 - Caendra Inc. © 2019 | p.104

 References
FlowViewer
https://ensight.eos.nasa.gov/FlowViewer/

Application labeling
https://tools.netsa.cert.org/yaf/applabel.html

DHCP
http://tools.netsa.cert.org/yaf/yafdhcp.html

yaf-file-mediator
https://tools.netsa.cert.org/confluence/display/tt/YAF+2.x+IPFIX+File+Mediator

IHRPv1 - Caendra Inc. © 2019 | p.105

 References
Deep packet inspection
https://tools.netsa.cert.org/yaf/yafdpi.html

rwcut
https://tools.netsa.cert.org/silk/rwcut.html

rwfilter
https://tools.netsa.cert.org/silk/rwfilter.html

rwfileinfo
https://tools.netsa.cert.org/silk/rwfileinfo.html

IHRPv1 - Caendra Inc. © 2019 | p.106

 References
rwstats
https://tools.netsa.cert.org/silk/rwstats.html

rwcount
https://tools.netsa.cert.org/silk/rwcount.html

rwscan
https://tools.netsa.cert.org/silk/rwscan.html

Network Traffic Analysis - SiLK

https://schd.ws/hosted_files/flocon2017/fa/flocon-2016-silk-tutorial.pptx

IHRPv1 - Caendra Inc. © 2019 | p.107

 References
Network Traffic Analysis with SiLK
https://tools.netsa.cert.org/silk/analysis-handbook.pdf

RVAsec: Jason Smith - Applied Detection and Analysis Using Flow Data

https://www.youtube.com/watch?v=ndfcfHiszHY

FlowViewer
https://ensight.eos.nasa.gov/FlowViewer/

flow-tools
https://github.com/adsr/flow-tools

IHRPv1 - Caendra Inc. © 2019 | p.108

 References
CapLoader
https://www.netresec.com/?page=CapLoader#trial

PCAP: Traffic Analysis Exercise

http://www.malware-traffic-analysis.net/2015/03/09/

nfdump’s man page

http://manpages.ubuntu.com/manpages/cosmic/en/man1/nfdump.1.html

Squid proxy
http://www.squid-cache.org/

IHRPv1 - Caendra Inc. © 2019 | p.109

 References
ENISA
https://www.enisa.europa.eu/

conficker worm
http://www.csl.sri.com/users/vinod/papers/Conficker/

iSiLK
https://tools.netsa.cert.org/isilk/index.html

Development & Deployment Guide for iSiLK Version 0.1.2

http://tools.netsa.cert.org/isilk/isilk-admin-guide.pdf

IHRPv1 - Caendra Inc. © 2019 | p.110

 References
User’s Guide for iSiLK version 0.3.2
http://tools.netsa.cert.org/isilk/isilk-user-guide.pdf

IHRPv1 - Caendra Inc. © 2019 | p.111