Metron - Prelude

Apache Metron is an open source Security Data Analytics Platform (SDAP).

The course helps you to learn:

 The capabilities of Metron

 The architecture of Metron
 Steam Processing Pipeline
 Using the pipeline for Security
 Metron as MaaS
 Use cases of Metron

 What is Metron?
 Apache Metron is an open-source centralized security tool
for monitoring and analysis.
 It is specifically designed to ingest, process, enrich, store and
perform analytics on real-time security data feeds, machine logs
and network traffic coming from a multitude of "data in motion"
sources.
 With Hadoop Community in mind, it is built to provide security
analytics framework, which is advanced yet scalable.

Evolution of Apache Metron

 From 2005 to 2008, Cybercrime spiked significantly, and Cisco offered
services through managed security operations centers.
 After 2008, with the emergence of Big Data ecosystems, costs of
traditional SIEMs increased, throwing Cisco's managed SOCs into
jeopardy.

 In 2013, Cisco tried to leverage open source tools and matured Hadoop,
OpenSOC was created, which is the first project to take advantage of
Hadoop, Kafka, and Storm.

 From September 2013 to April 2015, for around 24 months, a Cisco

team, with the help of a Hortonworks team, worked to create a next-
generation managed SOC service built on top of open source big data
technologies.

 In December 2015, Metron renamed from OpenSOC became an Apache

incubator project. It is built to be extensible and open to support a
multitude of customer environments.

 Finally, in April 2016, after 4 months of toil and innovation, the Metron
Community can launch the first Official release, i.e., Metron 0.1.
Note:

SIEM - Security Information and Event Management SOC - Security

Operations Center

A Real-World Scenario

Consider a real-time scenario where a massive amount of logs and network data keeps flowing in.
Traditional approach requires too many steps using multiple tools making security investigation tedious,
slow and also expensive.

The cost increases over time to keep the data stored to understand history for enough time. Contextual
information extraction through data collection is also an expensive job.

Even if all these issues are taken care of manually, we cannot be sure to detect a targeted event as there
are too many events to be reviewed promptly. It requires a no of human resources to monitor the events
in time so that the breach can be detected ASAP. The demerits of manual work are further fueled with
Hackers getting more sophisticated.

At this time, Metron comes to rescue, integrating with multiple tools and performing real-time analysis.
Uses of Metron
Capabilities

According to Apache:

Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced
behavioral analytics, and data enrichment while applying the most current threat intelligence
information to security telemetry within a single platform.

This is possible through:

 High rate mechanism to capture, store, and normalize security telemetry of multiple types.
 Application of enrichments like DNS information, threat intelligence and geo-location to
telemetry with Real-time processing.
 Efficiency in information storage for mining, data security, advanced analytics, and anomaly
detection.
 Rich UI for alert and data analysis and in-built support for advanced investigation tools.
Core Functionality

Metron functionality can be divided into four major categories:

  Security Data Lake/Vault
 Pluggable Framework

 Security Application

 Threat Intelligence Platform

Functionality - Explained
 Security Data Lake / Vault:
Facilitates Discovery analytics through storage of enriched
telemetry and Operational Analytics through the mechanism for
searching and querying. Everything is designed to be cost
effective.
 Pluggable Framework:
Provides a rich set of parsers for familiar security data sources
(like Fireye, NetFlow, Snort, pcap, Bro, Sourcefire). Custom
parsers can be added for new data sources through a pluggable
framework. Raw streaming data can be enriched by providing
more contextual data via enrichments. Apart from these, security
dashboards can be, and extensions can be plugged in for threat
intel feeds.
 Security Application:
Metron provides capabilities like hunting services, evidence store
and packet replay utilities for SOC analysts.
 Threat Intelligence Platform:
Anomaly detection and Machine learning algorithms can be
applied in real-time while events are streaming in which can be
considered as next-gen defense techniques.
 Who Uses Metron?
 Metron is commonly used by:
 SOC Analyst: Uses the tool to analyze the alerts and escalate
them.
 SOC Investigator: Investigates the complex attacks and try to
remove/quarantine the threats.
 SOC Manager: Assigns and Verifies the work with the Metron tool
to the proper individual.
 Forensic Investigator: Collects the evidence regarding the
breaches and takes care of legal affairs.
 Security Platform Operations Engineer: Sets up the tools
required, monitor them and implements the best technological
practices.
 Security Data Scientist: Responsible for working on data and
implement complex models to find patterns.
All the users together are known as User Personas of Metron.

How are Users Benefited?

Metron Benefits are dependent on User Perspective.
In PoV of an Analyst/Investigator:
 Centralized Alerts Console, Meta-Alerting feature and threat Intel-based
labeling improve productivity.
 Fully Enriched messages, Real-time search, and Simple UI helps in
contextual data collection.

 Granular access, behavior tagging, and evidence stores are provided

for investigation.

How are Users Benefited?

In PoV of a Security Data Scientist:
 Data finding assisted with single data store, multiple API access, and
standard access policies.
 Cleaning through telemetry normalization and partial validation at time
of ingestion.

 Munging with automatic data enrichment, labeling, and parallel

computation support.

 Visualizing with Jupyter, and Zeppelin support simplified through UI.

Going forward you will learn how all these features can be utilized to
achieve our requirements.

Trivia

There is a DC Comic Character with the same

name as Metron, a god of
Knowledge (coincidental, maybe not) in
search of greater knowledge than his own.
The Components

Metron has modules for:

 Normalizing Telemetry
 Enrichment
  Streaming Network Packets

 Analytics Service

 Dashboards

 Bulk loading threat intelligence and enrichment.

 UI customization

 Deployment Automation

All the Metron data is normalized and handled in Metron JSON format
through domain-specific Stellar Query and Stellar Transformation
Languages.

Architecture
With a fundamental view on the functionality, uses, and components of
Metron, let's look at how Metron components come together to achieve
the functionality.
Metron architecture can be viewed from two perspectives:
 Component Level Architecture.
 Logical Architecture

Move over to the next cards to know how they differ and learn the
composition of Metron.

Component Level Architecture

This architecture explains how Metron is an amalgamation of multiple
technologies at different levels to process the telemetry events.

How Does the Components Work?

  Metron is built on top of the Apache Hadoop ecosystem.
 This enables capturing, ingesting, enriching and storing of
streaming data at scale.

On looking at the dependencies/integrations:

 Kafka provides a unified data bus.

 Storm provides a distributed streaming framework.
 HBase provides a low latency key/value lookup store for
enrichments and profiles.
 Zookeeper provides a distributed configuration store.

Components Handling Telemetry

 Telemetry enrichment is pluggable.
 New telemetry can be enriched on live without restart using
topologies.
 User-defined functions can add new enrichment capabilities.
 Enrichments can be composed through the domain specific
language - Stellar.
 Example of an enrichment source is the data stored in HBase.
 Security Data lake can be used to index Enriched telemetry data.
 Supports pluggable Indexes like Solr, HDFS, and Elasticsearch.

Analytics Support by the Components

 We can apply Advanced analytics over streaming data.
 Streaming data can be sketched across time using Probabilistic data
structures (e.g., sketches) to set existence, approximate distribution,
set existence and queries for a distinct count.

 While managing models auto-discovery is possible through Zookeeper,

deployment using Yarn, and Stellar functions enable interrogation of
models.
Logical Architecture

The logical architecture explains how an event is processed as it flows

into the Metron Environment.
Processing an Event
Here is a brief overview of how an event is processed at each stage of
logical architecture, once it is ingested into the environment.
1. Telemetry event Buffer: Captures the raw data from custom
sources or like Apache Nifi and marks the beginning of processing
by Metron.
2. Process: Parses, Normalizes, Validates and Tags the event into a
standard Metron JSON format.
3. Enrich: Enrichment sources are utilized to add additional helpful
data elements to the event.
4. Label: The enriched data is labeled/recognized using the threat
intel metadata available from the threat intel feed data sources
like
5. Alert and Persist: Telemetry events at this phase can trigger
alerts. Event type alerts are where raw telemetry event itself is
an alert. They are indexed in the alert index store. When
telemetry has a threat intel, hit it is considered as a threat intel
hit alert.
6. UI and other services: These help in the efficient working of the
Metron users. Includes real-time search/dashboards, Data
modeling/feature engineering, and Integration/Extensibility
layers.

7 (a). Fast telemetry Ingest: Custom Metron probes for high volume
telemetry capture like PCAP, Bro, and DPI.
7 (b). Telemetry Ingest: Data capture in case of protocols like REST,
Syslog, custom API, HTTP. Apache Nifi is used by Metron to ingest data
at the source.
Data Sheet for Current Support
As you saw how the components come together and how an event is
processed, now, have a look at the tools/formats supported by the
current version of Metron at various levels.