Agentless System Crawler - InterConnect 2016

IMPORTANT info regarding IBM speaker guidelines and disclaimers
• If your presentation has forward looking content, it is mandatory that you put the forward disclaimer
as slide 2 in your presentation (this is the “Please Note” slide, third slide down in this template).
• All presentations, whether they have future content or not, must include the mandatory “Notices and
Disclaimers” – slides 8 and 9 in the template. Insert these slides just before the “Thank You” slide in
your deck.
• Please refer to the FAQ document in the Speaker Kit regarding additional legal guidance for use of
photos, logos, customer references and analyst information.
• It is recommended to have your material reviewed by Legal if you have any concerns regarding your
content.
• Please submit your final presentation, using the instructions in the online Speaker Kit, by February
5th
, 2016. Post your final file in native format using the following naming convention: session code.ppt
(For example, 1576.ppt)
• Disclosures regarding forward guidance is embedded in the tool and also available through this link:
• https://w3-03.ibm.com/finance/finsubp.nsf/WebPages/N01FF08SoftwareRevenueRecognitionGuidelinesRelatedtoProductDisclosures
• Please remove these instructions before finalizing your presentation.
1

Section Head Goes Here
Subhead

Agentless System Crawler
https://developer.ibm.com/open/agentless-system-crawler/
Canturk Isci
IBM Research, NY
@canturkisci
DeveloperWorks
SmartBar Talk
Mon Feb 22, 4:00 PM

Please Note:
4
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole
discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in
making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any
material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole
discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual
throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

- Provide unmatched deep, seamless visibility into cloud instances
- Drive operational insights to solve real-world pain points
- Provide unmatched deep, seamless visibility into cloud instances
Built-in Monitoring [& Analytics] Designed for Cloud

- Provide unmatched deep, seamless and unified visibility into ALL cloud instances
- Provide unmatched deep, seamless and unified visibility into ALL cloud instances
Built-in Monitoring [& Analytics] Designed for Cloud
Agentless System Crawler (ASC)

Key Driver: Agentless (Non-intrusive) Introspection
Patch
Relevance
Security
Monitor
Compliance
Audit

Traditional Monitoring vs. Crawlers
OS
Host
Wkld
Agent
Agent
Agent
Agent
OS
Host
Wkld A A
AA
V
M
OS Wkld A A
AA
Host
OS
Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
.
VMBMS Container
OS
Host
Wkld OS
Host
Wkld
V
M
OS Wkld
Host
OS
Wkld
Cont
. Wkld
Cont
. Wkld
Cont
.
VMBMS Container

Key Advantages
Key Advantages
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Why Agentless System Crawlers
magicmagic
 Monitoring built into the platform
not in end-user systems
 No complexity to end user
(They do nothing, all they see is the service)
 No agents/credentials/access
(nothing built into userworld)
 Works out of the box
 Makes data consumable
(lower barrier to data collection and analytics)
 Better Security for end user
(No attack surface, in userworld)
 Better Availability of monitoring
(From birth to death, inspect even defunct guest)
 Guest Agnostic
(Build for platform, not each user distro)
 Decoupled from user context
(No overhead/side-effect concerns)
 Monitoring done right for the
processes of the Cloud OS

”Users do not have to do anything to get this visibility. It is already there by default”
Container Cloud
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Metrics & Logs
Bus
Multitenant
Index
Logmet
Svc
Provisioning
Tenancy Info
State
Events
 Built-in in every compute node, all geos
 Enabled by default for all users in all prod
 O(10K) metrics/s & logs/s
Current State
Seamless: Built-in Monitoring & Logging in Bluemix Containers

Container Cloud
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Cool!
Happy User:
Effortless, painless
visibility in user world
magicmagic
Seamless: Built-in Monitoring & Logging in Bluemix Containers
”Users do not have to do anything to get this visibility. It is already there by default”

Deep Visibility: What We Actually Collect (and Annotate)
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator

Deep Visibility  Operational Insights/Analytics  Solve Real Problems
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
Index (Data)
Vuln. &
Compl.
Analysis
Secure
Config
Analysis
Forensic
Security &
Compl.
Pipeline
Service
Remediation
Service

Deep Visibility  Operational Insights/Analytics  Solve Real Problems
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
Index (Data)
Vuln. &
Compl.
Analysis
Secure
Config
Analysis
Forensic
Security &
Compl.
Pipeline
Service
Remediation
Service
Also at InterConnect:
Vulnerability Advisor
Session: SAD-7286
Sun: 11am & Wed:4pm

Crawler: How it Works for VMs
• Leverage VM Introspection (VMI) techniques to access VM Mem and Disk state
(We built bunch or our own optimizations that make this very efficient and practical)
• Can even remote both (decouple all from VM and host)
• Almost no new dependencies on host
• Currently support 1000+ kernel distros
Hypervisor
MEM
View
KB
APP
Analytics
Apps
Memory
Crawl
API
VM
OS
MEMDisk
Disk
View
Disk
Crawl
API
Cloud Analytics
Crawl
Logic Structured
view of
VM states
APP
APP
{
.......
.......
}
Frames

Crawler: How it Works for Containers
• Leverage Docker APIs for base container information
• Exploit container abstractions (namespace mapping and cgroups) for deeper insight
• Provide deep state info at scale with no visible overheads to end user
1) Get visibility into container world
by namespace mapping
2) Crawl the container
(Crawler dependencies still borrowed from host.
No need to inject into container!)
3) Return to original namespace
4) Push data to backend index

DEMO TIME
This Session
This Session
 Agentless System Crawler
 Bluemix Test Drive (live – ldwave)
https://developer.ibm.com/bluemix/2015/11/16/built-in-monitoring-and-logging-for-
bluemix-containers/
 LogCrawler and JSON Parsing
(live – CanoLibUK3)
 Vanilla LogCrawler
(20150619_LogCrawlerDemo)
 Crawl even Non-responsive systems
(oopsRconsole2)
 Out of Band SIEM
(QRadarDemo)
 TopoLog for Topology Discovery
(newTopo)
 RTop for Realtime Monitoring
(RtopAnnotatedMOV)
 Crawling for Rootkits with RConsole
(RConsoleAnnotatedMOV)
Sunday & Wednesday
Sunday & Wednesday
 Vulnerability Advisor
 Research Day Sessions
SAD-7286
Sun 11:00am
Wed 4:00pm

Bluemix Test Drive
Just start a Bluemix Container
(https://console.ng.bluemix.net/)
Just start a Bluemix Container
(https://console.ng.bluemix.net/)
Go to Container Overview
(Metrics show up in few mins)
Go to Container Overview
(Metrics show up in few mins)

… Bluemix Test Drive
Go to Monitoring and Logs
>> Monitoring
>> Monitoring

… Bluemix Test Drive
>> Logging
>> Logging

Notices and Disclaimers
22
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM
products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those
customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries
in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant
or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.
23
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other
intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee Portal to complete your
session surveys from your smartphone, laptop or conference kiosk.
DeveloperWorks SmartBar Talk/Demo:
Agentless System Crawler
@canturkisci

Agentless System Crawler - InterConnect 2016

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Agentless System Crawler - InterConnect 2016 (20)

Recently uploaded (20)

Agentless System Crawler - InterConnect 2016

Editor's Notes