DevOps for Red Team

Initial Access…
My journey into fun DevOps.
Or…”how to pervert CI/CD for fun and profit”…
Author: Joff Thyer © October 2023
About Joff
• Black Hills Information Security
• AntiSyphon Instructor
• Research/Dev/Initial Access ops.
• Unofficial title of “Malware Pit Boss”
• Credit to @OrthicOn for creating the Joff/Yoda
meme.

2
What?
• DevSecOps versus SecDevOps is a thing?
• According to Accunetix…

• Jedi mind hack.. This talk is not really about securing malware!

3
The defense landscape 
• Static Artifact Analysis
• Event Tracing for Windows
• Windows Kernel Mode Callbacks
• Windows DLL API Hooking
• Process Tree Analysis
• Memory Page Scanning
• Call Stack Tracing/Analysis
• Kernel Driver Block Lists

4
Static Artifact Analysis
• Direct malware hash match in Virus Total and friends.
• Unencrypted Shellcode Byte Sequence/Pattern
• High Entropy of Binary Artifact
• Any little tell-tale thing in an artifact that is unique to that
artifact!

5
Event Tracing for Windows
• Trace and log events raised by:
• User mode applications
• Kernel mode drivers
• API is in three components
• Controllers – start/stop sessions
• Providers supply the events
• Consumers which ingest the events
• Three types of providers
• MOF (classic mode)
• WPP instrumentation based
• Trace Logging  this is what provides the ETW logging.

6
Kernel Mode Callbacks
• Windows NT Kernel allows a signed driver to “register” a callback
routine.
• Very commonly used by AV/EDR/XDR
• High Level as follows:
• PsSetCreateProcessNotifyRoutine
• Register a callback for process creation/deletion events
• PsSetCreateThreadNotifyRoutine
• Register a callback for thread creation/deletion events
• PsSetLoadImageNotifyRoutine
• Register a callback for image load/mapping into memory
• ObRegisterCallbacks
• Register a callback for objects (process/thread/desktop handle is
opened/duplicated)
• CmRegisterCallback
• Register a callback for registry operations
7
Windows NTDLL API Hooking
• Replace second opcode with JMP to EDR/XDR DLL.
• Note: The second opcode is normally: MOV EAX, #SSN where #SSN
is the system call number.
• API call goes to the EDR/XDR code and can optionally be blocked.
• Hooks are often initiated in response to a kernel callback
notification to an EDR/XDR driver
• EDR/XDR DLL and Hooks may be installed for all processes
through callback notification in “PspImageLoadNotify” table!
• Note: starting to fall out of favor with the advent of memory scanners,
and call stack tracing

8
 Malware uses an interesting
malware exe/dll API call like:
NtCreateProcess()
1
NtCreateProcessEx()
NtCreateThread()
NtCreateThreadEx()
NTDLL
4

EDR/XDR User Mode Process

3 Driver message to inject DLL

into malware which then installs
NTDLL hooks. May trigger call
stack analysis/memory scan also.
Callback Function Table
0xffffffff01010101
0xffffffff02001234 2
EDR/XDR Kernel Driver
0xffffffff034400ff
Driver
EDR/XDR Kernel Driver On startup registers callbacks:
. - PsSetCreateProcessNotifyRoutine()
receives
. notification - PsSetCreateThreadNotifyRoutine()
up to 64 entries - PsSetLoadImageNotifyRoutine()
May also register as kernel mode
consumer of ETW events.

9
Process Tree Analysis
• Monitor and track process relationships
• Generate internal process tree data structure
• Look for anomalous relationships
• Look for anomalous child process behavior
• Often leverages AI models to compare with known good behaviors

10
Memory Page Scanning
• Malware typically will allocate a virtual memory page in a
process for malicious code
• Machine code injected can be a full DLL or just a byte array of
assembly opcodes
• A thread is created resident in same process or foreign process with
start address of this injected code.
• EDR/XDR can use “NtQueryInformationThread()” to
determine thread start address
• VirtualQueryEx() used to get memory section properties
• If start address NOT backed by disk AND memory is RX
•  THREAD IS INJECTED!

11
Call Stack Tracing/Analysis
• Background
• Whenever a function ( “call”) is called, the return address gets pushed onto the stack
• Upon return, it pops this address from the stack to populate the instruction pointer
before continuing (“ret”)
• EDR/XDR can use a kernel callback to trigger “call stack analysis”.
• Unwind the call stack
• Check to see if any “calls” were made from unbacked memory
• Flag alert
• Related
• PE/COFF “.pdata” file section contains exception handling information in Function Table Entries
(FTE’s)
• Contain exception handling code on a per function basis
• Offense: We can FAKE the stack so it looks LEGIT.
• Defense: Windows 11 with kernel-mode hardware enforced stack protection
• Shadow stacks are maintained and compared. BSOD if mismatch.

12
Kernel Driver Blocklists
• Our world is FULL of vulnerable (signed) kernel drivers
• Microsoft now enforcing blocklist in Windows 11
• Not an easy list to maintain
• Exploitation of signed vulnerable drivers is a juicy target
• At the kernel level…
• We can fiddle with process protections
• We can overwrite kernel notification callback tables
• We can disable driver signing
• We can “insert fun kernel thing here”!
• BEWARE:
• BugCheck/BSOD if developing at kernel level (the struggle is real…)

13
This Malware Dev Shit got real!
• To succeed, we have to:
• Make the software unique continuously to evade static
detections
• Perhaps squelch event tracing (optional/dangerous)
• Make any kernel callback notifications “not
interesting” or better “turn them off”
• Avoid getting our NTDLL “hooked”, “unhook” or not
use NTDLL
• Make our processThisrelationships “not
Photo by Unknown Author is licensed interesting”
under CC BY-SA

• Don’t create a suspicious “non disk backed” thread

• Evade all sandboxes
• Don’t use a driver on the block list

14
Now add in some requests…
• Hey Joff, can you give us all these things?
• .NET assemblies both DLL and EXE
• Native Windows EXE, and DLL’s
• MSI would be nice! This Photo by Unknown Author is
licensed under CC BY-NC-ND

• Can you throw in some AppX/MSIX and ClickOnce?

• How about a service EXE also?
• Dude, can you drop me an XLL?
• How about a reflexive injection DLL?
• … and more and more….

15
IMO Everything points  CI/CD
• Code obfuscation
• Source code recompilation for unique artifact
production
• Dynamic encryption key generation
• Automated packaging MSI/ClickOnce/AppX/MSIX
• Binary artifact signing
• Integration of 3rd party tools as needed
• Dynamic regeneration of indirect kernel system call
code stubs
• SysWhispers3 for example.
• Continuous defense evolution

16
Gitlab Community Edition and CICD
• Gitlab is not just about software repos!
• CI/CD pipeline features:
• Pipeline is configured via “.gitlab-ci.yml” file in repo home dir.
• YML syntax allows you to define build stages
• Different jobs are associated with each stage
• A pipeline run can be triggered simply by a “git commit/push” to a
repo, or a merge for example.
• Pipeline run trigger conditions can be defined.
• Pipelines can also have dynamically created “child” pipelines

17
Gitlab Runners
• A gitlab runner is installed on a standalone server or virtual
container
• The runner uses the gitlab API to register itself and service
pipeline runs.
• The runner “processes” the instructions in the YML file, and in
most cases will run a “script” indicated by the script tag.
• We deployed both Windows and Linux based runners.

18
 ESXi Hypervisor
Each linux host is running 8
“Zulu Foxtrot” Containers
All are individual runners.
8 CPU Cores, 40GB RAM
Linux Server01
Ubuntu 22.04 LTS running Docker

Linux Server02
Ubuntu 22.04 LTS running Docker

GitLab Server

NAS Storage Array

Windows Server01 2019
Full Visual Studio Stack, MAGE,
MakeAppx etc.

Each windows host is running

a single runner which can
4 CPU Cores, 32GB RAM
parallelize 4 jobs.
Windows Server02 2019
Full Visual Studio Stack, MAGE,
MakeAppx etc.

19
“Zulu Foxtrot” Docker Container
• This is a custom malware source code compilation framework.
• Supports different “modules” using a Python command line interface
• Support for multiple different programming languages
• C# compilation with Mono “mcs”
• Golang compilation
• C/C++ compilation using “mingw-gcc”
• Additional features include:
• Resource script definition and execution
• Source code templating and obfuscation system
• Binary artifact inflation
• Evasion modules in multiple languages (neuter AMSI, ETW etc…)
• PE/COFF analysis and fixes as needed
• Integrated gitlab runner!

20
Pipeline Stages

Create Zulu Foxtrot Compile, obfuscate, Create Artifact

Perform application ZIP results for
and Scare Crow and produce binary Manifest and prepare
Packaging Work! delivery to operator.
pipelines. artifacts child pipelines.

21
“Zulu Foxtrot” Parallelized Jobs!

Individual jobs can be

serviced by one of 16
docker containers across 2
hosts!

22
Python and YAML playing together
• The master parent pipeline is in “.gitlab-ci.yml” file and is
static
• Python scripts are called from this pipeline to generate more
CICD yml file content
• YML file artifacts get passed down from one stage to the next

23
Sequential pipeline stages.

These are the triggers (changes) needed to trigger

the entire pipeline run. Ie: Operator configured
items.

This entire UNIX shell script is executed for this

“__A0_PipelineYAML” job.
We derive the TARGET directory from the git
“commit author” and short sha hash contents.

We produce two YML files in this stage for child

pipeline use. We don’t need these for long so
expire in 30 mins.

24
Dynamically Triggered Child Pipelines

The actual malware production occurs

in the sequential “BakingMalware”
stage. These are triggered and
dependent on the
“__A0_PipelineYAML”.

25
How does the operator drive this?
• At present, there are two basic steps for the operator
• Edit a file called BuffetConfig.yml to enable features, and configure
different options
• If needed, supply shellcode in 32-bit, or 64-bit, or both in a specific
directory.
• Perform a “yolo” git commit.
• git commit –a –m “.” && git push
• Monitor the pipeline progress and download resulting ZIP files when
complete.
• Future plan is for a web interface for configuration rather than
tweaking the yml config file.

26
Example: BuffetConfig.yml
• This is a subset of the full file.

Here we can enable payload artifact inflation and

set a target size. We can also ask that the
resulting artifact’s PE checksum be corrected. In
this case, Zulu Foxtrot will read the options and
act accordingly.

27
Expected Result on single pipeline run
• Three different ZIP files get produced
• Payloads and manifest information
• Click Once Packaging
• MSIX/APPX Packaging
• With payload inflation target size of 20Mb (modest and
insufficient), typical artifact delivery stats are:
• 57 binary payload artifacts + Manifest in HTML/TXT
• 19 EXE’s both managed/unmanaged
• 26 DLL’s both managed/unmanaged
• 4 RDLL’s (reflexsive native/unmanaged)
• 6 XLL’s (Excel DLL)
• 9 MSIX/APPX packages
• 9 Click Once packages
• Over 1GB of data produced.

28
29
Advantages of CI/CD
• Unique binary artifact delivery every single pipeline run
• Large diversity of payload options
• Unique encryption keys for different aspects of binary
artifacts
• Unique shellcode encryption
• Unique source code obfuscation for C# code
• Unique assembly/machine code encryption for system call libraries
• Flexibility to quickly implement new techniques
• Easy to add payload post processing steps
• Static analysis of binary artifact becomes much more difficult.

30
Disadvantages
• LOTS of dependencies
• Multiple compilers
• Multiple tool versioning
• Ie: mage.exe, makeappx.exe, cl.exe, msbuild.exe, go.exe …
• Speed of pipeline troubleshooting
• Average pipeline run takes 3 – 10 minutes.
• Bloat: 8 different Python3 scripts to maintain on dynamic pipelines
• Sensitivity to configuration mistakes
• Some simple errors in the configuration YML can blow up the pipeline.
• Operator use of “git” is not a good user interface.
• Operator overwhelm factor
• You have so many toys, which one do I use?

31
I recently “polled” our Pen Testers
• “How many hours would it take you to manually produce
needed evasive binary artifacts you get from our pipeline?”
• Selected responses:
• Overall: 8 – 16 hours time saved per customer engagement
• My dev skills are limited, our pipeline is invaluable
• 2 days fumbling around the Internet looking for POC code and
hoping it works
• Your work has changed how I operate. I now focus more on the op
itself.

32
Tools: When developing …
• WinDBG (MS Store App version has nicer GUI)
• System Informer (formally “Process Hacker”)
• https://systeminformer.sourceforge.io/
• SysInternals Process Explorer
• API Monitor
• http://www.rohitab.com/apimonitor
• Visual Studio / Mono / Mingw-w64 cross compiler
• Python – for quick and dirty POC’s + pipeline tool dev.
• PE-Sieve
• https://github.com/hasherezade/pe-sieve
• DnSpy
• https://github.com/dnSpy/dnSpy
• Resource Hacker
• http://www.angusj.com/resourcehacker/

33
Software: .NET Obfuscation
• I use a custom source code templating system
• and Compress/Encrypted Assemblies inside of artifact
• In C# reflective loader: “Assembly.Load()”
• Useful to patch AMSI for bypass of scanning.
• Many projects out there (lean towards MSIL processing):
• https://github.com/yck1509/ConfuserEx
• https://github.com/Accenture/Codecepticon
• https://github.com/obfuscar/obfuscar
• https://github.com/0xb11a1/yetAnotherObfuscator
• https://github.com/BinaryScary/NET-Obfuscate

34
Software: NetClone / Adaptive DLL
Hijacking
• Capture DLL code functionality while proxying function call to
legit DLL
• Great technique for apps that still have DLL path search issues
• “Teams” – by dropping Dll in user appdata directory no less
• Ahem… Microsoft whats up with that?
• https://www.netspi.com/blog/technical/adversary-
simulation/adaptive-dll-hijacking/
• https://github.com/monoxgas/Koppeling

35
Software: SysWhispers3
• C library for implementing both direct and indirect kernel
syscalls.
• Integrate into pipeline to dynamically regenerate upon each
compilation
• Note: direct syscalls more challenging now…
• If the CPU “syscall” instruction comes from a non-NTDLL.DLL
mapped memory range, then it is suspicious.

• https://github.com/klezVirus/SysWhispers3

36
Wrapping Up
• By implementing a CI/CD approach, we:
• Save valuable operator time
• Bring malware to a wider group of pen testers
• Increase flexibility for implementing post processing packaged
techniques
• Increase consistency in pen tester initial access operations
• Gain visibility into what artifacts are being produced and deployed
• Provide a consistent platform for further malware
development/deployment
• Provide a platform to integrate new malware generation frameworks
• Keep providing sporting fun in the offense/defense arms race.

37
More Resources
• https://docs.gitlab.com/ee/ci/
• https://defuse.ca/online-x86-assembler.htm
• https://www.blackhillsinfosec.com/avoiding-memory-scanners/
• https://www.elastic.co/fr/security-labs/upping-the-ante-detecting-in-memory-threats-
with-kernel-call-stacks
• https://www.cobaltstrike.com/blog/behind-the-mask-spoofing-call-stacks-dynamically-
with-timers
• https://jsecurity101.medium.com/uncovering-windows-events-b4b9db7eac54
• https://codemachine.com/articles/x64_deep_dive.html
• https://www.crowdstrike.com/blog/crowdstrikes-advanced-memory-scanning-stops-
threat-actor/
• https://securityintelligence.com/x-force/direct-kernel-object-manipulation-attacks-etw-
providers/
• https://www.mdsec.co.uk/2023/09/nighthawk-0-2-6-three-wise-monkeys/

38
Questions?

https://github.com/yoda66/MAAS  A “paint by numbers” DIY approach.

This Photo by Unknown Author is licensed under CC BY-NC

39