Intrusion Detection With Splunk (Real-world Scenario)

Ingesting Data Sources

To proceed, we need access to data we can analyze and use for threat hunting. There are a few sources we can turn to. One source that Splunk provides, along with installation instructions, is BOTS. Alternatively, https://www.logs.to is a handy utility providing us with dummy logs in JSON format. If you upload data from logs.to, ensure your source type correctly extracts the JSON by adjusting the Indexed Extractions setting to JSON when crafting a new source type before uploading the JSON data.

By setting the time picker to All time and submitting the query below, we can retrieve all accessible events.

index="main" earliest=0

Searching Effectively

If you're new to Splunk, you might notice that certain queries take considerable time to process and return data, particularly when dealing with larger, more realistic data sets. Effective threat hunting in any SIEM hinges on crafting the right queries and targeting relevant data.

Let's dive into our data. Our first objective is to see what we can identify within the Sysmon data. We'll start by listing all our sourcetypes to approach this as an unknown environment from scratch. Run the following query to observe the possible sourcetypes

index="main" | stats count by sourcetype

This will list all the sourcetypes available in your Splunk environment. Now let's query our Sysmon sourcetype and take a look at the incoming data.

index="main" sourcetype="WinEventLog:Sysmon"

We can delve into the events by clicking the arrow on the left.

The extracted fields aid us in crafting more efficient searches. Here's the reasoning.

There are several ways we can run searches to achieve our goal, but some methods will be more efficient than others. We can query all fields, which essentially performs regex searches for our data assuming we don't know what field it exists in. For demonstration, let's execute some generalized queries to illustrate performance differences. Let's search for all possible instances of uniwaldo.local.

This should return results rather quickly. It will display any instances of this specific string found in any and all sourcetypes the way we ran it. It can be case insensitive and still return the same results. Now let's attempt to find all instances of this string concatenated within any other string such as "myuniwaldo.localtest" by using a wildcard before and after it.

index="main" *uniwaldo.local*

You'll observe that this query returns results much more slowly than before, even though the number of results is exactly the same! Now let's target this string within the ComputerName field only, as we might only care about this string if it shows up in ComputerName. Because no ComputerName only contains this string, we need to prepend a wildcard to return relevant results.

index="main" ComputerName="*uniwaldo.local"

You'll find that this query returns results much more swiftly than our previous search. The point being made here is that targeted searches in your SIEM will execute and return results much more quickly.

Embracing The Mindset Of Analysts, Threat Hunters, & Detection Engineers

We can utilize public resources such as the Microsoft Sysinternals guide for Sysmon. and apply the same approach to identify all Sysmon EventCodes prevalent in our data with this query.

index="main" sourcetype="WinEventLog:Sysmon" | stats count by EventCode

Our scan uncovers 20 distinct EventCodes. Before we move further, let's remind ourselves of some of the Sysmon event descriptions and their potential usage in detecting malicious activity.

Based on these EventCodes, we can perform preliminary queries. As previously stated, unusual parent-child trees are always suspicious. Let's inspect all parent-child trees with this query.

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | stats count by ParentImage, Image

We're met with 5,427 events, quite a heap to manually sift through. We have choices, weed out what seems benign or target child processes known to be problematic, like cmd.exe or powershell.exe. Let's target these two.

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 (Image="*cmd.exe" OR Image="*powershell.exe") | stats count by ParentImage, Image

The notepad.exe to powershell.exe chain stands out immediately. It implies that notepad.exe was run, which then spawned a child powershell to execute a command. The next steps? Question the why and validate if this is typical.

We can delve deeper by focusing solely on these events.

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 (Image="*cmd.exe" OR Image="*powershell.exe") Image="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"

We see the ParentCommandLine (just notepad.exe with no arguments) triggering a CommandLine of powershell.exe seemingly downloading a file from a server with the IP of 10.0.0.229!

Our path now forks. We could trace what initiated the notepad.exe, or we could investigate other machines interacting with this IP and assess its legitimacy. Let's unearth more about this IP by running some queries to explore all sourcetypes that could shed some light.

Among the few options in this tiny 5-machine environment, most will just inform us that a connection occurred, but not much more.

index="main" 10.0.0.229 sourcetype="linux:syslog"

Here we see that based on the data and the host parameter, we can conclude that this IP belongs to the host named waldo-virtual-machine on its ens160 interface. The IP seems to be doing some generic stuff.

This finding indicates that our machine has engaged in some form of communication with a Linux system, notably downloading executable files through PowerShell. This sparks some concerns, hinting at the potential compromise of the Linux system as well! We're intrigued to dig deeper. So, let's initiate another inquiry using Sysmon data to unearth any further connections that might have been established. index="main" 10.0.0.229 sourcetype="WinEventLog:sysmon" | stats count by CommandLine

At this juncture, alarm bells should be sounding! We can spot several binaries with conspicuously malicious names, offering strong signals of their hostile intent. We would encourage you to exercise your investigative skills and try to trace these attacks independently – both for practice and for the thrill of it!

From our assessment, it's becoming increasingly clear that not only was the spawning of notepad.exe to powershell.exe malicious in nature, but the Linux system also appears to be infected. It seems to be instrumental in transmitting additional utilities. We can now fine-tune our search query to zoom in on the hosts executing these commands.

index="main" 10.0.0.229 sourcetype="WinEventLog:sysmon" | stats count by CommandLine, host

Our analysis indicates that two hosts fell prey to this Linux pivot. Notably, it appears that the DCSync PowerShell script was executed on the second host, indicating a likely DCSync attack. Instead of making an assumption, we'll seek validation by designing a more targeted query, zeroing in on the DCSync attack in this case. Here's the query.

index="main" EventCode=4662 Access_Mask=0x100 Account_Name!=*$

Now, let's dissect the rationale behind this query. Event Code 4662 is triggered when an Active Directory (AD) object is accessed. It's typically disabled by default and must be deliberately enabled by the Domain Controller to start appearing. Access Mask 0x100 specifically requests Control Access typically needed for DCSync's high-level permissions. The Account_Name checks where AD objects are directly accessed by users instead of accounts, as DCSync should only be performed legitimately by machine accounts or SYSTEM, not users. You might be wondering how we can ascertain these are DCSync attempts since they could be accessing anything. To address this, we evaluate based on the properties field.

Properties: Control Access

{1131f6aa-9c07-11d1-f79f-00c04fc2dcd2}

{19195a5b-6da0-11d0-afd3-00c04fd930c9}

Upon researching, we find that the first one is linked to DS-Replication-Get-Changes-All, which, as per its description, "...allows the replication of secret domain data".

This gives us solid confirmation that a DCSync attempt was made and successfully executed by the Waldo user on the UNIWALDO domain. It's reasonable to presume that the Waldo user either possesses Domain Admin rights or has a certain level of access rights permitting this action. Furthermore, it's highly likely that the attacker has extracted all the accounts within the AD as well! This signifies a full compromise in our network, and we should consider rotating our krbtgt just in case a golden ticket was created.

However, it's evident that we've barely scratched the surface of the attacker's activities. The attacker must have initially infiltrated the system and undertaken several maneuvers to obtain domain admin rights, orchestrate lateral movement, and dump the domain credentials. With this knowledge, we will adopt an additional hunt strategy to try and deduce how the attacker managed to obtain Domain Admin rights initially.

We are aware of and have previously observed detections for lsass dumping as a prevalent credential harvesting technique. To spot this in our environment, we strive to identify processes opening handles to lsass, then evaluate whether we deem this behavior unusual or regular. Fortunately, Sysmon event code 10 can provide us with data on process access or processes opening handles to other processes. We'll deploy the following query to zero in on potential lsass dumping.

index="main" EventCode=10 lsass | stats count by SourceImage

We prefer sorting by count to make the data more comprehensible. While it's not always safe to make assumptions, it's generally accepted that an activity occurring frequently is "normal" in an environment. It's also harder to detect malicious activity in a sea of 99 events compared to spotting it in just 1 or 5 possible events. With this logic, we'll begin by examining any conspicuous strange process accesses to lsass.exe by any source image. The most noticeable ones are notepad (given its absurdity) and rundll32 (given its limited frequency). We can further explore these as we usually do.

index="main" EventCode=10 lsass SourceImage="C:\Windows\System32\notepad.exe"

We are investigating the instances of notepad opening the handle. The data at hand is limited, but it's clear that Sysmon seems to think it's related to credential dumping. We can use the call stack to glean additional information about what triggered what and from where to ascertain how this attack was conducted.

To the untrained eye, it might not be immediately apparent that the callstack refers to an UNKNOWN segment into ntdll. In most cases, any form of shellcode will be located in what's termed an unbacked memory region. This implies that ANY API calls from this shellcode don't originate from any identifiable file on disk, but from arbitrary, or UNKNOWN, regions in memory that don't map to disk at all. While false positives can occur, the scenarios are limited to processes such as JIT processes, and they can mostly be filtered out.

Questions.

Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through an SPL search against all data the other process that dumped lsass. Enter its name as your answer. Answer format: _.exe

Steps:

SPL search query:

index="main" EventCode=10 lsass | stats count by SourceImage

index="main" EventCode=10 lsass SourceImage="C:\Windows\system32\rundll32.exe"

Answer: rundll32.exe

Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through SPL searches against all data the method through which the other process dumped lsass. Enter the misused DLL's name as your answer. Answer format: _.dll

Steps:

index="main" *lsass *.dll OriginalFileName="*" | stats count by OriginalFileName

OR

index="main" sourcetype="WinEventLog:Sysmon" lsass *.dll CommandLine="*" | stats count by CommandLine

Answer: comsvcs.dll

Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through an SPL search against all data any suspicious loads of clr.dll that could indicate a C# injection/execute-assembly attack. Then, again through SPL searches, find if any of the suspicious processes that were returned in the first place were used to temporarily execute code. Enter its name as your answer. Answer format: _.exe

Steps:

SPL search:

index="main" sourcetype="WinEventLog:Sysmon" clr.dll Image="*" | stats count by Image

index="main" sourcetype="WinEventLog:Sysmon" *clr.dll Image="C:\Windows\System32\rundll32.exe"

Image: C:\Windows\System32\rundll32.exe ImageLoaded: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

Answer: rundll32.exe

Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through SPL searches against all data the two IP addresses of the C2 callback server. Answer format: 10.0.0.1XX and 10.0.0.XX

Steps:

SPL search:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=3 DestinationIp="*" Image="C:\Windows\system32\rundll32.exe"

Answer: 10.0.0.186 and 10.0.0.91

Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through SPL searches against all data the port that one of the two C2 callback server IPs used to connect to one of the compromised machines. Enter it as your answer.

Steps:

if it's a C2, it's most likely sending commands from the C2 IP to the target IP.

The C2 server IP is the IP address of the server controlled by the attacker, which sends commands to the compromised devices (bots) in the botnet.

  • The target IP is the IP address of the compromised device (bot) that receives commands from the C2 server.

In a typical C2 communication flow:

  1. The attacker sends commands from the C2 server IP to the target IP.

  2. The compromised device (bot) at the target IP receives the commands and executes them.

  3. The bot may send data or responses back to the C2 server IP.

So, in the context of the SPL search query, we're looking for events where the C2 server IP (ip.src) is sending commands to the target IP (ip.dst) over TCP or UDP protocols, which are commonly used for C2 communication.

SPL search:

index=main EventCode=3 SourceIp="10.0.0.186" DestinationPort="*"

OR

index=main (10.0.0.186 OR 10.0.0.91) AND EventCode=3

OR

index=main (10.0.0.186 OR 10.0.0.91) AND EventCode=3 | stats count by DestinationPort

Answer: 3389

PreviousSplunk ApplicationsNextDetecting Attacker Behavior With Splunk Based On TTPs

Last updated