Detecting suspicious child processes using ee-outliers and Elasticsearch

In this post, we will illustrate how ee-outliers can be used to detect suspicious child processes. This can be a very helpful way of spotting malicious endpoint activity during our Threat Hunting activities. A few examples where detecting suspicious child processes could help us:

• Detection of a malicious Microsoft Word file spawning cmd.exe
• Detection of a 0-day exploit embedded within an e-mail in Outlook spawning powershell
• … and many more potential scenarios.

If this is the first time you will be using ee-outliers, we strongly suggest you read through the documentation on GitHub.

Let’s get started!

Collecting the right data

To collect the right endpoint data, we will rely on osquery. We use the following query to periodically gather all the information on the workstations and servers we want to monitor:

 SELECT p.*, h.*, u.username, u.type as usertype, pp.name as parentname, pp.cmdline as parentcmdline FROM processes p LEFT JOIN users u ON p.uid = u.uid LEFT JOIN processes pp ON p.parent = pp.pid LEFT JOIN hash as h ON p.path = h.path;

This query will combine several data points around process execution including the user, process and parent process names, PID and PPID, etc. Using this information, we can start looking for outliers; in this specific case, we will look for suspiciously launched child processes.

Building the detection use case

For the detection use case, we have defined the following in the ee-outliers configuration file:

##############################
# TERMS - RARELY SEEN CHILD PROCESS
##############################
# detect processes with rare children
[terms_rare_childname]
es_query_filter=tags:endpoint AND meta.command.name:"get_all_processes_enriched" AND _exists_:OsqueryFilter.name AND _exists_:OsqueryFilter.parentname AND NOT OsqueryFilter.parentname.raw:""

aggregator=OsqueryFilter.parentname
target= OsqueryFilter.name
target_count_method=within_aggregator
trigger_on=low
trigger_method=pct_of_avg_value
trigger_sensitivity=1

outlier_type=process execution
outlier_reason=rare child process
outlier_summary=rare child process {OsqueryFilter.name} for {OsqueryFilter.parentname}

run_model=1
test_model=0

The use case can be explained as following:

• The es_query_filter selects all the Elasticsearch events that contain the process information we want. We also exclude all events that don’t have information on the parent process.
• The aggregator will create buckets of events per process parent name (for example adobe.exe) using field name OsqueryFilter.parentname.
• For each aggregator, the total number of times a specific child process name occurs is counted as a target field (for example cmd.exe), using field name OsqueryFilter.name.
• We trigger an outlier in case the spawned process occurs in less than 1% of the average count (for example: in case cmd.exe is observed only once being spawned from adobe.exe, and the average number of observations of all spawned processes by adobe.exe is 200, then cmd.exe will be spotted as an outlier, as 1 < 1% of 200).

Running ee-outliers

Next, we ran this specific use-case using ee-outliers over the last 7 days of events in our test-environment, featuring around 50 workstations and 632.788 total process execution events (the result of the “es_query_filter” argument we defined in our use case).

After running ee-outliers, it’s time to analyze the results!

Analyzing the results

The first thing we do is plot the outlier events in a histogram. ee-outliers also adds a confidence parameter to each outlier, which gives an indication on how far the event diverts from other events handled by the use case.

 

We notice a total of 94 outlier over the last 7 days, out of approximately 600.000 events. This means that about 0,015% of all process events are outliers – sounds like a manageable amount to manually investigate! However, to get to this result, we did add the following lines to the whitelist section in the outliers configuration file, in order to avoid lots of false positives:

rare_child_svchost=^.*rare child process .* for svchost\.exe.*$
rare_child_services=^.*rare child process .* for services\.exe.*$

The resulting outlier summaries generated by ee-outliers are given below:

Let’s look at a few of the outliers and investigate them.

rare child process cmd.exe for chrome.exe
This one immediately caught our attention! Why would chrome spawn a command shell? Looking into more details in the tagged event, we notice the following command is being executed:

C:\windows\system32\cmd.exe /d /c 
"C:\Users\CENSORED\AppData\Local\1password\app\7\\1Password.exe" 
chrome-extension://aomjjhallfgjeglCENSORED/ --parent-window=0 
< \\.\pipe\chrome.nativeMessaging.in.ced1670e4c8e503c > 
\\.\pipe\chrome.nativeMessaging.out.ced1670e4c8e503c

Interesting! This is the 1Password Chrome extension calling 1Password.exe by using cmd.exe. Not a sign of malicious activity, but certainly not a false positive as this is the type of rare activity the use case we built was designed for.

Similar to the previous one, this is Chrome calling the 7zip file manager to automatically extract a downloaded ZIP file.

rare child process TightVNCViewerPortable.exe for explorer.exe
As explorer.exe is the parent process of a lot of processes on Windows, this use case will also automatically flag processes that are rarely observed in your environment! In this specific case, a user was using a portable VNC client to connect to a remote workstation. As this portable VNC viewer is not part of the allowed software on the workstations, it was flagged, as it was only very rarely observed as being spawned from explorer.exe (as defined in our use case: in less than 1% of the average of all events observed that had explorer.exe as a parent). Other examples of this include “rare child process PremierColor.exe for explorer.exe” and “rare child process SUMo.exe for explorer.exe“.

Conclusions

In this blog post, we illustrated how ee-outliers can be used to quickly spot suspicious child processes, that can then be further analysed for malicious activity. This can be used by threat hunters to spot a broad range of anomalous activity, including corporate policy violations (running non-approved software), infected documents (Office documents spawning suspicious processes) all the way to previously unknown attacks (such as a maliciously crafted e-mail in Outlook resulting in a powershell process being executed in the background).

About the author

Daan Raman is in charge of NVISO Labs, the research arm of NVISO. Together with the team, he drives initiatives around innovation to ensure we stay on top of our game; innovating the things we do, the technology we use and the way we work form an essential part of this. Daan doesn’t like to write about himself in third-person. You can contact him at draman@nviso.be or find Daan online on Twitter and LinkedIn.