Today, we are excited to announce we are open-sourcing ee-outliers, our in-house developed framework to detect outliers in events stored in Elasticsearch!
The framework was developed for the purpose of detecting anomalies in security events, however it could just as well be used for the detection of outliers in other types of data. We have been developing ee-outliers in-house for the past year as part of Eagle Eye, our own security monitoring technology built on top of the ELK stack. As we are getting some great results from the outlier detection capabilities of ee-outliers, we have decided to share this work with the community as a way of giving back, as we are convinced this could be useful to a broad range of users: from individuals wanting to analyse their personal data to Security Teams building their own security monitoring capabilities.
The framework makes use of statistical models that are easily defined by the user in a configuration file. Below, you can find an example of a use case that is capable of detecting beaconing TLS connections. Similar use cases can be added by duplicating the use case, changing the query filter and the aggregator fields, and done!
In case the models detect an outlier, the relevant Elasticsearch events are enriched with additional outlier fields. These fields can then be dashboarded and visualized using the tools of your choice (Kibana or Grafana for example). Below, you can find the resulting tagged events based on the SSL beaconing use case above, dashboarded in Kibana.
The possibilities of the type of anomalies you can spot using ee-outliers is virtually limitless. A few examples of types of outliers we have detected ourselves using ee-outliers during threat hunting activities include:
- Detect beaconing (DNS, TLS, HTTP, etc.)
- Detect geographical improbable activity
- Detect obfuscated & suspicious command execution
- Detect fileless malware execution
- Detect malicious authentication events
- Detect processes with suspicious outbound connectivity
- Detect malicious persistence mechanisms (scheduled tasks, auto-runs, etc.)
- …
We welcome all contributions to the project, as well as feature suggestions and feedback. We will soon start sharing example threat hunting use cases, updates & ways in which we and other security teams can leverage ee-outliers on this blog, so keep an eye out for new content!
The project is hosted on Github, and can be found here:
https://github.com/NVISO-BE/ee-outliers
We look forward to your feedback and to see the ways in which you and your teams use ee-outliers to improve your own security monitoring and threat hunting activities!
About the author
Daan Raman is in charge of NVISO Labs, the research arm of NVISO. Together with the team, he drives initiatives around innovation to ensure we stay on top of our game; innovating the things we do, the technology we use and the way we work form an essential part of this. Daan doesn’t like to write about himself in third-person. You can contact him at draman@nviso.be or find Daan online on Twitter and LinkedIn.
3 thoughts on “Announcement: open-sourcing ee-outliers”