Creating Responders in The Hive

The Hive is an  open source Security Incident Response Platform (SIRP) that has gained quite some popularity over the last few years. One of the many reasons is the link with Cortex and its Analyzers and Responders. Analysts can automate the response to existing cases by initiating one or more Responders. This blog will show how a custom Responder can be used to assist in a user awareness program by creating automated responses to phishing related cases in The Hive.

The creators of The Hive already provide some info regarding the creation of Responders, although still some things are left undocumented, and some pitfalls are not mentioned. This blog is not meant to replace this documentation, but is more of an add-on:  we hope to give you some quick wins here that will save you some time, or just clarify things by giving a different example. If you haven’t read through the official documentation please check it first.

An easy way to start experimenting writing your own Responders with The Hive is making use of the Virtual Machine provided here. This way you can test and tweak all you want without risking to corrupt production data.

The use case

The reporting of suspicious emails by users is a key part of any user awareness program.  But as import as the user’s submission is the feedback he/she receives from analysts. The feedback would depend on the reported email being a true positive (a real phish) ,  or a false positive  (ordinary spam or a legit email). In case of a true positive , users need to receive some form of appreciation if you want them to keep doing it. So an automated “congratulation” email would be nice.

In case of a false positive , users still need to be thanked for their submission, however some guidelines for them to distinguish between a phish and an ordinary email or spam could reduce the workload of your analysts quite a bit. So actually we have 3 types of responses, which we could implement by creating  3 different ‘flavors’ of the same Responder – but we’re going to take a different approach here.

The basics

For your Responder to work, you would at least need to provide 2 files (which you’ll find an example at the end of the blog) :

• A JSON configuration file
• A Python file with the code itself

In this example we’re using Python but The Hive also supports Perl , Ruby and Scala

The most important fields in the JSON would be :

dataTypeList

This field specifies whether the Responder is applicable to a case or an alert or an observable. The difference lies in the input data that is passed on from The Hive to your Responder. Your script will be receiving a JSON structure that represents a case , an alert or a single observable from The Hive. Please note that depending on whether your Responder is working on a case or an alert, different attributes might be available to work on. For example: when working on an alert, the observables are directly available, but when working on a case, strangely enough there’s no direct way to access the observables from the case and you need a workaround which I will show you later.

In this example we opted to make the Responder applicable to cases because cases can be created and tweaked manually through the GUI, which allows for easy experimenting.

Let’s go over the different fields to configure!

command

The path to your script, relative from the ‘Responders’ folder, so in our example :

PhishFeedback/phish_feedback.py

Please note that the location where all responders should be placed is by default :

/opt/Cortex-Analyzers/responders/

But you can change this default path in the Cortex configuration file:

/etc/cortex/application.conf

When you are running The Hive in Docker using images from TheHive-Project , this would be inside the Cortex container and you would preferably map this folder to your host to make all your Responders persistent.

Another important remark is that all python scripts should be executable in the context of the ‘cortex’ user. You can achieve this by either setting the ownership to the cortex user or add execution rights for ‘others’. In your VM type :

chown 999 phish_feedback.py & chgrp 999 phish_feedback.py

OR

chmod o+x phish_feedback.py

config

This field allows you to define different ‘flavors’ of the same Responders: you can have one script that delivers multiple services, which will show up as different responders or analyzers. A good example of how to achieve this is the VirusTotal Analyzer that comes default with The Hive. Although it’s an Analyzer the concept of flavors works for both Analyzers and Responders. Check for the usage of the ‘service’ field in both the JSON files and the Python file.

configurationItems

Here you would typically want to define all parameters that need to be set by the users through the Cortex GUI. A small tip : when finishing your JSON files, double check with an (online) JSON validator whether the syntax is correct, because if not, Cortex will NOT give you any direct indication about this – instead your Responder just won’t show up in the list or won’t do anything.

Start using your Responder

So how do you actually start using your new Responder ?

Go to your cortex instance and in the top menu click  ‘Organisation’. In the menu-bar from your organization select ‘Responders’. There will be a button to ‘Refresh Responders’. If everything works well your new Responder should show up (if not check the attention points mentioned before)

Please note that if you don’t have similar menus available it probably means you are logged on as the superadmin of Cortex. To be able to configure Analyzers / Responders you need at least one organization defined in Cortex and one user for that organization under which you need to log in.

In addition, when you made changes to the configuration file of an already existing responder, you might want to disable the existing one, click the refresh button and then re-enable the Responder. It seems the only way for Cortex to pick up on your changes.

When enabling a Responder , you need to provide the necessary parameters as shown below, in our case they’re related to the sending of emails.

Also note the Max TLP and Max PAP fields: when you’re alert or case has a TLP or PAP level higher than the one configured here, your Responder will not work !

Once you were able to enable your Responder and have configured all required parameters your Responder should show up in the list when you click the “Action” icon on a case in The Hive. You’re now finally able to try out your new Responder and you will hopefully receive a notification that it ran successfully and you’ll see that a tag was added to the case saying “mail sent”.

However, when you’re testing a newly created Responder, chances are quite high that something goes wrong and the output in The Hive does not really tell you a lot about the reason why. Then it’s probably a good idea to check the ‘Jobs History’ in Cortex.  For each job you will see whether it ended successfully or not and it might give you some traceback logs from Python describing the error.

But sometimes the job history only shows “Invalid Output” as error message. Then it’s time to search the logs of The Hive or Cortex which will hopefully give you more insight in what’s going wrong.

The Hive application logs are stored under :

/var/log/thehive/application.log

The Cortex logs are under :

/var/log/cortex/application.log

And when debugging Cortex issues you would probably want to change the logging settings.

In /etc/cortex/logback.xml at the top set :

configuration debug="true"

To start using the new debugging settings , restart the cortex service with :

sudo service cortex restart

When using Docker , restart the Cortex container after modifying the logback.xml file.

Please show me the code now

Almost there! For you to be able to use the example there’s one convention you need to follow: the email address of the user who submitted the phishing report is expected to be available as a tag in the case.

user_email:user@domain

For testing you can just create an empty case and a manually add the tag. Also, in the code I’m checking whether the title of the case starts with “Phishing report” otherwise it returns an error message: this is just to avoid that you accidentally run the Responder on a different type of case.

Remember that I mentioned you could either use different flavors of the same Responder to send out different mails to the user or use another approach. The other approach is making use of “case custom fields”. These fields allow users to add data to cases in the form of strings, numbers, booleans or dates. You can create lists of acceptable values to limit your analysts’ choices to legitimate data. These fields can be associated with case templates or can be added to any case manually.

A big advantage is that you can create dashboards or statistics based on these custom fields. In our case we could create some fancy dashboard that shows the relation between all submitted Phishing Reports and their resolution status: phish , spam or legit email.

In order to add such a field you need to be an Admin user. In the Admin menu , click “Case custom Fields”and click the “Add Custom Field” button and fill in the values as shown below. Make sure you copy the ‘Possible values’ identically because we’re using them in our Responder code to decide which type of email we should send to the user.

You can download a zip with both the configuration file and the python code here.

Configuration file:

{
  "name": "Phish_Feedback",
  "version": "1.0",
  "author": "Tom Asselman NVISO",
  "url": "https://github.com/NVISO-BE/blogposts/tree/master/TheHive/Responders",
  "license": "AGPL-V3",
  "description": "Sends an email to the user with feedback about a phishing report",
  "dataTypeList": ["thehive:case"],
  "command": "PhishFeedback/phish_feedback.py",
  "baseConfig": "PhishFeedback",
  "configurationItems": [
    {
      "name": "from",
      "description": "email address from which the mail is send",
      "type": "string",
      "multi": false,
      "required": true
    },
    {
      "name": "smtp_host",
      "description": "SMTP server used to send mail",
      "type": "string",
      "multi": false,
      "required": true,
      "defaultValue": "localhost"
    },
    {
      "name": "smtp_port",
      "description": "SMTP server port",
      "type": "number",
      "multi": false,
      "required": true,
      "defaultValue": "25"
    },
    {
      "name": "smtp_user",
      "description": "SMTP server user",
      "type": "string",
      "multi": false,
      "required": true,
      "defaultValue": "user"
    },
    {
      "name": "smtp_pwd",
      "description": "SMTP server password",
      "type": "string",
      "multi": false,
      "required": true,
      "defaultValue": "pwd"
    }
  ]
}

Python code:

#!/usr/bin/env python3
# encoding: utf-8
from cortexutils.responder import Responder
import smtplib
import ssl
import requests
import json
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

class Phish_Feedback(Responder):

    def __init__(self):
        Responder.__init__(self)
        self.smtp_host = self.get_param(
            'config.smtp_host', 'localhost')
        self.smtp_port = self.get_param(
            'config.smtp_port', '25')
        self.smtp_user = self.get_param(
            'config.smtp_user', 'user')
        self.smtp_pwd = self.get_param(
            'config.smtp_pwd', 'pwd')
        self.mail_from = self.get_param(
            'config.from', None, 'Missing sender email address')

        self.mode = self.get_param('data.customFields.phishFeedback.string', None)
        if not self.mode:
             self.report({"Error" : "Custom field 'phish_feedback' was not set"})


    def run(self):
        Responder.run(self)
        title = self.get_param('data.title', None, 'Title is missing')
        if title.find("Phishing report") == -1:
                self.report({"response" : "Phish_Feedback is only applicable to cases which title starts with 'Phishing report'"})
        # Search recipient address in tags
        tags = self.get_param('data.tags', None, 'recipient address not found in tags')
        mail_tags = [t[11:] for t in tags if t.startswith('user_email:')]
        if mail_tags:
            mail_to = mail_tags.pop()
        else:
			self.report({"Error" : "Recipient address not found in observable"})

        if self.mode == "Phishing":
                description = """\rGreetings,
                         \rYou've recently reported a spear-phishing attempt to us'.
                         \rWe would like to congratulate you for spotting this attempt beacause it was indeed a malicious email!
                         \rThank you,
                        """

        elif self.mode == "Legit email":
                description = """\rGreetings,
                         \rYou've recently reported a spear-phishing attempt to us.
                         \rWe thank you for your submission , however  we would like to inform you that this email was legitimate. Can you please tell us why you thought it was malicious ?
                         \rIf you want tips on how to detect phishing emails , please check:
                         \rhttps://campagne.safeonweb.be/en/phishing
                         \rThank you,
                    """

        elif self.mode == "Spam":
                description = """\rGreetings,
                         \rYou've recently reported a spear-phishing attempt to us.
                         \rWe thank you for your submission , but we would like to inform you that this email was not malicious but just ordinary spam.
                         \rIf you would like tips on how to detect phishing emails , please check:
                         \rhttps://campagne.safeonweb.be/en/phishing
                         \rThank you
                         \r
                    """

        msg = MIMEMultipart()
        msg['Subject'] = "Phishing report feedback"
        msg['From'] = self.mail_from
        msg['To'] = mail_to
        msg.attach(MIMEText(description, 'plain'))

        context = ssl.SSLContext(ssl.PROTOCOL_TLS)
        server = smtplib.SMTP(self.smtp_host, self.smtp_port)
        server.starttls(context=context)
        server.login(self.smtp_user, self.smtp_pwd)
        server.sendmail(self.mail_from, mail_to, msg.as_string())
        server.quit()

        self.report({'Status': 'Mail sent'})

    def operations(self, raw):
        return [self.build_operation('AddTagToCase', tag='mail sent')]

if __name__ == '__main__':
        Phish_Feedback().run()

As mentioned before: a Responder working on a case as input object does not have a direct way to access the Observables from that case as opposed to when working on an alert . The following code shows you a workaround example how you can still reach them and use them in your code.

It basically retrieves the case_id from your input data and then uses a The Hive API call to search for the artifacts of that case and builds a dictionary out of them. You need to replace the APIKEY and hive_server variables with legit values.

APIKEY = 'Your API-KEY For The Hive"
case_id = self.get_param('data.caseId', None, 'CaseId is missing')
internal_case_id = self.get_param('data.id', None, 'Internal case id is missing')
hive_server = "https://your_hive_instance"
url= hive_server + "/api/case/artifact/_search"
header = {'Authorization': 'Bearer ' + APIKEY}
payload = {
                 "query": { "_parent": { "_type": "case", "_query": { "_id": internal_case_id } } } ,
                 "range" : "all"
          }
response = requests.post(url, json=payload, headers=header, verify=False)
#create a dictionary for our use case out of it
observable_dict = {}
observables = json.loads(response.text)
for observable in observables:
	observable_key=observable.get("dataType","dummy")
    observable_dict[observable_key] = observable.get("data","dummy")

      

Conclusion

I hope this blog has helped you to start writing your own Responders for The Hive . The number of different types of Responders you can create is only limited by your imagination and if done right, it can save lots of time for your SOC analysts and increase their productivity.