Introduction

A problem we all face when using threat intelligence data is getting rid of false positives in our data feeds. On the other hand, reporting of true positives is equally important as it allows to increase the level of trust in an indicator.

This post describes how you can report false and true positives from an analyst tool (Kibana) to MISP.

You can then use other tooling to remove attributes when they are above a certain false positive reporting threshold, or increase the severity level if they are above a true positive reporting threshold. Note that you can also use a PyMISP show_sightings.py script to get a report of the sightings.

To have this mechanism work we use the MISP sighting system and setup Kibana with the Url field formatter.

Sightings

Sightings is a system that allows people to react on threat attributes. It allows you to tell when you have seen a given attribute, giving it more credibility but it also supports you in signalling false positives, and give an expiration date for some attributes.

MISP supports two methods for sightings. The default is the built-in sightings solution, which tracks each individual sighting. An alternative is sightings-db by Sébastien Tricaud. Sightings DB supports the reporting of sightings at a very high speed, but it will only track first, last and count of sightings. This post is about the built-in sighting system. Sightings are not necessarily restricted to one server, you can sync them between servers by using PyMISP, see : Sync sightings between MISP instances.

Security Onion

Security Onion is a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It includes Elasticsearch, Logstash, Kibana, Snort, Suricata, Zeek (formerly known as Bro), Wazuh, Sguil, Squert, CyberChef, NetworkMiner, and many other security tools.

The interface used by security analysts to interact with Security Onion is most often Kibana, as part of the Elastic stack. For this post I focus on the Bro Intel dashboard of Security Onion but the same methodology can be used for other dashboard.

Reporting method

The reporting is done via two links (fields) in the dashboard, one for false positive and another one for true positive. The links point to a PHP dispatcher script running on the Security Onion host, which will then send the request to the MISP server.

Logstash configuration

The first step is to configure Elastic so that it supports our two new fields. Note that this works best if your Elastic cluster (or at least the Bro-indexes) do not contain data.

Copy logstash-template.json from /etc/logstash to /etc/logstash/custom. The ‘custom’ directory is the directory used by Security Onion with all customizations to Logstash. Edit the file and add the two fields in the mappings/doc section. Search for example for the field name “indicator” and then add the config

        "sighting_fp":{
          "type":"text",
          "fields":{
            "keyword":{
              "type":"keyword"
            }
          }
        },
        "sighting_tp":{
          "type":"text",
          "fields":{
            "keyword":{
              "type":"keyword"
            }
          }
        },

Then copy the Bro preprocessor 1124_preprocess_bro_intel.conf file also to /etc/logstash/custom. Edit the file and add below the line “rename => { “seen.indicator” => “indicator” }” these Logstash mutations. They will add the two necessary fields, sightings_fp and sightings_tp, to the event data.

add_field => { "sighting_fp" => "%{indicator}"}
add_field => { "sighting_tp" => "%{indicator}"}

After this change, you can restart Logstash.

sudo so-logstash-restart

Kibana configuration

The next step is to configure the Url field formatter. In Kibana go to Management > Kibana > Index Patterns. Then choose the index pattern *:logstash-*, search for ‘sigh’ (or sighting) and click on the Edit (pencil) button. Do this for both sighting_fp and sighting_tp.

Then set the Format to Url and add these values

• URL template : /report_sighting.php?indicator={{value}}&source=securityonion_bro&sighting_type=1
• Label template : {{value}}

The way this works is that Kibana will replace the value (aka the indicator) which is stored by Logstash in both the fields sighting_fp and sighting_tp with the URL supplied in URL template. It will then substitute the string {{value}} with the original value. This happens in real time. If you change the URL after the event is logged, then Kibana will still display the new URL.

PHP ‘Dispatcher’ script

The PHP dispatcher script needs to be placed in the web directory of Security Onion, /var/www/so. You can basically name it whatever you want but the name needs to correspond to the script called in the URL template. The script supports a couple of options and basically has the format

/report_sighting.php?(indicator|attribute_id|attribute_uuid)={{value}}[&source=__source__]&sighting_type=(0|1|2)

• If you supply indicator, then it will report a global sighting for an attribute. What this means is that if you have for example an attribute “1.1.1.1” and it exists in different MISP threat events, then using this option you can report the false positive for all the events where this attributes exist;
• The parameters attribute_id or attribute_uuid on the other hand allow you to specify an exact attribute ID or UUID.
• You can indicate a true positive by including the value sighting_type=0. A false positive is reported by sighting_type=1 and an expired attribute can be reported by using sighting_type=2.
• The source paramater is an optionally string that indicates the source sighting, for example Bro or Snort.

Key element

The key element to make this method work is that you report the MISP attribute value, the attribute ID or the attribute UUID somewhere in the event data logged to Elastic. Without any of these three values, it’s not possible to setup the reporting URL.

MISP Sightings

In case you have not yet used MISP sightings, they are visible via the event data.

Apache authentication

By default the PHP dispatcher script can be called by someone not authenticated to Security Onion. Depending on your use case this is not desirable. You can prevent this by adding this section to the Apache configuration. This will require a user to first authenticate to Security Onion.

                <Location /report_sighting.php>
                        AuthType form
                        AuthName "Security Onion"
                        AuthFormProvider external
                        AuthExternal so-apache-auth-sguil
                        Session On
                        SessionCookieName session path=/;httponly;secure;
                        SessionCryptoPassphraseFile /etc/apache2/session
                        ErrorDocument 401 /login-inline.html
                        AuthFormFakeBasicAuth On
                        Require valid-user
                </Location>

PHP script

The PHP script is available via Github at https://github.com/cudeso/tools/tree/master/sighting_kibana_to_misp but also below.

<?php
session_start();

$misp_key = "<yourkey>";
$misp_server = "<misp_server>";
$misp_verify_cert = False;
$sightings_url = $misp_server . "/sightings/add";
$source = "";
$sighting_type = 0;
$selector = "";
$value = "";

// We are either called by attribute_id, attribute_uuid or by value
if (isset($_REQUEST["attribute_id"])) {
    $value = $_REQUEST["attribute_id"];
    $selector = "id";
}
elseif (isset($_REQUEST["attribute_uuid"])) {
    $value = $_REQUEST["attribute_uuid"];
    $selector = "uuid";
}
elseif (isset($_REQUEST["indicator"])) {
    $value = $_REQUEST["indicator"];
    $selector = "value";
}

if (isset($value)) {
    if (isset($_REQUEST["source"])) {
    	$source = filter_var($_REQUEST["source"], FILTER_SANITIZE_STRING);
    }

    if (isset($_REQUEST["sighting_type"])) {
	    $sighting_type = abs(filter_var($_REQUEST["sighting_type"], FILTER_SANITIZE_NUMBER_INT));
        if ($sighting_type > 2) $sighting_type = False;  // 0: sighting ; 1: false positive ; 2: expiration
    }

    $ch = curl_init($sightings_url);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, $misp_verify_cert);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, $misp_verify_cert);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array("Accept: application/json","Content-Type: application/json","Authorization: ".$misp_key)); 

    $jsonData = array(
        "$selector" => $value,
        "source" => $source,
        "type" => $sighting_type 
    );

    $jsonDataEncoded = json_encode($jsonData);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded);
    $result = curl_exec($ch);

    ?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Report Sighting</title>
</head>
<body><h3>Sighting sent to <?php echo $misp_server; ?></h3>
</body>
</html>
<?php  
}
else {
    echo "<h3>Sighting not sent, invalid parameters supplied.</h3>";
}
?>