This article presents a case study of how Detection Engineering can be improved by applying Detection-as-Code principles in an established, operational environment.
The focus is deliberately practical. The environment already had an active detection library, ongoing development work, and regulatory requirements requiring the continuous development and deployment of new detections. The migration therefore had to improve the engineering model without causing material disruption to existing operations.
Previously, my colleague Stamatis Chatzimangou published a detailed series on this blog about Detection Engineering and Detection-as-Code, covering the underlying concepts and building blocks. If you are new to Detection-as-Code, I recommend reading that series first.
This case study summarises work performed in a recent customer project. Some details have been removed or adjusted to protect customer confidentiality. However, the technical challenges we faced and the essential migration steps remain intact, providing a realistic view of how such a transformation can be carried out in a live environment.
Starting Position
The environment contained multiple detection platforms, each requiring platform-specific detection implementations. This article uses Splunk as its primary example and explains how other platforms fit into the resulting structures and processes without covering them in detail.
The organisation had already taken initial steps towards Detection-as-Code. Some detection-related data was stored and versioned in Git repositories. Changes were reviewed through pull requests, and development work was tracked through issues on a GitHub Project board.
Each detection platform had its own repository containing the relevant configuration. In the Splunk case, the repository largely contained the complete apps directory that was deployed to the Splunk infrastructure. A manual deployment process copied this directory to the Splunk server and reloaded the configuration. Other platforms used separate repositories containing their respective configuration files.
In addition to the detection implementations in the platform configurations, each detection had a metadata document in YAML format. These documents were still stored on a file share. Test implementations based on Atomic Red Team were also available. However, each test file contained test cases for all detections covering the same MITRE ATT&CK technique, and the tests were stored within the Splunk repository.
The existing setup provided a useful starting point. Git-based storage and existing deployment processes meant that the project could focus on improving data handling and engineering workflows. The deployment mechanisms for the detection platforms did not need to be replaced, provided that the new process could still generate the required Splunk app configuration and equivalent artefacts for other platforms. Developers were also already familiar with Git, issues, pull requests, and project boards.
Weaknesses
At the same time, the existing setup had several significant weaknesses. The data was organised by detection platform rather than by the semantic unit developers worked on: the individual detection. Files belonging to one detection were distributed across multiple locations, including multiple repositories and a file share. As a result, a single change could require multiple pull requests and separate file-share changes. This made development and review unnecessarily difficult, increased the likelihood of human error, and made it hard to trace changes reliably.
For example, savedsearches.conf files in the Splunk apps contained several thousand lines of configuration for many unrelated detections. Such files are difficult to review as GitHub does not always render them conveniently. Further they can contain configuration fragments for detections that have already been retired but were never removed.
With this fragmented data, there was reduced visibility into data quality, completeness, and coverage. Thereby, reviews required substantial manual effort. Furthermore, without confidence in the data consistency and general quality, the files could only be used to deploy detections to the platforms, but they were not suited for further use, and a need existed for separately maintained reports and lists. In addition to the manual effort required to keep these up to date and in sync, significant potential was lost. Cleanly structured and validated data would enable fully automated report generation, data analysis, and further integrations (for example automated generation of compliance reports required by auditors). This would progressively eliminate manual work for developers, reviewers, and other stakeholders who need to assess information from the detection library.
Goals
To address these issues, we defined three goals:
- Restructure the existing data
The data structure should reflect the semantic structure of detections and the way developers interact with them. Platform-specific structures should be generated automatically by deployment processes. - Consolidate all detection-related data in GitHub
All data maintained outside GitHub should be migrated into GitHub, including metadata documents from the file share and information previously maintained in Excel sheets to create a single central source of truth. - Build a foundation for automation, reporting, and integrations
The restructured and consolidated data should enable initial automations and provide a framework for adding further automations in the future.
Target Data Structure
We designed a new repository structure that places all information required by detection developers in a single detection repository. This repository contains the complete detection library, covering detections for different platforms.
The existing repositories for individual detection platforms remain in place, but their scope changes. They contain only platform-specific configuration and deployment logic. This removes overlapping responsibilities between repositories: the Detection Engineering team owns the detection repository, while the relevant platform operations team owns each platform-specific repository.
The main branch of the detection repository represents the current detection library from the Detection Engineering perspective. Platform teams regularly consume the current state of that branch, package a specific release, and deploy the resulting platform configuration to the production environment.
With this consolidation, all detection development takes place in a single repository. Each change starts with an issue, is implemented on a feature branch following a GitHub Flow model, is tracked on a GitHub Project board, and is reviewed and approved through a single pull request.
The repository contains three main folders, as well as a README.md that explains the structure:
detections: currently active detectionsdetections_retired: archived detections that have been removed from active useutils: CLI tools, scripts, and libraries for detection developers and automations
Within detections, each detection has its own folder. The folders retain the existing naming convention for internal detection identifiers. While the convention itself could be improved, the migration deliberately limited the number of simultaneous changes. Keeping the familiar identifiers reduced adoption effort and avoided introducing an additional source of risk.
The detections_retired folder follows the same structure. Retiring a detection is therefore a simple and traceable operation: move the detection folder from detections to detections_retired.
All artefacts belonging to a detection are stored in its folder. Typical contents include:
meta.yml: metadata such as name, description, covered MITRE ATT&CK techniques, category, platform, known blind spots, known false positives, severity, references, and required data sourcestest-atomic.yml: an Atomic Red Team test definition that simulates the activity the detection is expected to identify- Platform-specific implementation files. For Splunk, these can include:
splunk-searches.conf: relevant configuration blocks fromsavedsearches.confsplunk-transforms.conf: relevant configuration blocks fromtransforms.confsplunk-macros.conf: relevant configuration blocks frommacros.confsplunk-lookups: lookup files used by the detection
The Splunk implementation files remain raw configuration blocks that are later placed in files within the apps directory. This is not ideal because some information is redundant with the meta.yml documents. However, we deliberately did not change this during the main migration in order to avoid introducing additional potential sources of error.
Overall, the resulting repository structure looks as follows:
.
├── detections
│ ├── DET00001
│ │ ├── meta.yml
│ │ ├── test-atomic.yml
│ │ ├── splunk-searches.conf
│ │ ├── splunk-transforms.conf
│ │ ├── splunk-macros.conf
│ │ └── splunk-lookups
│ │ └── DET00001_false_positives.csv
│ │
│ └── DET00002
│ ├── meta.yml
│ ├── test-atomic.yml
│ └── other_platform_implementation.json
│
├── detections_retired
│ └── ...
│
├── utils
│ └── ...
│
└── README.mdThis structure makes the detection, rather than the platform configuration file, the central unit of development. A developer working on a detection can find its metadata, tests, implementation files, and supporting lookups in one location.
Roadmap
With the target structure defined, we planned a migration that would minimise the impact on ongoing operations.
- Fix file formats
Several files, particularly metadata documents and Atomic Red Team tests, were intended to be machine-readable YAML files but did not consistently contain valid YAML. Encoding problems, indentation errors, and missing or incorrect keys had to be fixed before automated processing was possible. Some names also had to be corrected manually to allow reliable matching between related files during the restructuring. - Prepare supporting utilities for the new structure
Tooling had to be developed to generate the configuration structures required by detection platforms such as Splunk, as well as the folder structures required by Atomic Red Team. - Prepare and test the restructuring
The restructuring itself had to be scripted and tested. The restructuring process and supporting tooling had to reproduce the original platform-specific files or generate functionally equivalent configurations. - Complete as much in-progress development as possible
The migration would be based on the main branches.
Pending development work should therefore be reviewed and merged before the migration wherever possible, reducing the amount of post-migration work. - Execute the restructuring
This was the only phase that required a short freeze. For one day, the main branches and remaining non-Git locations would be frozen while the restructuring was executed. A final validation would then determine whether the migration had completed successfully. If successful, work would continue in the new structure; otherwise, the environment would be reverted and additional preparation would be required. - Migrate ongoing development work and open reviews
Only the main branches were frozen. Development could continue on feature branches during the restructuring, but those branches would remain tied to the old repositories and could no longer be merged there. The changes therefore had to be recreated on feature branches in the new repository. The old branches could then be deleted without being merged. - Split Atomic Red Team tests
At the start of the migration, Atomic Red Team test files were organised by MITRE ATT&CK technique rather than by detection. During the restructuring, the complete test definition was copied into each detection folder associated with the relevant technique. Afterwards, test cases that did not belong to that specific detection had to be removed. This activity was intentionally postponed because it did not affect platform deployment and was therefore not a blocker for resuming normal operations. - Migrate remaining data
As a final migration step, detection information still maintained in Excel sheets was moved into the metadata documents. The YAML schema was extended where needed to accommodate that information.
After the main restructuring, the next priority was to improve data quality and consistency. For this purpose, we developed a validator that could be executed manually as a CLI tool and automatically in a build pipeline. The validator was gradually extended with additional checks while existing inconsistencies were corrected. Integrated into the pipeline, the validator blocks pull requests that introduce inconsistencies and ensures that the achieved data quality is maintained.
With consistent and validated data in place, the organisation could use this foundation to build automations, reports, and further improvements.
Wrapping Up
The key design decision was to make the individual detection the central unit of storage and development.
Consolidating metadata, tests, implementation files, and supporting artefacts in a single repository reduced fragmentation without replacing established platform deployment processes.
Equally importantly, the migration was designed as an operational change rather than a purely technical refactoring. Preparing the data, validating the transformation, limiting the freeze period, and explicitly handling ongoing development made it possible to improve the engineering model while keeping the detection programme running.
In Part 2, we examine how this roadmap was executed in practice, including the data migration, preservation of Git history, and clean-up work that followed.
