[{"data":1,"prerenderedAt":815},["ShallowReactive",2],{"/en-us/blog/how-gitlabs-red-team-automates-c2-testing":3,"navigation-en-us":38,"banner-en-us":447,"footer-en-us":457,"blog-post-authors-en-us-Josh Feehs":699,"blog-related-posts-en-us-how-gitlabs-red-team-automates-c2-testing":713,"blog-promotions-en-us":753,"next-steps-en-us":805},{"id":4,"title":5,"authorSlugs":6,"body":8,"categorySlug":9,"config":10,"content":14,"description":8,"extension":27,"isFeatured":12,"meta":28,"navigation":12,"path":29,"publishedDate":20,"seo":30,"stem":35,"tagSlugs":36,"__hash__":37},"blogPosts/en-us/blog/how-gitlabs-red-team-automates-c2-testing.yml","How Gitlabs Red Team Automates C2 Testing",[7],"josh-feehs",null,"security-labs",{"slug":11,"featured":12,"template":13},"how-gitlabs-red-team-automates-c2-testing",true,"BlogPost",{"title":15,"description":16,"authors":17,"heroImage":19,"date":20,"body":21,"category":9,"tags":22},"How GitLab's Red Team automates C2 testing ","Learn how to apply professional development practices to Red Teams using open source command and control tools.",[18],"Josh Feehs","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749665667/Blog/Hero%20Images/built-in-security.jpg","2023-11-28","At GitLab, our [Red Team](https://handbook.gitlab.com/handbook/security/security-operations/red-team/) conducts security exercises that emulate real-world threats. By emulating real-world threats, we help assess and improve the effectiveness of the people, processes, and technologies used to keep our organization secure. To operate effectively, we must utilize professional development practices like the threat actors we emulate.\n\n[Threat actors](https://www.securonix.com/blog/threat-labs-security-advisory-new-starkvortex-attack-campaign-threat-actors-use-drone-manual-lures-to-deliver-merlinagent-payloads/) use open source command and control (C2) tools such as [Merlin](https://github.com/Ne0nd0g/merlin). While convenient, these tools have intentionally detectable features to discourage illegitimate use. Red Teams often need to customize and combine different open source options to evade detections in the environments they target.\n\nIn this blog, you'll learn how our team applies professional development practices to using open source C2 tools. We'll share how we implement continuous testing for the Mythic framework, our design philosophy, and a public project you can fork and use yourself.\n\nOur solution, available in [this public project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage), improves our Red Team operations in two ways. First, it contains a suite of **pytest** tests for the Mythic C2 framework. These validate functionality of both the Mythic server and multiple Mythic-compatible agents. Second, it leverages **GitLab CI/CD pipelines** to automatically run these tests after each code change. This enables iterative development and rapid validation of updates to Mythic or Mythic-compatible C2 agents.\n\n## Prerequisites\n\nCurrently, a few prerequisites fall outside the scope of test automation:\n\n- A Linux VM with Mythic, its Python requirements, and the HTTP profile installed. See the [Mythic installation guide](https://docs.mythic-c2.net/installation). We suggest binding Mythic's admin interface to localhost only.\n- A fork of [the ContinuousMage GitLab project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage) in GitLab.com or your own GitLab instance. You'll build on top of this to run your own automation. We highly suggest making this fork private, so you don't expose your test infrastructure or C2 code changes.\n- GitLab Runner installed on the VM (configured with the [shell executor](https://docs.gitlab.com/runner/executors/shell.html)) and registered with your GitLab instance. See the docs on [installing](https://docs.gitlab.com/runner/install/) and [registering](https://docs.gitlab.com/runner/register/) a runner or follow the instructions provided when configuring your pipeline later in this blog. You'll assign this runner to your project when we configure CI/CD.\n- Your forked project cloned onto your VM. This allows testing code changes (or new tests) before triggering the pipeline.\n\n## Project structure\n\nThe project contains three main portions that we will detail in this blog post:\n\n1. `pytest` test code for running integration tests for Mythic and Mythic-compatible C2 agents\n2. The source of those Mythic-compatible C2 agents, as git submodules\n3. The GitLab CI/CD pipeline configuration that ties it all together\n\n## Part 1: pytests\n\n[pytest](https://docs.pytest.org/en/7.4.x/) is a framework for writing tests in Python. We can leverage pytest to do integration testing of Mythic since it has its own [Python package](https://pypi.org/project/mythic/). The test suite goals are:\n\n1. Be simple and atomic.\n2. Provide adequate coverage to validate tool readiness.\n\nWe'll walk through a simple test verifying an agent can run the `ls` command, highlighting key code sections for customization.\n\n### Implementation\n\n#### pytest file\n\nWhen run on a directory, `pytest` automatically discovers tests in files prefixed with `test_` and test functions starting with `test_`. Our tests are asynchronous, needing the `pytest.mark.asyncio` decorator, because the Mythic APIs we are testing are asynchronous. If your machine is missing test dependencies, run `python3 -m pip install mythic pytest pytest-asyncio`.\n\nA test function skeleton is as follows:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    # Will do the test here\n    continue\n```\n\n#### The GlMythic class\n\nThe `GlMythic` class wraps Mythic APIs for ease of use in testing. Because its `init` function is async, a coroutine creates the object:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    glmythic = await gl_mythic.create_glmythic()\n```\n\nBy default, it connects to the Mythic DB using the `MYTHIC_ADMIN_PASSWORD` environment variable and is configured to test the agent specified via the `AGENT_TYPE` environment variable. We will set these in the CI/CD config later.\n\n#### Interacting with Mythic via GlMythic\n\nWe'll include the remainder of the test code here, with comments, and then discuss the most important parts.\n\nAs a reminder, one of the key goals of this project was to make completely atomic tests. Each test only relies on a running Mythic server with the specific agent and HTTP containers loaded. As the test suite grows, it may be worth running a secondary set of tasks that relies on an already-existing agent connection. Currently, every test creates, downloads, and executes a new agent.\n\n### Test and deploy\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n\n    glmythic = await gl_mythic.create_glmythic()\n\n    # Unique payload_path per test\n    payload_path = \"/tmp/test_agent_ls\"\n\n    # Wraps agent create, download, and execute\n    proc = await glmythic.generate_and_run(payload_path=payload_path)\n\n    # Wait for callback\n    time.sleep(10)\n\n    # Uses the display_id field to determine most recent callback\n    # Assumes that the most recent callback is the one created by this test\n    callback = await glmythic.get_latest_callback()\n\n    # Issue the ls command, blocking on output\n    output = await mythic.issue_task_and_waitfor_task_output(\n        mythic=glmythic.mythic_instance,\n        command_name=\"ls\",\n        parameters=\"\",\n        callback_display_id=callback[\"display_id\"],\n        timeout=20,\n    )\n\n    # Clean up (no longer need the agent)\n    proc.terminate()     os.remove(payload_path)\n\n    # If the ls failed, there will be no output\n    # This test could also look for files in the repo (where the agent runs)\n    assert len(output) > 0\n\n```\n\nThe longest running portion of this test will be the call to `generate_and_run`, as agent builds within Mythic can take from seconds to minutes or even hang altogether. For your initial set of tests, sign in to the Mythic server and watch the **Payloads** screen for potential issues. In our testing, agent builds failed to complete around 5% of the time, depending on the agent. If you experience repeated build failures, reload your agent container with `sudo ./mythic-cli install folder \u003Cagent_directory> -f`.\n\nTo run the tests, run `pytest \u003Ctestfile_directory>`.\n\n## Part 2: Agent source as submodules\n\nBecause Mythic agents are often updated, we include the agent repos as git submodules in our test project. This allows us to update to new agent versions when they are released and use our project's version control to keep tool versions static for known good builds. These submodules are all located in the `agents` folder.\n\nWe'll discuss adding more agents to this project later in this blog.\n\n## Part 3: GitLab CI/CD pipeline\n\nNow that you have working pytests, you can automate your tests to run whenever you want. In our case, we chose to run our tests on merge requests and tagged commits (which are likely to be tool releases). We will be using [GitLab CI/CD pipelines](https://docs.gitlab.com/ee/ci/pipelines/) to perform our automated tests.\n\n### Configuring the pipeline\n\nNow is the time to set your GitLab CI/CD settings. To find these settings, go to your repository -> `Settings` -> `CI/CD`.\n\nThe first setting you'll want to set is your `Runner`. If you set up a runner as one of your prerequisite steps earlier, you can assign it here. If not, click `New project runner` and work through that process to create and set up your runner on your Mythic server. When you are prompted to choose a runner type on install, choose the [shell executor](https://docs.gitlab.com/runner/executors/shell.html). If your team uses shared runners for other CI/CD pipelines, you will want to make sure that shared runners are disabled for this project, given that your shared runners are unlikely to be able to talk to Mythic directly.\n\n![runner-settings](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/runner-settings.png)\n\nNext, you need to set your `Variables`. The `GlMythic` class uses the `MYTHIC_ADMIN_PASSWORD` environment variable to be able to actually sign into Mythic, so you need to make sure that the pipeline runner's environment is set up correctly.\n\nTo do this, click the `Add variable` button and add the `MYTHIC_ADMIN_PASSWORD` variable with the appropriate value. If you don't know your Mythic admin password, on the Mythic server in the directory where you installed Mythic, `cat .env | grep MYTHIC_ADMIN_PASSWORD` will give you the password.\n\nBecause GitLab handles merge requests in a detached state, you need to unclick the `Protect Variable` box, because that would prevent the pipeline from viewing the variable on a merge request otherwise. Because the variable is not protected, any branch committed back to your server can access your CI variables. This may pose a security risk if you allow remote access to your Mythic server (versus binding to localhost) and if you allow arbitrary users to access your repository. For this reason, our public repo does not have the environment variables. We use a private copy to perform testing, and suggest you do the same.\n\nAdditionally, set the `AGENT_TYPE` variable to the name of the agent you want to use. At time of release, valid agent types are `poseidon` or `merlin`. The section about adding more agents to the test suite will go into more detail.\n\nYou can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.\n\nNow that the pipeline is configured to use the runner and pick up the environment variables that you need, the only thing left to do is to set up your pipeline. This step is quite simple: If you add the `.gitlab-ci.yml` file to the root of your repository, GitLab will pick that up as the pipeline config on your next commit. Here is our example pipeline, which we will explain momentarily.\n\n```yaml\ninstall:\n  stage: install\n  script:\n    - sudo /opt/Mythic/mythic-cli install folder \"${CI_PROJECT_DIR}\"/agents/\"${AGENT_TYPE}\" -f\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n\ntest:\n  stage: test\n  script:\n    - pytest \"${CI_PROJECT_DIR}\"/mythic-test\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n```\n\nAll of the variables set above are made available by GitLab as part of every pipeline. This pipeline has two stages, `install` and `test`. Both stages are set to only run on merge requests or if the commit being evaluated has a specific tag. The `install` stage will install your C2 agent into Mythic using its local folder install. This makes sure that the Mythic server has your latest C2 code changes installed. Next, the `test` stage runs the set of pytest tests that we created. The `install` stage will run very quickly, and the `test` stage will run a little more slowly, given that it's doing the work of creating and interacting with Mythic agents.\n\n### Pipeline in action\n\nYou can do a couple of things to validate that your pipeline is working. First, if you are performing a merge request, there will be a section at the beginning of the merge request that will link to the pipeline. The screenshot below shows that the pipeline has passed, but you can click into the pipeline by clicking on its number even when it's running.\n\n![Pipeline passing](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/merge-pipeline-pass.png)\n\nYou can then click into the stage that's running (or one that has already run) to view its output.\n\n![Pipeline task output](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/pipeline-task-output.png)\n\nAnd there you are! You now have working `pytest` tests for a Mythic agent that run every time you make a merge request.\n\n## Adapting for other agents\n\nWe tested our test suite against Poseidon and Merlin. Although the initial tests (generate, download and exec, ls) work the same for both agents, Poseidon and Merlin require different parameters for their `upload` commands. Unfortunately, this means that not all tests will be agent agnostic.\n\nAs a result, each `GlMythic` object that is created is told what type of agent it is testing. The coroutine for creating an object allows you to pass in the agent type as a variable, and defaults to using the `AGENT_TYPE` environment variable to determine which agent is being tested.\n\n```python\nasync def create_glmythic(  username=\"mythic_admin\",\n                            password=os.getenv(\"MYTHIC_ADMIN_PASSWORD\"),\n                            server_ip=\"127.0.0.1\",\n                            server_port=7443,\n                            agent_type=os.getenv(\"AGENT_TYPE\")):\n```\n\n### Agent source\n\nTo add more agents for testing, the first thing to do is to import your agent as a git submodule:\n\n```bash\ncd agents\ngit submodule add \"${URL_TO_YOUR_AGENT}\"\n```\n\nCommit your changes, and your agent is tracked as part of the repo.\n\n### Test compatibility\n\nYou'll need to validate that existing tests work with your agent. For tests to work, the parameters passed to the commands must match those in the test suite, with `upload` to be most likely to fail.\n\nThis is okay! Within the `test_agent_upload` test function, you'll see example code that specifies a different upload command for Merlin and Poseidon. Simply follow this structure for your own agent, passing your agent's parameters to the `mythic.issue_task_and_waitfor_task_output` function call.\n\nIf you are using another open source C2 and are unsure of the correct parameters to pass, you can use the Mythic UI. Interact with one of your agents and run the `upload` command to see what params you need to pass. If you do this for Poseidon, it will look like the following:\n\n![upload-parameters](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/upload-parameters.png)\n\nOur test suite should be pretty easy to add to any Linux-based Mythic agent that supports the [HTTP C2 profile](https://github.com/MythicC2Profiles/http). Because the GitLab Runner installs the agent into Mythic (and Mythic is made to run on Linux), the runner is expecting to be on a Linux machine. Additional effort and test modifications will be required to run the test suite against a Windows or MacOS agent.\n\n## A quick win\n\nAs we worked on this project, we were continuously running our test suite against both Poseidon and Merlin. Unexpectedly, in early October 2023, our test for Poseidon's `upload` function started to fail. After a quick investigation, we identified that a bug had been introduced, present in Poseidon 2.0.2, that caused file uploads to fail.\n\nWe took our information to one of the Poseidon developers, Cody Thomas ([@its_a_feature_](https://twitter.com/its_a_feature_)), and he quickly identified the underlying issue and [fixed the problem](https://github.com/MythicAgents/poseidon/commit/83de4712448d7ed948b3e2d2b2f378d530b3a42a).\n\nThis highlights the usefulness of continuous testing. Instead of running into a potential bug during a Red Team exercise, we identified the issue beforehand and were able to report the bug so the issue was fixed.\n\nWe sincerely thank the Mythic, Merlin, and Poseidon developers for open sourcing their hard work. Many Red Teams around the world are able to perform high-quality security assessments in part because of the hard work of C2 developers who open source their tools. We also want to specifically thank Cody Thomas for addressing this bug within 20 minutes of notification. His responsiveness and attention to detail are unmatched.\n\n## Share your feedback\n\nThis post has demonstrated both the value of continuous testing and shown how to implement continuous testing for your own use, using GitLab. If you have worked alongside these examples, you've implemented some continuous testing for the Mythic framework and have tests that you can use for Merlin, Poseidon, or your own Mythic agent(s).\n\nAt GitLab, we always seek feedback on our work. If you have any questions or comments, please open an issue on [our project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage). You can also propose improvements via a merge request. We believe that everyone should be able to contribute, so we welcome any contributions, big or small.\n\n> [Try GitLab Ultimate for free today.](https://gitlab.com/-/trials/new)\n\n## Related reading\n- [Stealth operations: The evolution of GitLab's Red Team](https://about.gitlab.com/blog/stealth-operations-the-evolution-of-gitlabs-red-team/)\n- [How we run Red Team operations remotely](https://about.gitlab.com/blog/how-we-run-red-team-operations-remotely/)\n- [Use GitLab and MITRE ATT&CK Navigator to visualize adversary techniques](https://about.gitlab.com/blog/gitlab-mitre-attack-navigator/)\n- [Monitor web attack surface with GitLab](https://about.gitlab.com/blog/monitor-web-attack-surface-with-gitlab/)\n",[23,24,25,26],"security","testing","integrations","tutorial","yml",{},"/en-us/blog/how-gitlabs-red-team-automates-c2-testing",{"title":15,"description":16,"ogTitle":15,"ogDescription":16,"noIndex":31,"ogImage":19,"ogUrl":32,"ogSiteName":33,"ogType":34,"canonicalUrls":32},false,"https://about.gitlab.com/blog/how-gitlabs-red-team-automates-c2-testing","https://about.gitlab.com","article","en-us/blog/how-gitlabs-red-team-automates-c2-testing",[23,24,25,26],"AAAkk1iB_bUCrOUp_OJFV0eFl0EbUaMQytG-8Mb6Ys0",{"data":39},{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":367,"minimal":398,"duo":417,"switchNav":426,"pricingDeployment":437},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,88,183,188,288,348],{"text":62,"config":63,"cards":65},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The intelligent orchestration platform for DevSecOps",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":81,"description":82,"link":83},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"text":89,"left":12,"config":90,"link":92,"lists":96,"footer":165},"Product",{"dataNavLevelOne":91},"solutions",{"text":93,"config":94},"View all Solutions",{"href":95,"dataGaName":91,"dataGaLocation":44},"/solutions/",[97,121,144],{"title":98,"description":99,"link":100,"items":105},"Automation","CI/CD and automation to accelerate deployment",{"config":101},{"icon":102,"href":103,"dataGaName":104,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[106,110,113,117],{"text":107,"config":108},"CI/CD",{"href":109,"dataGaLocation":44,"dataGaName":107},"/solutions/continuous-integration/",{"text":73,"config":111},{"href":78,"dataGaLocation":44,"dataGaName":112},"gitlab duo agent platform - product menu",{"text":114,"config":115},"Source Code Management",{"href":116,"dataGaLocation":44,"dataGaName":114},"/solutions/source-code-management/",{"text":118,"config":119},"Automated Software Delivery",{"href":103,"dataGaLocation":44,"dataGaName":120},"Automated software delivery",{"title":122,"description":123,"link":124,"items":129},"Security","Deliver code faster without compromising security",{"config":125},{"href":126,"dataGaName":127,"dataGaLocation":44,"icon":128},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[130,134,139],{"text":131,"config":132},"Application Security Testing",{"href":126,"dataGaName":133,"dataGaLocation":44},"Application security testing",{"text":135,"config":136},"Software Supply Chain Security",{"href":137,"dataGaLocation":44,"dataGaName":138},"/solutions/supply-chain/","Software supply chain security",{"text":140,"config":141},"Software Compliance",{"href":142,"dataGaName":143,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":145,"link":146,"items":151},"Measurement",{"config":147},{"icon":148,"href":149,"dataGaName":150,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[152,156,160],{"text":153,"config":154},"Visibility & Measurement",{"href":149,"dataGaLocation":44,"dataGaName":155},"Visibility and Measurement",{"text":157,"config":158},"Value Stream Management",{"href":159,"dataGaLocation":44,"dataGaName":157},"/solutions/value-stream-management/",{"text":161,"config":162},"Analytics & Insights",{"href":163,"dataGaLocation":44,"dataGaName":164},"/solutions/analytics-and-insights/","Analytics and insights",{"title":166,"items":167},"GitLab for",[168,173,178],{"text":169,"config":170},"Enterprise",{"href":171,"dataGaLocation":44,"dataGaName":172},"/enterprise/","enterprise",{"text":174,"config":175},"Small Business",{"href":176,"dataGaLocation":44,"dataGaName":177},"/small-business/","small business",{"text":179,"config":180},"Public Sector",{"href":181,"dataGaLocation":44,"dataGaName":182},"/solutions/public-sector/","public sector",{"text":184,"config":185},"Pricing",{"href":186,"dataGaName":187,"dataGaLocation":44,"dataNavLevelOne":187},"/pricing/","pricing",{"text":189,"config":190,"link":192,"lists":196,"feature":275},"Resources",{"dataNavLevelOne":191},"resources",{"text":193,"config":194},"View all resources",{"href":195,"dataGaName":191,"dataGaLocation":44},"/resources/",[197,229,247],{"title":198,"items":199},"Getting started",[200,205,210,215,220,225],{"text":201,"config":202},"Install",{"href":203,"dataGaName":204,"dataGaLocation":44},"/install/","install",{"text":206,"config":207},"Quick start guides",{"href":208,"dataGaName":209,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":211,"config":212},"Learn",{"href":213,"dataGaLocation":44,"dataGaName":214},"https://university.gitlab.com/","learn",{"text":216,"config":217},"Product documentation",{"href":218,"dataGaName":219,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":221,"config":222},"Best practice videos",{"href":223,"dataGaName":224,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":226,"config":227},"Integrations",{"href":228,"dataGaName":25,"dataGaLocation":44},"/integrations/",{"title":230,"items":231},"Discover",[232,237,242],{"text":233,"config":234},"Customer success stories",{"href":235,"dataGaName":236,"dataGaLocation":44},"/customers/","customer success stories",{"text":238,"config":239},"Blog",{"href":240,"dataGaName":241,"dataGaLocation":44},"/blog/","blog",{"text":243,"config":244},"Remote",{"href":245,"dataGaName":246,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":248,"items":249},"Connect",[250,255,260,265,270],{"text":251,"config":252},"GitLab Services",{"href":253,"dataGaName":254,"dataGaLocation":44},"/services/","services",{"text":256,"config":257},"Community",{"href":258,"dataGaName":259,"dataGaLocation":44},"/community/","community",{"text":261,"config":262},"Forum",{"href":263,"dataGaName":264,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":266,"config":267},"Events",{"href":268,"dataGaName":269,"dataGaLocation":44},"/events/","events",{"text":271,"config":272},"Partners",{"href":273,"dataGaName":274,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":276,"textColor":277,"text":278,"image":279,"link":283},"#2f2a6b","#fff","Insights for the future of software development",{"altText":280,"config":281},"the source promo card",{"src":282},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":284,"config":285},"Read the latest",{"href":286,"dataGaName":287,"dataGaLocation":44},"/the-source/","the source",{"text":289,"config":290,"lists":292},"Company",{"dataNavLevelOne":291},"company",[293],{"items":294},[295,300,306,308,313,318,323,328,333,338,343],{"text":296,"config":297},"About",{"href":298,"dataGaName":299,"dataGaLocation":44},"/company/","about",{"text":301,"config":302,"footerGa":305},"Jobs",{"href":303,"dataGaName":304,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":304},{"text":266,"config":307},{"href":268,"dataGaName":269,"dataGaLocation":44},{"text":309,"config":310},"Leadership",{"href":311,"dataGaName":312,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":314,"config":315},"Team",{"href":316,"dataGaName":317,"dataGaLocation":44},"/company/team/","team",{"text":319,"config":320},"Handbook",{"href":321,"dataGaName":322,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":324,"config":325},"Investor relations",{"href":326,"dataGaName":327,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":329,"config":330},"Trust Center",{"href":331,"dataGaName":332,"dataGaLocation":44},"/security/","trust center",{"text":334,"config":335},"AI Transparency Center",{"href":336,"dataGaName":337,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":339,"config":340},"Newsletter",{"href":341,"dataGaName":342,"dataGaLocation":44},"/company/contact/#contact-forms","newsletter",{"text":344,"config":345},"Press",{"href":346,"dataGaName":347,"dataGaLocation":44},"/press/","press",{"text":349,"config":350,"lists":351},"Contact us",{"dataNavLevelOne":291},[352],{"items":353},[354,357,362],{"text":51,"config":355},{"href":53,"dataGaName":356,"dataGaLocation":44},"talk to sales",{"text":358,"config":359},"Support portal",{"href":360,"dataGaName":361,"dataGaLocation":44},"https://support.gitlab.com","support portal",{"text":363,"config":364},"Customer portal",{"href":365,"dataGaName":366,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":368,"login":369,"suggestions":376},"Close",{"text":370,"link":371},"To search repositories and projects, login to",{"text":372,"config":373},"gitlab.com",{"href":58,"dataGaName":374,"dataGaLocation":375},"search login","search",{"text":377,"default":378},"Suggestions",[379,381,385,387,391,395],{"text":73,"config":380},{"href":78,"dataGaName":73,"dataGaLocation":375},{"text":382,"config":383},"Code Suggestions (AI)",{"href":384,"dataGaName":382,"dataGaLocation":375},"/solutions/code-suggestions/",{"text":107,"config":386},{"href":109,"dataGaName":107,"dataGaLocation":375},{"text":388,"config":389},"GitLab on AWS",{"href":390,"dataGaName":388,"dataGaLocation":375},"/partners/technology-partners/aws/",{"text":392,"config":393},"GitLab on Google Cloud",{"href":394,"dataGaName":392,"dataGaLocation":375},"/partners/technology-partners/google-cloud-platform/",{"text":396,"config":397},"Why GitLab?",{"href":86,"dataGaName":396,"dataGaLocation":375},{"freeTrial":399,"mobileIcon":404,"desktopIcon":409,"secondaryButton":412},{"text":400,"config":401},"Start free trial",{"href":402,"dataGaName":49,"dataGaLocation":403},"https://gitlab.com/-/trials/new/","nav",{"altText":405,"config":406},"Gitlab Icon",{"src":407,"dataGaName":408,"dataGaLocation":403},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":405,"config":410},{"src":411,"dataGaName":408,"dataGaLocation":403},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":413,"config":414},"Get Started",{"href":415,"dataGaName":416,"dataGaLocation":403},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":418,"mobileIcon":422,"desktopIcon":424},{"text":419,"config":420},"Learn more about GitLab Duo",{"href":78,"dataGaName":421,"dataGaLocation":403},"gitlab duo",{"altText":405,"config":423},{"src":407,"dataGaName":408,"dataGaLocation":403},{"altText":405,"config":425},{"src":411,"dataGaName":408,"dataGaLocation":403},{"button":427,"mobileIcon":432,"desktopIcon":434},{"text":428,"config":429},"/switch",{"href":430,"dataGaName":431,"dataGaLocation":403},"#contact","switch",{"altText":405,"config":433},{"src":407,"dataGaName":408,"dataGaLocation":403},{"altText":405,"config":435},{"src":436,"dataGaName":408,"dataGaLocation":403},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":438,"mobileIcon":443,"desktopIcon":445},{"text":439,"config":440},"Back to pricing",{"href":186,"dataGaName":441,"dataGaLocation":403,"icon":442},"back to pricing","GoBack",{"altText":405,"config":444},{"src":407,"dataGaName":408,"dataGaLocation":403},{"altText":405,"config":446},{"src":411,"dataGaName":408,"dataGaLocation":403},{"title":448,"button":449,"config":454},"See how agentic AI transforms software delivery",{"text":450,"config":451},"Watch GitLab Transcend now",{"href":452,"dataGaName":453,"dataGaLocation":44},"/events/transcend/virtual/","transcend event",{"layout":455,"icon":456,"disabled":12},"release","AiStar",{"data":458},{"text":459,"source":460,"edit":466,"contribute":471,"config":476,"items":481,"minimal":688},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":461,"config":462},"View page source",{"href":463,"dataGaName":464,"dataGaLocation":465},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":467,"config":468},"Edit this page",{"href":469,"dataGaName":470,"dataGaLocation":465},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":472,"config":473},"Please contribute",{"href":474,"dataGaName":475,"dataGaLocation":465},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":477,"facebook":478,"youtube":479,"linkedin":480},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[482,529,583,627,654],{"title":184,"links":483,"subMenu":498},[484,488,493],{"text":485,"config":486},"View plans",{"href":186,"dataGaName":487,"dataGaLocation":465},"view plans",{"text":489,"config":490},"Why Premium?",{"href":491,"dataGaName":492,"dataGaLocation":465},"/pricing/premium/","why premium",{"text":494,"config":495},"Why Ultimate?",{"href":496,"dataGaName":497,"dataGaLocation":465},"/pricing/ultimate/","why ultimate",[499],{"title":500,"links":501},"Contact Us",[502,505,507,509,514,519,524],{"text":503,"config":504},"Contact sales",{"href":53,"dataGaName":54,"dataGaLocation":465},{"text":358,"config":506},{"href":360,"dataGaName":361,"dataGaLocation":465},{"text":363,"config":508},{"href":365,"dataGaName":366,"dataGaLocation":465},{"text":510,"config":511},"Status",{"href":512,"dataGaName":513,"dataGaLocation":465},"https://status.gitlab.com/","status",{"text":515,"config":516},"Terms of use",{"href":517,"dataGaName":518,"dataGaLocation":465},"/terms/","terms of use",{"text":520,"config":521},"Privacy statement",{"href":522,"dataGaName":523,"dataGaLocation":465},"/privacy/","privacy statement",{"text":525,"config":526},"Cookie preferences",{"dataGaName":527,"dataGaLocation":465,"id":528,"isOneTrustButton":12},"cookie preferences","ot-sdk-btn",{"title":89,"links":530,"subMenu":539},[531,535],{"text":532,"config":533},"DevSecOps platform",{"href":71,"dataGaName":534,"dataGaLocation":465},"devsecops platform",{"text":536,"config":537},"AI-Assisted Development",{"href":78,"dataGaName":538,"dataGaLocation":465},"ai-assisted development",[540],{"title":541,"links":542},"Topics",[543,548,553,558,563,568,573,578],{"text":544,"config":545},"CICD",{"href":546,"dataGaName":547,"dataGaLocation":465},"/topics/ci-cd/","cicd",{"text":549,"config":550},"GitOps",{"href":551,"dataGaName":552,"dataGaLocation":465},"/topics/gitops/","gitops",{"text":554,"config":555},"DevOps",{"href":556,"dataGaName":557,"dataGaLocation":465},"/topics/devops/","devops",{"text":559,"config":560},"Version Control",{"href":561,"dataGaName":562,"dataGaLocation":465},"/topics/version-control/","version control",{"text":564,"config":565},"DevSecOps",{"href":566,"dataGaName":567,"dataGaLocation":465},"/topics/devsecops/","devsecops",{"text":569,"config":570},"Cloud Native",{"href":571,"dataGaName":572,"dataGaLocation":465},"/topics/cloud-native/","cloud native",{"text":574,"config":575},"AI for Coding",{"href":576,"dataGaName":577,"dataGaLocation":465},"/topics/devops/ai-for-coding/","ai for coding",{"text":579,"config":580},"Agentic AI",{"href":581,"dataGaName":582,"dataGaLocation":465},"/topics/agentic-ai/","agentic ai",{"title":584,"links":585},"Solutions",[586,588,590,595,599,602,606,609,611,614,617,622],{"text":131,"config":587},{"href":126,"dataGaName":131,"dataGaLocation":465},{"text":120,"config":589},{"href":103,"dataGaName":104,"dataGaLocation":465},{"text":591,"config":592},"Agile development",{"href":593,"dataGaName":594,"dataGaLocation":465},"/solutions/agile-delivery/","agile delivery",{"text":596,"config":597},"SCM",{"href":116,"dataGaName":598,"dataGaLocation":465},"source code management",{"text":544,"config":600},{"href":109,"dataGaName":601,"dataGaLocation":465},"continuous integration & delivery",{"text":603,"config":604},"Value stream management",{"href":159,"dataGaName":605,"dataGaLocation":465},"value stream management",{"text":549,"config":607},{"href":608,"dataGaName":552,"dataGaLocation":465},"/solutions/gitops/",{"text":169,"config":610},{"href":171,"dataGaName":172,"dataGaLocation":465},{"text":612,"config":613},"Small business",{"href":176,"dataGaName":177,"dataGaLocation":465},{"text":615,"config":616},"Public sector",{"href":181,"dataGaName":182,"dataGaLocation":465},{"text":618,"config":619},"Education",{"href":620,"dataGaName":621,"dataGaLocation":465},"/solutions/education/","education",{"text":623,"config":624},"Financial services",{"href":625,"dataGaName":626,"dataGaLocation":465},"/solutions/finance/","financial services",{"title":189,"links":628},[629,631,633,635,638,640,642,644,646,648,650,652],{"text":201,"config":630},{"href":203,"dataGaName":204,"dataGaLocation":465},{"text":206,"config":632},{"href":208,"dataGaName":209,"dataGaLocation":465},{"text":211,"config":634},{"href":213,"dataGaName":214,"dataGaLocation":465},{"text":216,"config":636},{"href":218,"dataGaName":637,"dataGaLocation":465},"docs",{"text":238,"config":639},{"href":240,"dataGaName":241,"dataGaLocation":465},{"text":233,"config":641},{"href":235,"dataGaName":236,"dataGaLocation":465},{"text":243,"config":643},{"href":245,"dataGaName":246,"dataGaLocation":465},{"text":251,"config":645},{"href":253,"dataGaName":254,"dataGaLocation":465},{"text":256,"config":647},{"href":258,"dataGaName":259,"dataGaLocation":465},{"text":261,"config":649},{"href":263,"dataGaName":264,"dataGaLocation":465},{"text":266,"config":651},{"href":268,"dataGaName":269,"dataGaLocation":465},{"text":271,"config":653},{"href":273,"dataGaName":274,"dataGaLocation":465},{"title":289,"links":655},[656,658,660,662,664,666,668,672,677,679,681,683],{"text":296,"config":657},{"href":298,"dataGaName":291,"dataGaLocation":465},{"text":301,"config":659},{"href":303,"dataGaName":304,"dataGaLocation":465},{"text":309,"config":661},{"href":311,"dataGaName":312,"dataGaLocation":465},{"text":314,"config":663},{"href":316,"dataGaName":317,"dataGaLocation":465},{"text":319,"config":665},{"href":321,"dataGaName":322,"dataGaLocation":465},{"text":324,"config":667},{"href":326,"dataGaName":327,"dataGaLocation":465},{"text":669,"config":670},"Sustainability",{"href":671,"dataGaName":669,"dataGaLocation":465},"/sustainability/",{"text":673,"config":674},"Diversity, inclusion and belonging (DIB)",{"href":675,"dataGaName":676,"dataGaLocation":465},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":329,"config":678},{"href":331,"dataGaName":332,"dataGaLocation":465},{"text":339,"config":680},{"href":341,"dataGaName":342,"dataGaLocation":465},{"text":344,"config":682},{"href":346,"dataGaName":347,"dataGaLocation":465},{"text":684,"config":685},"Modern Slavery Transparency Statement",{"href":686,"dataGaName":687,"dataGaLocation":465},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":689},[690,693,696],{"text":691,"config":692},"Terms",{"href":517,"dataGaName":518,"dataGaLocation":465},{"text":694,"config":695},"Cookies",{"dataGaName":527,"dataGaLocation":465,"id":528,"isOneTrustButton":12},{"text":697,"config":698},"Privacy",{"href":522,"dataGaName":523,"dataGaLocation":465},[700],{"id":701,"title":18,"body":8,"config":702,"content":704,"description":8,"extension":27,"meta":708,"navigation":12,"path":709,"seo":710,"stem":711,"__hash__":712},"blogAuthors/en-us/blog/authors/josh-feehs.yml",{"template":703},"BlogAuthor",{"name":18,"config":705},{"headshot":706,"ctfId":707},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683068/Blog/Author%20Headshots/Screenshot_2023-11-28_at_9.12.13_AM.png","g5S7qgnlO5aJJ00brs77P",{},"/en-us/blog/authors/josh-feehs",{},"en-us/blog/authors/josh-feehs","GCxiCFjrkcnCx0oF_E3Ps7yjaL35GgFUFRhUekMz-kw",[714,728,742],{"content":715,"config":726},{"body":716,"title":717,"description":718,"category":9,"tags":719,"authors":722,"heroImage":724,"date":725},"\n***Note: The GitLab product did not use any of the compromised package versions mentioned in this post.***\n\nIn the span of 12 days, four separate supply chain attacks revealed that continuous integration and continuous delivery (CI/CD) pipelines have become a high-value target for sophisticated threat actors.\n\nBetween March 19 and March 31, 2026, threat actors compromised:\n\n* an open-source security scanner (Trivy)\n* an infrastructure-as-code (IaC) security scanner (Checkmarx KICS)\n* an AI model gateway (LiteLLM)\n* a JavaScript HTTP client (axios)\n\nEach attack shared the same surface: the build pipeline.\nThis article shows [what happened](#trusted-by-millions-compromised-in-minutes), [why pipelines can be uniquely vulnerable](#the-patterns-behind-these-attacks), and how centralized policy enforcement with GitLab — using policies defined below — can [block, detect, and contain these classes of attack](#how-gitlab-pipeline-execution-policies-address-each-attack-pattern) before they reach production.\n\n\n## Trusted by millions, compromised in minutes\n\nHere is the timeline of the supply chain attacks:\n\n### March 19: Trivy security scanner becomes an attack vector\n\n[Trivy](https://github.com/aquasecurity/trivy) is one of the most widely used open-source vulnerability scanners in the world. It is the tool teams run *inside their pipelines* to find vulnerabilities.\n\nOn March 19, a threat actor group known as [TeamPCP used compromised credentials](https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/) to force-push malicious code into 76 of 77 version tags of the `aquasecurity/trivy-action` GitHub Action and all 7 tags of `aquasecurity/setup-trivy`. Simultaneously, they published a trojanized Trivy binary (v0.69.4) to official distribution channels. The payload was credential-stealing malware that harvested environment variables, cloud tokens, SSH keys, and CI/CD secrets from every pipeline that ran a Trivy scan.\n\nThe incident was assigned [CVE-2026-33634](https://nvd.nist.gov/vuln/detail/CVE-2026-33634) with a CVSS score of 9.4. The Cybersecurity and Infrastructure Security Agency (CISA) added it to the Known Exploited Vulnerabilities catalog within days.\n\n### March 23: Checkmarx KICS falls next\nUsing stolen credentials, TeamPCP pivoted to Checkmarx’s open-source KICS (Keeping Infrastructure as Code Secure) project. They compromised the `ast-github-action` and `kics-github-action` GitHub Actions, [injecting the same credential-stealing malware](https://thehackernews.com/2026/03/teampcp-hacks-checkmarx-github-actions.html). Between 12:58 and 16:50 UTC on March 23, any CI/CD pipeline referencing these actions was silently exfiltrating sensitive data, such as API keys, database passwords, cloud access tokens, SSH keys, and service account credentials.\n\n### March 24: LiteLLM compromised via stolen Trivy credentials\n\nLiteLLM, an LLM API proxy with 95 million monthly downloads, was the next target. TeamPCP [published backdoored versions](https://thehackernews.com/2026/03/teampcp-backdoors-litellm-versions.html) (1.82.7 and 1.82.8) to PyPI using credentials harvested from LiteLLM’s own CI/CD pipeline, which used Trivy for scanning.\n\nThe malware targeting Version 1.82.7 used a base64-encoded payload injected directly into `litellm/proxy/proxy_server.py` that executed at import time. The version targeting 1.82.8 used a `.pth` file, a Python mechanism that executes automatically during interpreter startup. Simply installing LiteLLM was enough to trigger the payload. Attackers encrypted the stolen data (SSH keys, cloud tokens, .env files, cryptocurrency wallets) and exfiltrated it to `models.litellm.cloud`, a lookalike domain.\n\n### March 31: Source code for AI coding assistant leaked via simple packaging mistake\nWhile the TeamPCP campaign was still unfolding, a software company shipped an npm package containing a 59.8 MB source map file — one that referenced its AI coding assistant's complete, unminified TypeScript source code, hosted in the company's own Cloudflare R2 bucket.\n\nThe leak exposed 1,900 TypeScript files, 512,000+ lines of code, 44 hidden feature flags, unreleased model codenames, and the full system prompt for anyone who knew where to look. As engineer [Gabriel Anhaia explained](https://dev.to/gabrielanhaia/claude-codes-entire-source-code-was-just-leaked-via-npm-source-maps-heres-whats-inside-cjo), “A single misconfigured .npmignore or files field in package.json can expose everything.”\n### March 31: axios and another trojan in the supply chain\nThat same day, a sophisticated campaign [targeted the axios npm package](https://thehackernews.com/2026/03/axios-supply-chain-attack-pushes-cross.html), a JavaScript HTTP client with over 100 million weekly downloads.\n\nA compromised maintainer account published backdoored versions (1.14.1 and 0.30.4). It injected a malicious dependency (`plain-crypto-js@4.2.1`) that deployed a Remote Access Trojan capable of running on macOS, Windows, and Linux. Both release branches were hit within 39 minutes, with the malware designed to self-destruct after execution.\n\n## The patterns behind these attacks\n\nAcross these five incidents, three distinct attack patterns emerge, and all of them exploit the implicit trust that CI/CD pipelines place in their inputs.\n\n### Pattern 1: Poisoned tools and actions\n\nThe TeamPCP campaign exploited a fundamental assumption: that the security tools running *inside* your pipeline are themselves trustworthy. When a GitHub Action tag or a PyPI package version resolves to malicious code, the pipeline executes it with full access to environment secrets, cloud credentials, and deployment tokens. There is no verification step because the pipeline trusts the tag.\n\n**A recommended pipeline-level control:** Pin tools and actions to immutable references (commit SHAs or image digests) rather than mutable version tags. Where pinning is not practical, verify the integrity of tools and dependencies against known-good checksums or signatures. Block execution if verification fails.\n\n### Pattern 2: Packaging misconfigurations that leak IP\n\nA misconfigured build pipeline shipped debugging artifacts straight into the production package. A misconfigured `.npmignore` or files field in package.json is all it takes. A pre-publish validation step should catch this every time.\n\n**A recommended pipeline-level control:** Before any package is published, run automated checks that validate the package contents against an allowlist, flag unexpected files (source maps, internal configs, .env files), and block the publish step if the checks fail.\n\n### Pattern 3: Vulnerabilities in transitive dependencies\n\nThe axios attack targeted not just direct users of axios, but anyone whose dependency tree resolved to the compromised version. A single poisoned dependency in a lockfile can thus propagate through an entire organization’s build infrastructure.\n\n**A recommended pipeline-level control:** Compare dependency checksums against known-good lockfile state. Detect unexpected new dependencies or version changes. Block builds that introduce unverified packages.\n\n## How GitLab Pipeline Execution Policies address each attack pattern\n\nGitLab Pipeline Execution Policies ([PEPs](https://docs.gitlab.com/user/application_security/policies/pipeline_execution_policies/)) enable security and platform teams to inject mandatory CI/CD jobs into every pipeline across an organization, regardless of what a developer defines in their `.gitlab-ci.yml`. Jobs defined in PEPs cannot be skipped, even with `[skip ci]` or `[no_pipeline]` directives. Jobs can be executed in *reserved* stages (`.pipeline-policy-pre` and `.pipeline-policy-post`) that bookend the developer’s pipeline.\n\nWe have published ready-to-use pipeline execution policies for all three patterns as an open-source project: [Supply Chain Policies](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies). These policies are independently deployable, and each one ships with violation samples that you can use to test them. Here is how each one works.\n\n### Use case 1: Prevent accidental exposure in package publishing\n\n**Problem:** A source map file ended up in the npm package of an AI coding tool after the build pipeline skipped publish-time validation.\n\n**PEP approach:** We built an open-source Pipeline Execution Policy for exactly this class of error: [Artifact Hygiene](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/artifact-hygiene.gitlab-ci.yml?ref_type=heads).\n\nThe policy injects `.pipeline-policy-pre` jobs that auto-detect the artifact type (npm package, Docker image, or Helm chart) and inspect the contents before any publish step runs. For npm packages, it performs three checks:\n\n1. **File pattern blocklist.** Scans npm pack output for source maps (.map), test directories, build configs, IDE settings, and src/ directories.\n\n2. **Package size gate.** Blocks packages exceeding 50 MB, like the 59.8 MB package that leaked the AI tool.\n\n3. **sourceMappingURL scan.** Detects external URLs (the R2 bucket pattern that exposed a major AI company’s source), inline data: URIs, and local file references embedded in JavaScript bundles.\n\nWhen violations are found, the pipeline fails with a clear report in the failed CI job logs:\n```text\n=============================================\nFAILED: 3 violation(s) found\n=============================================\nBLOCKED: dist/index.js.map (matched: \\.map$)\nBLOCKED: dist/index.js contains external sourceMappingURL\nBLOCKED: dist/utils.js contains inline sourceMappingURL\n\nThis check is enforced by a Pipeline Execution Policy. If this is a false positive, contact the security team to update the policy project or exclude this project.\n```\nThe policy has no user-configurable CI variables. Developers cannot disable or bypass it. Exceptions are managed by the security team at the policy level, ensuring a deliberate process and a clean audit trail.\n\nThe repository includes a test project with intentional violations (examples/leaky-npm-package/) so you can see the policy in action before deploying it to your organization. The [README](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/README.md) includes a complete quick-start guide for setup and deployment.\n\n**What this catches:** Any one of these controls would likely have prevented the AI company's source code leak:\n\n* The source map file triggers the file pattern blocklist.\n* Its 59.8 MB size triggers the size gate.\n* The sourceMappingURL pointing to an external R2 bucket triggers the URL scan.\n\n### Use case 2: Detect dependency tampering and lockfile manipulation\n\n**Problem:** The axios attack introduced a malicious transitive dependency (`plain-crypto-js`) that executed a RAT on install. Anyone who ran npm install during the compromise window pulled in the trojan.\n\n**PEP approach:** The [Dependency Integrity policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/dependency-integrity.gitlab-ci.yml) injects .pipeline-policy-pre jobs that auto-detect the package ecosystem (npm or Python) and perform three checks:\n\n**For npm projects** (triggered by `package-lock.json`, `yarn.lock`, or `pnpm-lock.yaml`):\n\n1. **Lockfile integrity.** Runs `npm ci --ignore-scripts`, which fails if `node_modules` would differ from what the lockfile specifies. This catches cases where package.json was updated but the lockfile was not regenerated, and also verifies SRI integrity hashes.\n2. **Blocked package scan.** Cross-references the lockfile’s full dependency tree against `blocked-packages.yml`, a GitLab-maintained list of known-compromised package versions. The shipped blocklist includes `axios@1.14.1`, `axios@0.30.4`, and `plain-crypto-js@4.2.1`.\n3. **Undeclared dependency detection.** After install, compares the contents of node_modules against the lockfile. Any package present on disk but absent from the lockfile indicates tampering (e.g., a compromised postinstall script that fetches additional packages).\n\n**For Python projects** (triggered by `requirements.txt`, `Pipfile.lock`, `poetry.lock`, or `uv.lock`):\n\n1. **Lockfile integrity.** Installs in an isolated virtual environment and verifies that the install succeeds from the lockfile.\n2. **Blocked package scan.** Same blocklist approach. The shipped list includes `litellm==1.82.7` and `litellm==1.82.8`.\n3. **.pth file detection.** Scans site-packages for `.pth` files containing executable code patterns (`import os`, `exec(`, `eval(`, `__import__`, `subprocess`, `socket`). This is the exact mechanism the LiteLLM backdoor used.\n\nWhen a violation is found:\n\n```text\n=============================================\nFAILED: 1 violation(s) found\n=============================================\nBLOCKED: axios@1.14.1 is a known-compromised package\n\nThis check is enforced by a Pipeline Execution Policy.\n```\n\nThe policy runs in *strict mode*: any dependency not present in the committed lockfile blocks the pipeline. If a developer needs to add a dependency, they commit the updated lockfile. The policy verifies that the installed version matches the committed version. If something appears that was not committed (e.g., a transitive dependency injected via a compromised upstream package), the pipeline blocks.\n\n**What this catches:** The introduction of `plain-crypto-js` as a new, previously unseen dependency would be flagged by the undeclared dependency check. The `axios@1.14.1` version would be caught by the blocked package scan. The LiteLLM `.pth` file would be caught by the `.pth` detection check. Each attack has at least one, and often two, independent detection signals.\n\n### Use case 3: Detect and block compromised tools before execution\n\n**Problem:** TeamPCP replaced trusted Trivy and Checkmarx GitHub Action tags with malicious versions. Any pipeline referencing those tags executed credential-stealing malware.\n\n**PEP approach:** The [Tool Integrity policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/tool-integrity.gitlab-ci.yml) injects a `.pipeline-policy-pre` job that queries the GitLab CI Lint API (or falls back to evaluate the `.gitlab-ci.yml`), extracts the container image references, and compares it against an approved images allowlist maintained by the security team.\n\nThe allowlist (`approved-images.yml`) supports three controls per image:\n\n**Approved repositories:** Only images from repositories on the list are permitted. An unknown repository blocks the pipeline.\n\n**Allowed tags:** Only specific tags are permitted within an approved repository. This prevents drift to untested versions.\n\n**Blocked tags:** Known-compromised versions can be explicitly blocked even if the repository is approved. The shipped allowlist blocks `aquasec/trivy:0.69.4` through `0.69.6`, the exact versions TeamPCP trojanized.\n\nWhen a violation is found, the pipeline fails before any other job runs:\n\n```text\n=============================================\nFAILED: 1 violation(s) found\n=============================================\nBLOCKED: aquasec/trivy:0.69.4 (job: trivy-scan)\n\n - tag '0.69.4' is known-compromised\n\nThis check is enforced by a Pipeline Execution Policy.\n```\n\nThe allowlist is maintained via MRs against the policy project. To add a new approved image, the security team opens an MR. To respond to a new compromise, they add a blocked tag. No code changes required, just YAML.\n\n**What this catches:** When images with unapproved tags are detected, the policy compares the image repository names and tags to an allowlist. A failed match blocks the pipeline before any scanner executes, preventing credential exfiltration.\n\n*Note: By extending the sample above, PEPs can be used to force pinning to digests over tags, which is immune to force pushes. This sample demonstrates a more basic tag-based enforcement pattern.*\n\n## Beyond PEPs: GitLab’s supply chain defenses\n\nPipeline Execution Policies are the enforcement layer, but they work best as part of a broader defense-in-depth strategy. GitLab provides several capabilities that complement PEPs for supply chain protection:\n\n### Secret detection\n\n[GitLab secret detection](https://docs.gitlab.com/user/application_security/secret_detection/) prevents credentials from landing in the repository in the first place, significantly reducing what a compromised pipeline tool can harvest. In the context of the March 2026 attacks:\n\n* Credentials stored in repositories are both easier for attackers to discover and slower to rotate. The Trivy incident showed that even the rotation process can be exploited: Aqua Security's rotation was not atomic, and the attacker captured newly issued tokens before the old ones were fully revoked. GitLab Secret Detection includes automatic revocation for leaked GitLab tokens and a partner API that notifies third-party providers to revoke their credentials, accelerating response when a breach does occur.\n\n* Secret detection combined with proper secret management (short-lived tokens, vault-backed credentials, minimal pipeline secret exposure) limits what an attacker can reach even when a trusted tool turns hostile.\n\n### Dependency scanning via software composition analysis (SCA)\n\nGitLab [dependency scanning](https://docs.gitlab.com/user/application_security/dependency_scanning/) identifies known vulnerabilities in project dependencies by analyzing lockfiles and manifests. In the context of the March 2026 attacks:\n\n* For LiteLLM, the compromised versions (1.82.7, 1.82.8) are tracked in GitLab's advisory database, flagging affected Python projects automatically.\n\n* For axios, dependency scanning identifies the compromised versions (1.14.1, 0.30.4) across every project in the organization, giving security teams a single view for assessing blast radius and prioritizing credential rotation.\n\n* Similarly, all npm packages compromised by TeamPCP's CanisterWorm propagation are also flagged if used.\n\n[GitLab Container Scanning](https://docs.gitlab.com/user/application_security/container_scanning/) detects vulnerable container images used in your deployments. For the Trivy compromise, Container Scanning flags the trojanized Trivy Docker images (0.69.4 through 0.69.6) when they appear in your container registry or deployment manifests.\n\n### Merge request approval policies\n\n[Merge request approval policies](https://docs.gitlab.com/user/application_security/policies/merge_request_approval_policies/) can require security team approval before changes to dependency lockfiles or CI/CD configurations are merged. This ensures a human checkpoint for the types of changes that supply chain attacks typically introduce.\n\n### Coming soon: Dependency Firewall, Artifact Registry, and SLSA Level 3 Attestation & Verification\n\nUpcoming GitLab supply chain security capabilities harden policy enforcement at two critical control points: the registry and the pipeline. The Dependency Firewall and Artifact Registry will block non-conforming packages, while SLSA Level 3 attestation will provide cryptographic proof that artifacts were produced by approved pipelines and remain unmodified. Together, they will give security teams verifiable control over what enters and exits the software supply chain.\n\n## What this means for your organization\n\nAmidst rising AI-assisted threats, attacks on CI/CD pipelines are becoming commonplace. The TeamPCP campaign shows how a single compromised credential can cascade across an ecosystem of trusted tools.\n\nIf your organization used any of the affected components, operate with the assumption that all of your pipeline secrets were exposed: rotate them immediately and audit systems for persisted backdoors. Either way, regularly rotating credentials and using short-lived tokens limits the blast radius of any future compromise.\n\nHere is what we recommend:\n\n1. **Pin dependencies to checksums, when possible.** Mutable version tags (like the ones TeamPCP hijacked) are not a security boundary. Use SHA-pinned references for all [CI/CD components](https://docs.gitlab.com/ci/components/#manage-dependencies) or actions and container images.\n\n2. **Run pre-execution integrity checks.** Use Pipeline Execution Policies to verify tool and dependency integrity *before* any pipeline job runs. This is the `.pipeline-policy-pre` stage.\n\n3. **Audit what you publish.** Every package publish step should include automated validation of the artifact contents. Source maps, environment files, and internal configuration should never leave your build environment. The [Supply Chain Policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies) project provides a ready-to-deploy starting point for npm, Docker, and Helm artifacts.\n\n4. **Detect dependency drift.** Compare dependency resolutions against committed lockfiles on every pipeline run. Monitor for unexpected new dependencies.\n\n5. **Centralize policy management.** Do not rely on developers remembering to include security checks. Enforce them at the group or instance level through policies that developers cannot remove or skip.\n\n6. **Assume your security tools are targets.** If your vulnerability scanner, static application security testing (SAST) tool, or AI gateway can be compromised, it will be. Limit each tool to its least necessary privileges and verify that it can't reach anything else.\n\n## Protect your pipelines with GitLab\n\nOver two weeks, attackers compromised production pipelines at organizations running some of the most widely adopted tools in the software development ecosystem.\n\nThe lesson is clear: Build pipelines need the same degree of centralized, policy-driven protection that we apply to networks and cloud infrastructure.\n\nGitLab Pipeline Execution Policies provide that enforcement layer. They ensure that security checks run on every pipeline, in every project regardless of individual project configurations. Combined with dependency scanning, secret detection, and merge request approval policies, they can block, detect, and contain the class of attacks we saw in March 2026.\n\nThe [Supply Chain Policies](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies) project provides a working Pipeline Execution Policy that catches the exact class of error behind the major AI company’s leak, with coverage for npm packages, Docker images, and Helm charts. Clone it, deploy it to your group, and ensure that all of your pipelines are ready for the supply chain attacks to come.\n\nTo get started with centralized pipeline policies, sign up for a [free trial of GitLab Ultimate](https://about.gitlab.com/free-trial/devsecops/).\n\n\n*This blog post contains \"forward-looking statements\" within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934. Although we believe that the expectations reflected in these statements are reasonable, they are subject to known and unknown risks, uncertainties, assumptions and other factors that may cause actual results or outcomes to differ materially. Further information on these risks and other factors is included under the caption \"Risk Factors\" in our filings with the SEC. We do not undertake any obligation to update or revise these statements after the date of this blog post, except as required by law.*","Pipeline security lessons from March supply chain incidents","Learn how centralized pipeline policies can detect and block the patterns behind a series of recent attacks.",[23,720,26,721],"product","features",[723],"Grant Hickman","https://res.cloudinary.com/about-gitlab-com/image/upload/v1772630163/akp8ly2mrsfrhsb0liyb.png","2026-04-07",{"featured":31,"template":13,"slug":727},"pipeline-security-lessons-from-march-supply-chain-incidents",{"content":729,"config":740},{"body":730,"category":9,"date":731,"tags":732,"title":735,"description":736,"authors":737,"heroImage":739},"After an incident wraps up, every incident response or security operations center faces the same uncomfortable question: What did we miss, and why? Answering that question well takes real work — someone has to read through the incident timeline, map the attacker's actions to detection opportunities, identify the alerts that should have fired but didn't, and translate those findings into concrete improvements. Done manually, it's time-consuming, inconsistent, and easy to deprioritize when the next incident is already knocking.\n\nAt GitLab, our Signals Engineering team is responsible for building and maintaining the detections that protect the platform and the company. We deal with the same detection gap problem that every security team does so we’ve automated detection gap analysis with [GitLab Duo Agent Platform](https://about.gitlab.com/gitlab-duo-agent-platform/) to improve our assessment of those gaps and how we can close them.\n\nIn this article, you'll learn our strategy, which includes two AI agents you can use in your environment: the built-in Security Analyst Agent and a custom agent we built and named the Detection Engineering Assistant.\n\n\n## The detection gap problem\n\nA detection gap is exactly what it sounds like: an attacker took an action, and your detections didn't catch it. Gap analysis is the process of systematically reviewing security incidents to identify those missed opportunities and determine what new or improved detections would close them.\n\nThe challenge isn't that gap analysis is conceptually hard. It's that it requires careful, methodical reading of incident data and mapping those events to your detection coverage. For a single incident, a skilled analyst can do it well. But across a steady stream of incidents, with multiple engineers contributing, it's difficult to maintain consistency and easy to let the review become shallow.\n\nWe wanted a process that was repeatable, thorough, and embedded directly in the workflow where our security incidents already live: GitLab issues.\n\n## What is GitLab Duo Agent Platform?\n\n[GitLab Duo Agent Platform](https://about.gitlab.com/blog/gitlab-duo-agent-platform-is-generally-available/) is GitLab's framework for building and deploying agentic AI agents that can reason, take actions, and integrate natively with GitLab resources like issues, merge requests, and code. Unlike a simple chat interface, agents in Duo Agent Platform can be given specific roles, domain knowledge, and access to tools, making them effective for domain-specific workflows like security operations.\n\nGitLab Duo Agent Platform gives you two practical paths:\n\n1. **Use a pre-built agent** — GitLab ships several out-of-the-box agents, including a Security Analyst Agent designed for security-related tasks.  \n2. **Build your own agent** — You can create a custom agent in just a few minutes by giving it a name, a description, and a system prompt. The system prompt is where the real power lies.\n\nBoth paths are viable for detection gap analysis. Let's look at each.\n\n## 1. Security Analyst Agent\n\nThe easiest way to get started is with [Security Analyst Agent](https://docs.gitlab.com/user/duo_agent_platform/agents/foundational_agents/security_analyst_agent/), which comes pre-configured with security domain knowledge and can be invoked directly from a GitLab issue.\n\nTo use the agent for gap analysis, we navigate to a closed incident issue and ask the agent to review the incident description, timeline, tasks, and comments to identify where detections were absent or insufficient. The agent reads the issue content — including comments, linked artifacts, and timeline details — and reasons over it to surface potential gaps. It can identify undetected tactics, techniques, and procedures (TTPs) mapped to MITRE ATT&CK and suggest areas where new detection rules could improve coverage.\n\nThis works well for a quick first pass, especially if your incident issues are well-documented. Security Analyst Agent is knowledgeable about general security concepts, common attacker behaviors, and detection principles. For teams just getting started with AI-assisted operations, it provides immediate value with no configuration required.\n\nThat said, the pre-built agent doesn't know your specific environment, including your SIEM, your log sources, your detection stack, or your team's detection engineering standards. For us, that meant the recommendations, while valid in general, sometimes missed the specific context we needed to translate them into actionable detections. That's what led us to build our own agent.\n\n## 2. Building the Detection Engineering Assistant\n\n[Creating a custom agent in GitLab Duo Agent Platform](https://docs.gitlab.com/user/duo_agent_platform/agents/custom/) is surprisingly straightforward. From the Duo Agent Platform interface, you give the agent a name (we called ours the **Detection Engineering Assistant**), a brief description, and a system prompt. That's it. The agent is ready to use.\n\nThe system prompt is the most important part. It's the agent's knowledge base: everything it knows about your team, your environment, your standards, and how it should reason about its work. A thin, vague system prompt produces thin, vague output. A verbose, carefully crafted system prompt produces an agent that behaves like a knowledgeable member of your team.\n\nHere's the approach we took when writing our system prompt for the Detection Engineering Assistant:\n\n### Define the agent's role and scope clearly\n\nWe opened the system prompt by telling the agent exactly what it is and what it's responsible for. Not just \"you are a security analyst.\" We specifically prompted: \"You are a detection engineering assistant for GitLab's Signals Engineering team, responsible for analyzing security incidents and identifying gaps in our detection coverage.\" This framing anchors every response it produces.\n\n### Encode your detection philosophy\n\nWe wrote out what \"a good detection\" means to us: low false positive rates, high signal fidelity, and actionable alerts that provide responders with the context they need. We explained our preference for behavioral detections over IOC-based detections where possible, and described how we think about the tradeoff between coverage breadth and alert fatigue.\n\n### Give it context on your tech stack and log sources\n\nAn agent can only recommend what you can actually build. We told the agent which log sources we ingest, what our SIEM looks like, and what data is and isn't available to us. This means when it recommends a new detection, it does so in terms of what we can actually implement, not hypothetical telemetry we don't have.\n\n### Ground it in MITRE ATT&CK\n\nWe told the agent to organize its gap findings using ATT&CK tactics and techniques. This gives us consistent, structured output that maps directly to how we track coverage internally, and makes it easy to prioritize which gaps to address first.\n\n### Set expectations for output format\n\nWe specified exactly what we want the agent to produce: a structured list of detection gaps, each with the relevant ATT&CK technique, a description of what was missed, the log source or data that could support a detection, and a recommended approach. A consistent output format makes the findings easier to triage and turn into engineering work.\n\n### Example system prompt excerpt\n\n*Note: Our full Detection Engineering Assistant system prompt is 1,870 words and 337 lines. The example below is just a small example of what a full custom system prompt can be.* \n\n\n```text\nYou are the Detection Engineering Assistant for GitLab's Security Operations team. Your role is to analyze closed security incidents and identify gaps in our detection capabilities.\n\nWhen reviewing an incident, you should:\n1. Identify each distinct attacker action or technique described in the incident timeline\n2. For each action, assess whether our existing detections would have caught it\n3. For any action that would not have been detected, document it as a detection gap\n\nFor each gap, provide:\n- MITRE ATT&CK Technique ID and name (e.g., T1078 - Valid Accounts)\n- A plain-language description of what happened and why it wasn't detected\n- The log source or telemetry that could support a detection (e.g., authentication logs, process execution events, network flow data)\n- A recommended detection approach, written in terms our SIEM can implement\n\nOur SIEM ingests [log sources]. Our detection standards prioritize behavioral patterns over static IOCs. Avoid recommending detections that would generate significant false positives without a high-confidence tuning path...\n```\n\nA system prompt this specific produces dramatically more useful output than a generic one. The agent stops giving you general security advice and starts giving you detection engineering recommendations.\n\n## Running gap analysis on incidents\n\nWith the Detection Engineering Assistant configured, the workflow is simple. At the close of an incident, we open the incident issue in GitLab and invoke the assistant. It reads the full issue — the incident summary, timeline, investigative notes, and any linked resources — and returns a structured gap analysis.\n\nA typical output looks like this:\n\n**Gap: Lateral movement via valid credentials not detected**\n\n* **ATT&CK:** T1078.004 — Valid Accounts: Cloud Accounts  \n* **What happened:** An attacker used a valid access token to authenticate to an auxiliary GitLab instance. No alert fired because we lacked authentication baseline detections for that instance.  \n* **Log source:** Authentication logs from `example.gitlab.com`  \n* **Recommended approach:** Create a detection that alerts on first-time authentication from a user account to `example.gitlab.com` within a 90-day rolling window, with suppression for accounts with established access patterns.\n\nThis kind of structured output goes directly into our engineering backlog. We treat the agent's analysis as a high-quality first draft. It gets reviewed by a human engineer who validates the findings, checks whether gaps are already covered by detections we haven't documented, and adds context before it becomes an engineering issue. But the hard work of reading the incident and generating the initial findings is automated.\n\n## What we've learned\n\nA few things stand out from building and iterating on this workflow:\n\n**The system prompt is a living document** — Every time the agent produces an output that misses something obvious or gets the framing wrong, we update the prompt. The agent's quality is a direct reflection of how well we've encoded our domain knowledge into it.\n\n**Incident documentation quality matters** — An agent can only reason over what's written down. Incidents with detailed, structured timelines produce much better gap analysis than sparse or informal ones. Building the gap analysis workflow created an unexpected second benefit: it gave us a concrete reason to improve our incident documentation standards.\n\n**This is a force multiplier, not a replacement** — The Detection Engineering Assistant doesn't replace a skilled detection engineer, but it does amplify one. The engineer still reviews the findings, validates the recommendations, and makes the final call on what goes into the backlog. But the time spent on the initial analysis drops significantly, and the consistency across incidents improves.\n\n## Get started\n\nIf you want to build your own detection gap analysis agent, here's where to start:\n\n1. Review your last three to five closed incidents and note what a good gap analysis would have surfaced for each.  \n2. Use those observations to draft a system prompt that encodes your environment, standards, and preferred output format.  \n3. Create a [custom agent](https://docs.gitlab.com/user/duo_agent_platform/agents/custom/) in GitLab Duo Agent Platform with your prompt.  \n4. Run it against one of your incidents and iterate on the prompt based on the output.\n\nThe detection gap problem isn't going away. But with GitLab Duo Agent Platform, you can make the analysis repeatable, consistent, and embedded directly in the place where your security work already happens. \n\n> Start [a free trial of GitLab Duo Agent Platform](https://about.gitlab.com/gitlab-duo-agent-platform/) today!\n","2026-03-10",[23,733,26,734,721,720,532],"security research","AI/ML","Automating detection gap analysis with GitLab Duo Agent Platform","Learn how GitLab's Signals Engineering team uses our AI platform to automatically surface detection gaps from security incidents — no manual review required.",[738],"Matt Coons","https://res.cloudinary.com/about-gitlab-com/image/upload/v1773147991/op5xyroonltdwqix0x3u.png",{"featured":12,"template":13,"slug":741},"automating-detection-gap-analysis-with-gitlab-duo-agent-platform",{"content":743,"config":751},{"title":744,"description":745,"authors":746,"heroImage":724,"date":748,"body":749,"category":9,"tags":750},"How GitLab built a security control framework from scratch","GitLab's Security Compliance team created a custom control framework to scale across multiple certifications and products — here's why and how you can, too.\n",[747],"Davoud Tu","2026-03-04","GitLab's Security Compliance team discovered that existing security control frameworks lacked the customization to fit the platform's multi-product, cloud-native environment.\n\nSo we built our own.\n\nHere's what we learned and why creating your own custom security control framework might be the right move for your compliance program.\n\n## The journey through frameworks\n\nWhen I joined GitLab's Security Compliance team in November 2022, we were using the [Secure Controls Framework](https://securecontrolsframework.com/) to manage controls across our external certifications and internal compliance needs. But as our requirements grew, we realized we needed something more comprehensive. \n\nWith FedRAMP authorization on our roadmap, we chose to adopt [NIST SP 800-53](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) next. NIST SP 800-53 includes more than 1,000 controls, but its comprehensiveness isn’t perfectly suited to GitLab’s environment.\n\nWe didn't need to implement every NIST control, only those applicable to our specific requirements. Our focus was on the quality of controls rather than quantity. Implementing unnecessary controls doesn't improve security; in fact, too many can make an environment less secure as individuals find ways to circumvent overly restrictive or irrelevant controls. \n\nSome controls also lacked the necessary granularity for our needs. For example, NIST’s AC-2 “Account Management” control covers account creation and provisioning, account modification and disabling, account removal and termination, shared and group account management, and account monitoring and reviews.\n\nIn practice, these are _at least_ six distinct controls with different owners, testing procedures, and risks. For attestations like SOC 2, each activity is tested as a separate control because they have different evidence requirements and operational contexts. NIST's all-encompassing AC-2 didn't match how we actually operate controls or how auditors actually assess us, and we needed controls granular enough to reflect our operational environment.  \n\nWe found ourselves constantly customizing, adding, and adapting NIST controls to fit our environment. At some point, we realized we weren't really using NIST SP 800-53 anymore, we were building our own framework on top of it. We decided a custom control framework, one tailored to GitLab’s environment, would best accommodate our multi-product offering and each product’s unique compliance needs.\n\n## Building the GitLab Control Framework\n\nThrough five methodical steps, we built our own common controls framework: the GitLab Control Framework (GCF).\n\n### 1. Analyze what we need\n\nWe reviewed our existing controls and mapped every requirement from external certifications we already maintained, certifications on our roadmap, and our internal compliance program: \n\n**External certifications:**\n\n* SOC 2 Type II  \n* ISO 27001, ISO 27017, ISO 27018, ISO 42001  \n* PCI DSS  \n* TISAX  \n* Cyber Essentials  \n* FedRAMP\n\n**Internal compliance needs:**\n\n* Controls for mission-critical systems that are not in-scope for external certifications   \n* Controls for systems with access to sensitive data\n\nThis gave us the baseline: what controls must exist to meet our compliance obligations.\n\n### 2. Learn from industry frameworks\n\nNext, we compared our requirements against industry-recognized frameworks:\n\n* NIST SP 800-53  \n* NIST Cybersecurity Framework (CSF)  \n* Secure Controls Framework (SCF)  \n* Adobe and Cisco Common Controls Framework (CCF)\n\nHaving adopted frameworks in the past, we wanted to learn from their structure and ensure we weren't missing critical security domains, controls, or best practices.\n\n### 3. Create custom control domains\n\nThrough this analysis, we created 18 custom control domains tailored to GitLab's environment:\n\n\n| Abbreviation | Domain | Scope of controls |\n| :---- | :---- | :---- |\n| AAM | Audit & Accountability Management | Logging, monitoring, and maintaining audit trails of system activities |\n| AIM | Artificial Intelligence Management | Specific to AI system development, deployment, and governance |\n| ASM | Asset Management | Identifying, tracking, and managing organizational assets |\n| BCA | Backups, Contingency, and Availability Management | Business continuity, disaster recovery, and system availability |\n| CHM | Change Management | Managing changes to systems, applications, and infrastructure |\n| CSR | Customer Security Relationship Management | Customer communication, transparency, and security commitments |\n| DPM | Data Protection Management | Protecting data confidentiality, integrity, and privacy |\n| EPM | Endpoint Management | Securing end-user devices and workstations |\n| GPM | Governance & Program Management | Security governance, policies, and program oversight |\n| IAM | Identity, Authentication, and Access Management | User identity, authentication mechanisms, and access control |\n| INC | Incident Management | Detecting, responding to, and recovering from security incidents |\n| ISM | Infrastructure Security Management | Network, server, and foundational infrastructure security |\n| PAS | Product and Application Security Management | Security capabilities built into the GitLab product that are dogfooded to secure GitLab's own development, such as branch protection & code security scanning |\n| PSM | People Security Management | Personnel security, training, and awareness |\n| SDL | Software Development & Acquisition Life Cycle Management | Secure SDLC practices and third-party software acquisition |\n| SRM | Security Risk Management | Risk assessment, treatment, and management |\n| TPR | Third Party Risk Management | Managing security risks from vendors and suppliers |\n| TVM | Threat & Vulnerability Management | Identifying and remediating security vulnerabilities |\n\n\u003Cbr>\u003C/br>\n\n\nEach domain groups related controls into logical families that align with how GitLab's security program is actually organized and operated. This structure provides a methodical approach for adding, updating, or removing controls as our needs evolve.\n\n### 4. Add context and data\n\nWith our domains defined, we needed to address two critical challenges: how to represent controls across multiple products without duplicating the framework, and how to capture meaningful implementation context to actually operate and audit at scale. \n\n#### Scaling across multiple products\n\nGitLab provides multiple product offerings: GitLab.com (multi-tenant SaaS on GCP), GitLab Dedicated (single-tenant SaaS on AWS), and GitLab Dedicated for Government (GitLab’s single-tenant FedRAMP offering on AWS). Each offering has different infrastructure, compliance scopes, and audit requirements. We needed to support product-specific audits without creating entirely separate frameworks.\n\nWe designed a control hierarchy where **Level 1 controls are the framework**, defining what should be implemented at the organizational level. **Level 2 controls are the implementation**, capturing the product-specific details of how each requirement is actually fulfilled.\n\n```mermaid\n%%{init: { \"fontFamily\": \"GitLab Sans\" }}%%\ngraph TD\n    accTitle: Control Hierarchy\n    accDescr: Level 1 requirements cascade to Level 2 implementations.\n    \n    L1[\"Level 1: Framework\u003Cbr/>What must be implemented\"];\n    L2A[\"Level 2: GitLab.com\u003Cbr/>How it's implemented\"];\n    L2B[\"Level 2: Dedicated\u003Cbr/>How it's implemented\"];\n    L2C[\"Level 2: Dedicated for Gov\u003Cbr/>How it's implemented\"];\n    L2D[\"Level 2: Entity\u003Cbr/>(inherited by all)\"];\n    \n    L1-->L2A;\n    L1-->L2B;\n    L1-->L2C;\n    L1-->L2D;\n```\n\n\u003Cbr>\u003C/br>\n\nThis separation allows us to maintain one framework with product-specific implementations, rather than managing duplicate frameworks for each offering. Entity controls apply organization-wide and are inherited by GitLab.com, GitLab Dedicated, and GitLab Dedicated for Government.\n\n#### Adding context to controls\n\nTraditional control frameworks track minimal information: a control ID, description, and owner. The GCF takes a different approach and its superpower is the extensive metadata we track for each control. Beyond just stating the control description or implementation statement, we capture:\n\n* Control owner: Who is accountable for the control and its risk?  \n* Environment: Does this apply organization-wide (Entity, inherited by all product offerings), to GitLab.com, or to Dedicated?  \n* Assets: What specific systems does this control cover?  \n* Frequency: How often is the control performed or tested?  \n* Nature: Is it manual, semi-automated, or fully automated?  \n* Classification: Is this for external certifications or internal risk?  \n* Testing details: How do we assess it? What evidence do we collect?\n\nThis context transforms the GCF from a simple control list into an operationalized control inventory.\n\nWith this structure, we can answer questions like: \n\n* Which controls apply to GitLab.com for our SOC 2 audit vs. GitLab Dedicated? → Filter by environment: GitLab.com  \n* What controls does the Infrastructure team own? → Filter by owner   \n* Which controls can we automate? → Filter by nature: Manual \n\n### 5. Iterate, mature, and scale\n\nThe GCF isn't static and was designed to evolve with our business and compliance landscape.\n\n#### Pursuing new certifications\n\nBecause we've operationalized context into the GCF, we can quickly determine the scope and gaps when pursuing new certifications (ISMAP, IRAP, C5, etc.): \n\n1. Determine scope: Which product has the business need (GitLab.com, GitLab Dedicated, or both)?\n2. Map requirements: Do existing controls already cover the new certification requirements?   \n3. Identify gaps: What new controls need to be created?  \n4. Update mappings: Link existing controls to the new certification requirements.\n\n#### Adapting to new regulations\n\nWhen new regulations emerge or existing requirements change: \n\n* Review existing controls: Does an existing control already cover the new requirement?   \n* Update or create: Either update existing control language or create a new control.  \n* Apply the most stringent: When multiple certifications have similar requirements, we implement the most stringent version — secure once, comply with many.\n* Map across certifications: Link the control to all relevant certification requirements.\n\n#### Managing control lifecycle\n\nThe framework adapts to various changes:\n\n* Requirement changes: When certifications update their requirements, we review impacted controls and update descriptions or mappings.\n* Deprecated controls: If a requirement is removed or a control is no longer needed, we mark it as deprecated and remove it from our monitoring schedule.  \n* New risks identified: Risk assessments may identify gaps requiring new internal controls.\n\n## The power of common controls: One control, multiple requirements\n\nSecuring once and complying with many isn't just a principle, it has tangible benefits across how we prepare for audits, support control owners, and pursue new certifications. Here's what that looks like in practice, both qualitatively and in the numbers. \n\n### Qualitative results\n\nSince implementing the GCF, we've seen significant improvements in how we manage compliance: \n\n#### Integrated audit approach\n\nThe GCF enables us to maintain one framework with controls mapped to multiple certification requirements, instead of managing separate control sets for each audit. One control can satisfy SOC 2, ISO 27001, and PCI DSS requirements simultaneously.\n\n#### Faster audit preparation\n\nThrough the GCF, we maintain one consolidated request list instead of separate lists for each audit. Because we've defined controls with specific context, our request lists say \"Okta user list\" instead of generic \"production user list,\" eliminating ambiguity and interpretation. We're not collecting “N/A” evidence or leaving it up to auditors to interpret what \"production\" means in our environment. Everything is already scoped to our actual systems.\n\n#### Reduced stakeholder burden\n\nThis integration directly reduces burden on our stakeholders. Control owners provide evidence once instead of responding to separate requests from SOC 2, ISO, and PCI auditors. When we collect evidence for access controls, it satisfies SOC 2, ISO 27001, and PCI DSS requirements simultaneously. One control, one test, one piece of evidence with multiple certifications and requirements satisfied.\n\n#### Efficient gap assessments\n\nWhen pursuing new certifications or launching new features, the operationalized context enables more efficient gap analysis. We can determine which controls already exist, what's missing, and what implementation is required. \n\n### Quantifiable results\n\n**Control efficiency:**\n\n* Reduced SOC controls by 58% (200 controls → 84\\) for GitLab.com and 55% (181 → 82) for GitLab Dedicated  \n* One framework now supports 8+ certifications \n\n**Audit efficiency:**\n\n* Consolidated 4 audit request lists into 1, reducing requests by 44% (415 → 231)  \n* 95% evidence acceptance rate before fieldwork for recent PCI audits\n\n**Framework scale:**\n\n* 220+ active controls across 18 custom domains  \n* Mapped to 1,300+ certification requirements  \n* Supports multiple product offerings\n\n## The path forward\n\nThe GCF continues to evolve as we add security and AI controls, pursue new certifications, and refine our approach. \n\n**For security compliance practitioners:** Don't be afraid to build your own framework if industry standards don't fit. The upfront investment pays dividends in scalability, efficiency, and controls that actually make sense for your environment. Sometimes the best framework is the one you design yourself.\n\n> If you found this helpful, check out our complete [GitLab Control Framework documentation](https://handbook.gitlab.com/handbook/security/security-assurance/security-compliance/sec-controls/), where we detail our framework methodology, control domains, and field structures.",[23,26],{"featured":12,"template":13,"slug":752},"how-gitlab-built-a-security-control-framework-from-scratch",{"promotions":754},[755,769,780,791],{"id":756,"categories":757,"header":759,"text":760,"button":761,"image":766},"ai-modernization",[758],"ai-ml","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":762,"config":763},"Get your AI maturity score",{"href":764,"dataGaName":765,"dataGaLocation":241},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":767},{"src":768},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":770,"categories":771,"header":772,"text":760,"button":773,"image":777},"devops-modernization",[720,567],"Are you just managing tools or shipping innovation?",{"text":774,"config":775},"Get your DevOps maturity score",{"href":776,"dataGaName":765,"dataGaLocation":241},"/assessments/devops-modernization-assessment/",{"config":778},{"src":779},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":781,"categories":782,"header":783,"text":760,"button":784,"image":788},"security-modernization",[23],"Are you trading speed for security?",{"text":785,"config":786},"Get your security maturity score",{"href":787,"dataGaName":765,"dataGaLocation":241},"/assessments/security-modernization-assessment/",{"config":789},{"src":790},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":792,"paths":793,"header":796,"text":797,"button":798,"image":803},"github-azure-migration",[794,795],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":799,"config":800},"See how GitLab compares to GitHub",{"href":801,"dataGaName":802,"dataGaLocation":241},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":804},{"src":779},{"header":806,"blurb":807,"button":808,"secondaryButton":813},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":809,"config":810},"Get your free trial",{"href":811,"dataGaName":49,"dataGaLocation":812},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":503,"config":814},{"href":53,"dataGaName":54,"dataGaLocation":812},1776444481592]