Controlling the Source: Abusing Source Code Management Systems
For full details on this research, see the X-Force Red whitepaper “Controlling the Source: Abusing Source Code Management Systems”. This material is also being presented at Black Hat USA 2022.
Source Code Management (SCM) systems play a vital role within organizations and have been an afterthought in terms of defenses compared to other critical enterprise systems such as Active Directory. SCM systems are used in the majority of organizations to manage source code and integrate with other systems within the enterprise as part of the DevOps pipeline, such as CI/CD systems like Jenkins. These SCM systems provide attackers with opportunities for software supply chain attacks and can facilitate lateral movement and privilege escalation throughout an organization.
This blog post will review a background on SCM systems, along with detailing ways to abuse some of the most popular SCM systems including GitHub Enterprise, GitLab Enterprise and Bitbucket to perform various attack scenarios. These attack scenarios include reconnaissance, manipulation of user roles, repository takeover, pivoting to other DevOps systems, user impersonation, and maintaining persistent access. X-Force Red’s source code management attack toolkit (SCMKit) will also be shown to perform and facilitate these attacks. Additionally, defensive guidance for protecting these SCM systems will be outlined.
There are many ways to interact with and track source code, along with compiled source code assets. Some of the common terms used in this process are source control, version control, and source code management. The terms “source control” and “version control” are often used interchangeably with each other. However, there are differences between these two terms. Source control is specifically for tracking changes in source code, whereas version control also includes tracking changes for binary files and other file types. An example of this would be version control tracking changes to compiled executables, whereas source control would be tracking the changes to the underlying C# or C++ source files that were compiled into that executable. Git is a popular source control tool, and Subversion is a popular version control tool.
To use source control in a practical manner as part of the development process, source code management (SCM) systems are used. These systems allow tracking changes to source code repositories and allow developers to resolve conflicts when merging code commits from multiple people concurrently. Some of these SCM systems are more popular than others and have been adopted by enterprises, as they integrate into the development process in a more reliable manner. Some of the popular SCM systems include GitHub Enterprise, GitLab Enterprise and Bitbucket. Due to the role and importance of these SCM systems, they can be abused to facilitate software supply chain attacks and lateral movement within an organization.
SCM systems are heavily used during the “build” phase of a project in the DevOps pipeline, as shown in the below diagram. All other phases depend on the source code that is developed and maintained within the SCM system. Once a source code project is ready to be compiled and built, it will get pushed to a Continuous Integration (CI) server. After that, it will be tested, scanned, and deployed for use in production.
DevOps Pipeline Diagram
An attack that has been gaining popularity recently is software supply chain attacks. In this attack, an attacker injects itself into the development process at one of the phases to deploy malicious code into production. This is typically performed in the “build” phase. For organizations that provide software to other organizations, this can enable the compromise of multiple organizations. One of the most notable software supply chain attacks was the SolarWinds breach, which impacted many organizations in the private and public sector.
The below diagram shows the opportunities an attacker has during the development process to implement a software supply chain attack. This research focuses on the highlighted areas of “B” and “C”, as it relates to the compromise of SCM systems. However, the compromise of these SCM systems can also lead to other scenarios such as “D” where an attacker can use an SCM system to compromise a build platform system.
Software Supply Chain Attack Opportunity Diagram
SCM systems can be used as an initial access point to other DevOps systems that are used in separate phases of the DevOps lifecycle. Being able to pivot to the build system to compromise the CI/CD platform or pivoting to the package repository system to compromise the distribution platform are other scenarios where an attacker could perform a software supply chain attack. Two detailed examples of this are shown in the whitepaper.
The below attack scenarios are notable for an attacker to attempt against GitHub Enterprise, GitLab Enterprise, and Bitbucket. These have been useful as a part of X-Force Red’s Adversary Simulation engagements. This is not an exhaustive list of every single attack path available to execute on these SCM systems. The table, below, lists the attack scenarios that are detailed in the whitepaper. A few of these scenarios will be shown in subsequent sections.
Yes
No
Yes
No
Table of SCM attack scenarios
Using administrative access (specifically site admin role), an attacker can give themselves write access to any repository within GitHub Enterprise. In the below example, we are attempting to view a repository in GitHub Enterprise that our compromised administrative user (adumbledore) does not have access to.
Viewing locked repository
Using the site admin role, you can choose to unlock the repository via the “Unlock” button shown below. This will unlock the repository for the user for two hours by default.
Viewing screen to unlock repository
You must provide a reason to unlock the repository, and this reason is logged along with the request.
Adding reason to unlocking repository
Now you can see we have successfully unlocked the repository, and it is unlocked for two hours for the adumbledore user account. Code within this repository can then be accessed or modified.
Showing repository has been unlocked
There are two options an attacker has if they have administrative access to GitLab Enterprise or GitHub Enterprise and would like to impersonate another user. The first option is to impersonate a user login via the web interface, and the second option is to create an impersonation token. This will be shown specifically with GitLab Enterprise.
When viewing a user via the admin area in GitLab Enterprise, there is a button available in the top right-hand corner labeled “Impersonate”.
Impersonate user button in hpotter profile
After clicking the “Impersonate” button, you will be logged in as the user you are wanting to impersonate. In this instance, we are impersonating the hpotter user account.
Showing impersonation of hpotter
An attacker with admin access to GitLab Enterprise can also impersonate another user by creating an impersonation token. This can be performed via the web interface or the Users REST API. Using the web interface as an admin, you can navigate to the “Impersonation Tokens” section for the user account that you would like to impersonate; add the details for your token including name, expiration date, and scope of permissions.
Creating impersonation token
After you have created your impersonation token, the token value will be listed for use. The user that is impersonated cannot see this impersonation token when accessing GitLab Enterprise as themselves; it is only visible to other admin users.
Showing created impersonation token
An attacker can also create an impersonation token via the Users REST API as shown with the example curl command, below.curl -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=someName-impersonate" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_user" --data "scopes[]=read_repository" --data "scopes[]=write_repository" --data "scopes[]=sudo" "https://gitlabHost/api/v4/users/userIDNumberToImpersonate/impersonation_tokens"
Output after creating impersonation token via API
In Bitbucket, there is a feature called Bamboo that can be installed and configured to facilitate a CI/CD pipeline. If a repository is using a CI/CD pipeline with Bamboo, it will contain a directory named “bamboo-specs” within the root of the repository, along with a Bamboo configuration file. This configuration file will either be a YAML file (bamboo.yaml) or a Java spec file (pom.xml). If an attacker would like to discover any repositories that are configured with a CI/CD pipeline via Bamboo, they can search for “bamboo-specs” in either the web interface or REST API.
Discovering repos with CI/CD integration via Bamboo
If you have write access or admin access to a repository, the Bamboo configuration file can be modified. In this case, we are modifying the bamboo.yaml file to add our SSH key to the server where the Bamboo agent is running. This can be performed via the Git command line tool as well to commit the changes to the Bamboo configuration file.
Modifying Bamboo yaml file
This will immediately trigger the CI/CD pipeline to run as shown below.
Showing successful job status
When viewing the output from the pipeline, we can see our SSH key was added, and it printed the hostname of the server where the SSH key was added. At this point, we would be able to perform lateral movement to that server via our SSH key.
Viewing pipeline logs
An attacker has three primary options in terms of maintaining persistent access to SCM systems. This can be performed either by creating a personal access token, impersonation token, or adding a public SSH key. All these options are detailed in the whitepaper.
The persistence option that will be shown is creating a personal access token in GitLab Enterprise. This can be performed via the web interface as a regular user or can be performed via the Users REST API as an administrator. The below screenshot shows creating a personal access token called “persistence-token”.
Creating personal access token for hpotter user
You can see the created personal access token and the token value, below.
Showing token value created
An attacker can also create a personal access token via the Users REST API as shown with the below example curl command. In GitLab Enterprise, this requires admin permissions.curl -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=hgranger-persistence-token" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_repository" --data "scopes[]=write_repository" "https://gitlabHost/api/v4/users/UserIDNumber/personal_access_tokens"
Creating access token via API
At X-Force Red, we wanted to take advantage of the REST API functionality available in the most common SCM systems seen during engagements and add the most useful functionality in a proof-of-concept tool called SCMKit. The goal of this tool is to provide awareness of the abuse of SCM systems, and to encourage the detection of attack techniques against SCM systems. SCMKit will be shown at Black Hat USA 2022 Arsenal.
SCMKit allows the user to specify the SCM system and attack module to use, along with specifying valid credentials (username/password or API key) to the respective SCM system. Currently, the SCM systems that SCMKit supports are GitHub Enterprise, GitLab Enterprise and Bitbucket Server. The attack modules supported include reconnaissance, privilege escalation and persistence. Other functionality that are available in the non-public version of SCMKit were not included in consideration for defenders, such as user impersonation and built-in credential searching. SCMKit was built in a modular approach so that new modules and SCM systems can be added in the future by the information security community. The tool and full documentation are available on the X-Force Red GitHub. A few examples will be shown in the next sections.
SCMKit has multiple modules available to perform reconnaissance of repositories, files, code, and other resources specific to various SCM systems such as GitLab Runners. The below example shows using the “codesearch” module in SCMKit. In this scenario, we are searching for any code in Bitbucket Server that contains “API_KEY” to try and discover API key secrets within source code.
Code search example for API key with SCMKit
There are several other reconnaissance modules that apply only to certain SCM systems. For example, there is a reconnaissance module to discover GitLab Runners that you have access to via the “runnerlist” module.
GitLab Runner reconnaissance example with SCMKit
Another capability available in SCMKit is to add another regular user to the admin role. The example, below, shows adding a regular user under our control (hgranger, in this case) to the site admin role in GitHub Enterprise via the “addadmin” module.
Adding site admin example via SCMKit
You can see the change that took effect in GitHub Enterprise after performing the site admin addition via SCMKit, as the hgranger user is now a member of the site admins group.
Showing hgranger added as site admin
There are two persistence modules within SCMKit that include the use of personal access tokens or SSH keys. This can be useful to maintain access to an SCM system. The below example shows creating an SSH key for the hgranger user account in Bitbucket via the “createsshkey” module.
Creating SSH key example with SCMKit
We can list all active SSH keys for a given user via the “listsshkey” module, as shown below. You will see the SSH key we added for the hgranger user listed.
Listing SSH keys example with SCMKit
There are multiple static signatures that can be used to detect the usage of SCMKit. These can be found in the Yara rule on the SCMKit repository.
A static user agent string is used when attempting each module in SCMKit. The user agent string is “SCMKIT-5dc493ada400c79dd318abbe770dac7c”. A Snort rule is provided on the SCMKit repository.
Additionally, any access tokens or SSH keys that are created in SCM systems using SCMKit will be prepended with “SCMKit-” in the name. This can be filtered in the respective SCM system to indicate an access token or SSH key was created using SCMKit.
Ensure that the below logs are being sent to your SIEM. This also lists the location of the logs on the server for each respective SCM system.
Logs of interest – GitHub Enterprise
/var/log/gitlab/gitlab-rails/application_json.log
/var/log/gitlab/gitlab-rails/production.log
Logs of interest – GitLab Enterprise
Logs of interest – Bitbucket Server
Below are the various filters you can apply to the logs to detect the attacks demonstrated in this research. Use these filters to build a baseline and detect anomalous activity in your environment.
Table of search queries for various attack types – GitHub Enterprise
API Log
Web Log
‘get’ AND ‘/search’
‘get’ AND (‘/search’ OR ‘repository/tree’)
‘search’
Production Log
API Log
‘impersonate’
‘post’ AND ‘impersonation_tokens’
‘impersonation_tokens’
API Log
‘put’ AND ‘”key”:”admin”,”value”:”true”‘
API Log
‘post’ AND ‘profile/keys’
‘post’ AND ‘personal_access_tokens’
‘post’ AND ‘user/keys’
Table of search queries for various attack types – GitLab Enterprise
Audit Log
‘new.permission’ AND ‘admin’
Audit Log
‘post’ AND ‘ssh/account/keys/add’
‘personal access token created’
‘user added ssh access key’
Table of search queries for various attack types – Bitbucket Server
Additionally, the below items should be considered when configuring these SCM systems:
Source code management systems contain some of the most sensitive information in organizations and are a key component in the DevOps lifecycle. Depending on the role of an organization, compromise of these systems can lead to the compromise of other organizations. These systems are a high value to an attacker, and need more visibility from the information security community, as they are currently an afterthought compared to other systems such as Active Directory. It is X-Force Red’s goal that this research will bring more attention and inspire future research on defending these critical enterprise systems.
A special thank you to the below people for giving feedback on this research and providing whitepaper content review.
Brett has been in Information Security for several years working for multiple Fortune 500 companies across different industries. He has focused on both offen…
3 min read – In October 2022, the world marked the 19th Cybersecurity Awareness Month. October might be over, but employers can still talk about awareness of digital threats. We all have another chance before then: National Computer Security Day. The History of National…
3 min read – The protection of the SAP systems, as mission-critical applications, is becoming the priority for the most relevant organizations all over the world. The security hardening of SAP systems is key in these uncertain times, where threat actors start seeing SAP…
In September 2022, Microsoft patched an information disclosure vulnerability in SPNEGO NEGOEX (CVE-2022-37958). On December 13, Microsoft reclassified the vulnerability as “Critical” severity after IBM Security X-Force Red Security Researcher Valentina Palmiotti discovered the vulnerability could allow attackers to remotely execute code. The vulnerability is in the SPNEGO Extended Negotiation (NEGOEX) Security Mechanism, which allows a client and server to negotiate the choice of security mechanism to use. This vulnerability is a pre-authentication remote code execution vulnerability impacting a wide…
Applications have historically been deployed and created in a manner reminiscent of classic shopping malls. First, a developer builds the mall, then creates the various stores inside. The stores conform to the dimensions of the mall and operate within its floor plan. In older approaches to application development, a developer would have a targeted system or set of systems for which they intend to create an application. This targeted system would be the mall. Then, when building the application, they would…
Command & Control (C2) frameworks are a very sensitive component of Red Team operations. Often, a Red Team will be in a highly privileged position on a target’s network, and a compromise of the C2 framework could lead to a compromise of both the red team operator’s system and control over beacons established on a target’s systems. As such, vulnerabilities in C2 frameworks are high priority targets for threat actors and Counterintelligence (CI) operations. On September 20, 2022, HelpSystems published…
From 2020 to 2021, there was a 33% increase in the number of reported incidents caused by vulnerability exploitation, according to the 2022 X-Force Threat Intelligence Index. A large percentage of these exploited vulnerabilities were newly discovered; in fact, four out of the top five vulnerabilities in 2021 were newer vulnerabilities. Vulnerability exploitation was the second most common initial infection vector observed by IBM Security X-Force in 2021, falling closely behind phishing. Cybercriminals are finding new ways of bypassing security…
Analysis and insights from hundreds of the brightest minds in the cybersecurity industry to help you prove compliance, grow business and stop threats.
source
—
Note that any programming tips and code writing requires some knowledge of computer programming. Please, be careful if you do not know what you are doing…