The Wiz Research Team recently discovered a supply chain vulnerability in IBM Cloud that they say is the first to impact a cloud provider’s infrastructure.
In a dramatic flair, they named the flaw Hell’s Keychain.
The security issues were reported to IBM Cloud in late August, and were patched in early September. Before it was patched, an attacker with knowledge of the vulnerability could run malicious code and modify data being stored by any IBM Cloud customer using PostgreSQL.
A Recipe for Access: Forbidden Link and Keychain Secrets
The Wiz researchers – Ronen Shustin, Shir Tamari, Nir Ohfeld and Sagi Tzadik – wrote in a blog post, “In our experience, the recipe for a cloud service provider (CSP) supply-chain attack features two ingredients: the forbidden link and the keychain. The forbidden link represents network access – specifically, it is the link between a production environment and its build environment.”
The keychain, they said, “symbolizes the collection of one or more scattered secrets the attacker finds throughout the target environment. Although both components are individually unhygienic, they form a fatal compound when combined.”
In the case of Hell’s Keychain, the three keychain secrets were a Kubernetes service account token, a private container registry password, and CI/CD server credentials. The forbidden link connected a personal PostgreSQL instance to the IBM Cloud Databases build environment.
Also read:
- Software Supply Chain Security Guidance for Developers
- How Hackers Compromise the Software Supply Chain
SQL Injection and Container Registry Scraping
The researchers first uncovered a SQL injection vulnerability that allowed them to execute arbitrary commands on the underlying virtual machine hosting their database instance, which they used to map the internal environment and look for new attack surfaces.
Their actions triggered an alert from IBM Cloud’s security team, which eventually gave them permission to continue their research.
During their exploration of the environment, they found a Kubernetes API token, which they used to access the Kubernetes API and view a list of other pods running PostgreSQL instances. They then used container registry scraping to find four credentials that could be used to access several container registries.
“Our query of IBM Cloud’s IAM API revealed that this was an API key capable of accessing IBM Cloud’s Container Registry Images that appeared to have read-write authorization! We then used the ibmcloud-cli to log in to the specific container registry with this key,” they wrote.
While it turned out that the key’s description was inaccurate – they weren’t able to write to the container registry – they still saw their findings as severe. “Had a malicious actor obtained these credentials, they could have pulled and explored hundreds of images belonging to IBM Cloud’s managed database services,” they wrote.
Also read: How to Prevent SQL Injection Attacks
Network Access to IBM Cloud Build Servers
The researchers scanned the container images they had access to and found several sensitive secrets in overlooked files, including FTP credentials and internal artifact repository credentials for IBM Cloud internal services.
They then examined the historical commands used to build the container’s image to learn which artifacts were involved. “When we tried to access these servers from the machine hosting the PostgreSQL instance, we were shocked to learn that we had network access to internal IBM Cloud build servers! We then proceeded to authenticate to them using the artifact repository credentials, and in doing so successfully uncovered the forbidden link,” they wrote.
Finally, they tested their permissions by creating files in the repositories used in the build process of the PostgreSQL image. “This proved that we could overwrite arbitrary files in the packages that would have been installed on every PostgreSQL instance, establishing the supply-chain attack path,” they wrote.
Lessons to Learn
The Wiz researchers said Hell’s Keychain “illustrates how scattered plaintext credentials across your environment can impose a huge risk on your organization by impairing its integrity and tenant isolation. Moreover, the vulnerability emphasizes the need for strict network controls and demonstrates how pod access to the Kubernetes API is a common misconfiguration that can result in unrestricted container registry exposure and scraping.”
According to the researchers, there are three key lessons to be learned from their findings:
- Constantly monitor your environment for scattered secrets
- Ensure your production environment has strict network controls
- Configure your container registry to prevent malicious actors from scraping it
“IBM Cloud rapidly investigated and fixed the vulnerabilities and security issues we discovered,” they wrote. “We enjoyed working with IBM Cloud’s security team which took the issues very seriously by addressing them promptly and professionally.”
Further reading:
