Trouble Keeping Track of Your Keys? So Does Toyota: Lessons Learned from a Key Management Breach

Introduction: What happened in the Toyota key management breach? 

1. A hardcoded secret to the data server for Toyota’s connectivity app has been exposed publicly on GitHub for almost 5 years. This server stored customer data, including PII.
2. The visibility of the source code repository was inadvertently changed to public in December 2017.
3. Toyota blamed a development subcontractor for the error of pushing the hardcoded secret into the source code.

Quick win #1: Scan for hardcoded secrets

Many commercial and open-source products provide secret scanning capabilities. Each product has pros and cons, which I am not going to discuss in this blog post. But the quickest win would be either using a free unlimited secrets scanner (shameless promotion for Arnica) or run an open-source tool that iterates through all repositories and identifies hardcoded secrets. For example, you can install the free version of Semgrep and iterate through all repositories to find secrets and many other vulnerabilities, at least for the first iteration.  

Here are the steps to get there:

1. Generate a GitHub Personal Access Token and grant it the “repo” scope.

2. If your organization is protected by SAML, grant the access for this token into the organization.  
3. Run the following Python script.

4. Triage the results in the file and ask the devs to fix them.

Quick win #2: Get alerts when repositories become public

Setup a webhook to receive notifications when a repository’s visibility changes and notify the relevant stakeholders in their relevant channel. In this example, the webhook is automatically set within 5 minutes through Tines. Here are the steps:

1. Create an account in Tines. You can use 3 workflows for free.
2. Create a webhook step with its default settings. Copy the URL, as you will need it in the next step.

3. Go to the webhook configurations page (https://github.com/organizations/{YOUR_ORG}/settings/hooks) and create a new webhook.
4. Paste the URL from Tines, select the content type as “application/json” and change the events trigger to “let me select individual events.”

5. Select the checkbox of “visibility changes” and click Save.

6. Go back to Tines and add an email action. Set the action to send an email to a desired destination and include the repo name in the action, as in the screenshot below.

7. Try to change a dummy repository from private to public and check your inbox. Here is the result on my side.

Quick win #3: Ensure CODEOWNERS is enforced  

Code reviews are vital for code quality and security. GitHub enables this capability via the  CODEOWNERS file. However, it is easy to misconfigure the file since it requires both the configuration of the file and the enforcement of the branch protection policy to require a review from the CODEOWNERS. Here is an example script that can help you to determine if any of the CODEOWNERS files and policies are misconfigured.

Caveat: the assumption of this script is that CODEOWNERS is configured in the default branch. At Arnica, we developed a more comprehensive classification mechanism to determine which branches are important to protect.  

Would this solution prevent the secret from appearing in git history? No, but it will enforce another set of eyes on code changes to help identify secrets and other risky code changes prior to pushing them to a production environment.

Conclusion: Prioritize key management for secure access control and to protect against potential breaches

The actions listed above will provide quick value, but as illustrated by Toyota’s recent leak, properly securing your development ecosystem requires a comprehensive analysis of developer permissions, identification of abnormal and risky developer behavior, and regular scans for hardcoded secrets.