Public Cloud Security Breaches Documenting their mistakes so you don't make them.

Ubiquiti (2020)

Date Created: 2023-04-13 Last Updated: 2023-05-27 Post Author:   Mark Gaddy
 AWS | Insider Threat | Data Exfiltration | 

Incident Details

Victimized Company:Ubiquiti
Incident Dates:2020-12-10 to 2020-12-26
Disclosure Date:2021-01-11
Current Status:Perpetrator Sentenced

In December of 2020, Ubiquiti suffered a breach at the hands of an employee. This employee masked his presence via a VPN and was able to clone the company’s GitHub repository and alter logs in AWS to hide their presence and evidence of the breach. After the attacker leaked false details of the attack to a well-known security blogger, Ubiquiti’s stock lost 4 billion dollars in value.

Incident

Details of the Incident

Beginning in December of 2020, a senior cloud engineer at Ubiquiti used his employee credentials to exfiltrate source code and customer information. Before starting the data exfiltration, the employee validated the credentials from his home network. Then shortly after successfully testing the credentials, he turned on a VPN to mask his IP address, logged into the company’s Github account, and began cloning repositories to his local machine.

While cloning the repositories with sensitive information, the attacker’s internet connection and VPN dropped. As a result, for around 30 minutes, the employee was cloning the repositories using his residential IP.

After cloning most of the repositories from GitHub, the employee went into the company’s AWS account, again under the mask of his VPN, and altered the lifecycle retention policy on the S3 Bucket holding the logs relating to his attack. The employee configured the lifecycle policy to delete after one day.

Within the following days, the other Ubiquiti employees realized an incident had occurred and began responding. The employee joined the team handling the incident to monitor Ubiquiti’s response.

In January, senior Ubiquiti employees received a ransom email demanding 25 BTC not to release the customer information. After the company did not respond, the information was leaked to the public. Rather than paying the ransom, the company changed all employee passwords and released a security update to all customers urging them to reset their passwords and enable multi-factor authentication on their accounts.

In March, a whistleblower reached out to reporter Brian Krebs, released details of the previously unknown incident to the public, and claimed Ubiquiti was downplaying the incident.

Shortly after, the FBI identified the employee as Nickolas Sharp and searched his residence. In December 2021, Nickolas Sharp was charged with four counts relating to the incident. Sharp pleaded guilty in February 2023 and is scheduled to be sentenced in May 2023.

As documented in the Government’s sentencing report (pg 13):

Sharp stole dozens of so-called secrets files from Company-1’s AWS Secrets Manager, a cloud password management tool. Sharp also stole over 1,400 AWS task definitions files, and over 1,100 GitHub code repositories where Company-1 stored its development files and proprietary code.

Timeline

Date Event
December 10th, 2020 An attacker logs into Ubiquiti’s AWS infrastructure. The attacker ran the “GetCallerIdentity” command to return the username and account information for the AWS account.
December 21st, 2020 An attacker masked by a Surfshark VPN begins cloning Ubiquiti’s GitHub Repositories.
December 22nd, 2020 The Internet at the Sharp residence goes down for 30 minutes. Cloning commands continue when it comes back up, but VPN is turned off temporarily.
December 22-26th, 2020 “Sharp manually renamed at least 18 AWS sessions that he opened to conduct the attack to make it appear as if one of five other Company-1 DevOps team members was using the [access] Key.”1
December 26th, 2020 Nickolas Sharp altered the lifecycle retention policy on the S3 Bucket storing the CloudTrail events to erase logs of their attack after a day.
December 28th, 2020 Other employees detected the incident and began investigating.
January 7th, 2021 Senior employees received a ransom email from the attacker.
January 9th, 2021 The attacker sent another email after not receiving the ransom, which contained a link to a public Keybase, which contained proprietary data of Ubiquiti.
January 11th, 2021 Ubiquiti emails customers stating to change passwords and enable multi-factor authentication, disclosing a breach had occurred.
March 21st-28th, 2021 An anonymous whistleblower, who turned out to be Nickolas Sharp, reaches out to Brian Krebs, outlining details of the attack and stating Ubiquiti was downplaying the breach.
March 24th, 2021 Federal Bureau of Investigation executes a search warrant on the Sharp residence.
March 31st, 2021 Ubiquiti releases a statement regarding the attack.
December 1st, 2021 Nickolas Sharp is indicted.
February 2nd, 2023 Nickolas Sharp enters a Guilty plea in the case of this attack.
May 10th, 2023 Sharp sentenced to 6 years in prison

Attribution / Perpetrator

Per the Department of Justice, Nickolas Sharp “pled guilty to intentionally damaging a protected computer, wire fraud, and making false statements to the Federal Bureau of Investigation.” The Department of Justice Identified Nickolas Sharp as the attacker when his VPN temporarily disconnected and revealed his home IP Address.

Long-term impact:

After Brian Krebs released his article, Ubiquiti’s stock price plummeted over 20%, resulting in a $4 billion loss to shareholders. Ubiquiti filed a defamation lawsuit against Brian Krebs for his article. The company later dropped the lawsuit after Krebs agreed to take down his original post about the incident. In his retraction, Krebs acknowledged that the sole basis of his reporting was the person federal prosecutors have since indicted.

Summary of Coverage

Cloud Security lessons learned.

As a “senior software engineer … responsible for software development and cloud infrastructure security”, Nickolas Sharp needed admin level permissions to perform his job functions. However the ancient adage Quis custodiet ipsos custodes still applies. While cloud engineers should have admin level access to AWS, the following practices still should be applied:

  • Monitoring of privileged activity
  • Ensuring that logging data is written to an AWS account that is not used for other activities
  • Monitoring & questioning any activities in AWS accounts that service a security, audit, or compliance purpose
  • Enable S3 Versioning on buckets that contain security sensitive log data
  • Enable MFA Delete or two-person control via SCPs

As Sharp already had privileged access, it is unclear what role the cloning of GitHub repositories played in this data breach. However, the following GitHub practices might have made the insider activity more clear to the company:

  • Maintaining audit data of cloned repos
  • Using a secrets scanning service like GitGuardian or GitHub Advanced Security
  • Ensuring that every member of the GitHub organization does not have the ability to clone all the repos.

–

  1. GOVERNMENT’S SENTENCING MEMORANDUM REGARDING DEFENDANT NICKOLAS SHARP (pg 7) ↩︎