Live Blog on SolarWinds Cyber Attack: 'Grave Impact'

Welcome to the SecureWorld Live Blog about the SolarWinds cyber attack.

This is where our media team is posting an ongoing list of significant updates related to the attack.  Cybersecurity experts around the world are looking at this attack from all angles and that means a steady stream of new information. 

If you already know the basics of this attack, skip past the next couple of paragraphs and get right to the updates.

In 60 seconds: what happened in the SolarWinds cyber attack?

You're probably familiar with notices telling you to update to the latest version of something on your phone, your tablet or your laptop. Corporations and governments must do similar updates, on a massive scale, with the help of automation.

For 18,000 companies and governments around the world, some of these updates arrive from an IT management company called SolarWinds. And specifically, from its Orion product.

Nation-state backed hackers (think of them as digital soldiers) somehow got inside SolarWinds update process and secretly placed malware into a few months of updates.  They also managed to digitally 'sign' the updates, which made them look legitimate. 

The end result is that when Solar Winds sent out the software updates receiving networks saw the legitimate information but not the hidden malware. And organizations unknowingly accepted both. This is a supply-chain attack which acts like a Trojan-horse. 

The malware in these updates gave the nation-state backed attackers a foothold in computer networks around the globe, including within a number of U.S. Government agencies. Read our original story, here, for more details on how the attack worked or continue reading below for updates.

December 23 update: investigation notes from SolarWinds

SolarWinds continues to update its event FAQs and it says its latest research boosts confidence on the window of infection here. 

• Orion products downloaded, implemented, or updated during the relevant period [March - June 2020] contained the inserted vulnerability (SUNBURST);
• Orion products downloaded and implemented before the relevant period and not updated during the relevant period did not contain the vulnerability;
• Orion products downloaded and implemented after the relevant period did not contain the vulnerability; and
• Previously affected version of the Orion products that were updated with a build released after the relevant period no longer contained the vulnerability; however, the server on which the affected Orion products ran may be compromised during the period in which the vulnerability existed."

December 20 update: new statements about the cyber attack

Over the weekend, Secretary of State Mike Pompeo made his first detailed statement on the attack, attributing it to the Kremlin:

“This was a very significant effort, and I think it’s the case that now we can say pretty clearly that it was the Russians that engaged in this activity.”

Meanwhile, Kevin Mandia of FireEye was on CBS News over the weekend and asked if he agreed with the assessment from many that Russia is behind the supply chain attack. He said we will absolutely be able to attribute the attack with certainty, but not quite yet:

"As we continue the investigation, as it gets broader, as more people learn the TTPs of this attack, we're going to bring it back and we're going to get attribution. Not '92% right', not 'consistent with' but 100%. Let's just get it right so we can appropriately respond, period."

Attack victim update: The Wall Street Journal says it has identified 24 organizations that unknowingly downloaded the compromised software. Do you recognize any of these names?

• Cisco Systems
• Intel
• Nvidia
• Deloitte
• VMware 
• Belkin International (which sells routers under Belkin and LinkSys brands)
• California Department of State Hospitals
• Kent State University

Cisco, for its part, revealed it does not even use the particular product that nation-state attackers were able to compromise. That's an interesting development.

From SiliconANGLE:

“At this time, there is no known impact to Cisco offers or products,” Cisco said in a statement. “We continue to investigate all aspects of this evolving situation with the highest priority.”

Cisco added that although it doesn’t use SolarWinds Orion for its enterprise network management or monitoring, “we have identified and mitigated affected software in a small number of lab environments and a limited number of employee endpoints. We continue to investigate all aspects of this evolving situation with the highest priority.”

December 19 update on supply chain security attack

SolarWinds continues to answer more questions about the attack in an FAQ section on its website. Here is a key question the company addressed about the decision to move forward and install new and reportedly secure updates from the company

Question: If your environment was compromised, why is it safe for us to install these updates/trust your code? OR What are you doing to prevent future incidents moving forward?

Answer: 
"Our investigations are early and ongoing, but since the SUNBURST vulnerability was discovered, we have reviewed our environment, giving an initial focus on ensuring the security of our build environment, including our source code repositories. We have reviewed the architecture of the build environment, the privileged and non-privileged users that have access to the build environment, and the network surrounding the build environment.

We have retained third-party cybersecurity experts to assist in an investigation of these matters, including whether the SUNBURST vulnerability was exploited as a point of any infiltration of any customer systems, and in the development of appropriate mitigation and remediation plans to help ensure future releases are protected from SUNBURST or other vulnerabilities.

We are also cooperating and sharing information with our partners, vendors, the Federal Bureau of Investigation, the U.S. intelligence community, and other government agencies to assist in investigations related to this incident and to avoid vulnerabilities in future releases. Our investigation is ongoing, and we will continue to focus on ensuring that our environments are secure and protected."

And on the question of how widespread the SUNBURST malware infections are, Microsoft created a map based on detections from its Defender Antivirus software:

If you click and expand the map, you can see why Microsoft says the following:

"This identifies customers who use Defender and who installed versions of SolarWinds’ Orion software containing the attackers’ malware. As this makes clear, this aspect of the attack created a supply chain vulnerability of nearly global importance, reaching many major national capitals outside Russia. This also illustrates the heightened level of vulnerability in the United States."

December 18 update: SolarWinds not the only attack method

Can news of a major supply chain data breach become even more unsettling? It sure can and it just did.

A new US-CERT alert reveals what some sources had been saying all along:

 "The SolarWinds Orion supply chain compromise is not the only initial infection vector this APT actor leveraged."

What does the U.S. Government mean, specifically? CISA believes it is linked to the Outlook Web App and a stolen key:

"CISA is investigating incidents that exhibit adversary TTPs consistent with this activity, including some where victims either do not leverage SolarWinds Orion or where SolarWinds Orion was present but where there was no SolarWinds exploitation activity observed.

Volexity has also reported publicly that they observed the APT using a secret key that the APT previously stole in order to generate a cookie to bypass the Duo multi-factor authentication protecting access to Outlook Web App (OWA).

Volexity attributes this intrusion to the same activity as the SolarWinds Orion supply chain compromise, and the TTPs are consistent between the two. This observation indicates that there are other initial access vectors beyond SolarWinds Orion, and there may still be others that are not yet known."

Here is the US-CERT issued update.

Also, the U.S. Department of Energy revealed it was breached during the cyberattack:

"At this point, the investigation has found that the malware has been isolated to business networks only, and has not impacted the mission essential national security functions of the Department, including the National Nuclear Security Administration (NNSA).

When DOE identified vulnerable software, immediate action was taken to mitigate the risk, and all software identified as being vulnerable to this attack was disconnected from the DOE network.” said DOE SpokeswomanShaylyn Hynes.

Also new: was the SolarWinds supply chain breach in the works for more than a year? Security firm ReversingLabs analyzed things:

"...there was a previous version that was tampered with by the attackers: version 2019.4.5200.8890, from October 2019, and this version had only been slightly modified. While it doesn’t contain the malicious backdoor code, it does contain the .NET class that will host it in the future.

This first code modification was clearly just a proof of concept. Their three step action plan: Compromise the build system, inject their own code, and verify that their signed packages are going to appear on the client side as expected. Once these objectives were met, and the attackers proved to themselves that the supply chain could be compromised, they started planning the real attack payload."

December 17 update: Intel, Cisco among possible victims of attack

The U.S. Government announced the creation of a Cyber Unified Coordination Group (UCG) to coordinate a whole-of-government response to this significant cyber incident.

• The FBI is taking the lead on attribution and pursuit of the threat actors.
• The Office of the Director of National Intelligence (ODNI) is "helping to marshal all of the Intelligence Community’s relevant resources to support this effort and share information across the United States Government."
• And CISA is leading the private sector/public sector collaboration and issuing any directives.

Read the Cyber Command Coordination Group statement. 

Next, how about major tech players as potential victims of the SolarWinds supply chain attack? That's what some researchers are saying now:

"...Chinese cybersecurity firm RedDrip Team on Wednesday released on GitHub a decoder tool that can be used to crack a list of obscured hostnames—in other words, a partial list of victims—with which the C2 communicated.

The more than 1,700 hostnames listed include endpoints—sometimes more than one—inside such U.S. technology firms as Belkin, Cisco, Intel and Nvidia; government agencies and universities; as well as organizations such as the Hewlett Foundation," reported InfoRisk Today.

December 16 update

As you can imagine, lots of analysis and threat hunting is underway right now. Here is what Symantec found among its customer base:

"Symantec has identified more than 2,000 computers at over 100 customers that received Trojanized software updates. We have found a small number of organizations where a second stage payload (Backdoor.Teardrop) was used."

And December 16 is when Microsoft made some changes relating to "Solarigate" as it calls the incident:

"Customers should consider any device with the binary as compromised and should already be investigating devices with this alert. Starting on Wednesday, December 16 at 8:00 AM PST, Microsoft Defender Antivirus will begin blocking the known malicious SolarWinds binaries. This will quarantine the binary even if the process is running. We also realize this is a server product running in customer environments, so it may not be simple to remove the product from service."

Now to previous updates, below.

CISA emergency directive to unplug SolarWinds Orion products

The Cybersecurity and Infrastructure Security Agency (CISA) moved quickly to cut off government servers from SolarWinds updates.

At this point, all government agencies should have stopped receiving updates from the SolarWinds Orion product, and for those federal agencies without the expertise to stop these updates, CISA is ordering them to unplug:

"Affected agencies shall immediately disconnect or power down SolarWinds Orion products, versions 2019.4 through 2020.2.1 HF1, from their network. Until such time as CISA directs affected entities to rebuild the Windows operating system and reinstall the SolarWinds software package, agencies are prohibited from (re)joining the Windows host OS to the enterprise domain.

Affected entities should expect further communications from CISA and await guidance before rebuilding from trusted sources utilizing the latest version of the product available. Additionally:

a. Block all traffic to and from hosts, external to the enterprise, where any version of SolarWinds Orion software has been installed.

b. Identify and remove all threat actor-controlled accounts and identified persistence mechanisms."

CISA spells out 'grave impact' to U.S. government

CISA has determined that this exploitation of SolarWinds products poses an unacceptable risk to Federal Civilian Executive Branch agencies and requires emergency action. This determination is based on:

• Current exploitation of affected products and their widespread use to monitor traffic on major federal network systems;

• High potential for a compromise of agency information systems;

• Grave impact of a successful compromise.

Department of Homeland Security says 'multiple agencies' impacted

And we now have more acknowledgement from the U.S. Department of Homeland Security (DHS) about the serious nature of this attack. It sounds like the first two agencies we know were breached were just the beginning.

DHS Assistant Secretary for Public Affairs Alexei Woltornist commented briefly:

"The Department of Homeland Security is aware of cyber breaches across the federal government and working closely with our partners in the public and private sector on the federal response. As the federal lead for cyber breaches of civilian federal agencies, the Department of Homeland Security's Cybersecurity and Infrastructure Security Agency (CISA) has already issued Emergency Directive 21-01 to the federal government to address compromises related to SolarWinds."

SolarWinds updates on supply chain attack

SolarWinds is updating things from its side, too.

"Known affected products: Orion Platform versions 2019.4 HF 5 and 2020.2 with no hotfix or with 2020.2 HF 1, including:

Application Centric Monitor (ACM)
Database Performance Analyzer Integration Module (DPAIM)
Enterprise Operations Console (EOC)
High Availability (HA)
IP Address Manager (IPAM)
Log Analyzer (LA)
Network Automation Manager (NAM)
Network Configuration Manager (NCM)
Network Operations Manager (NOM)
Network Performance Monitor (NPM)
NetFlow Traffic Analyzer (NTA)
Server & Application Monitor (SAM)
Server Configuration Monitor (SCM)
Storage Resource Monitor (SRM)
User Device Tracker (UDT)
Virtualization Manager (VMAN)
VoIP & Network Quality Manager (VNQM)
Web Performance Monitor (WPM)

And the company completed more investigative work as of December 15:

"We have scanned the code of all our software products for markers similar to those used in the attack on our Orion Platform products identified above, and we have found no evidence that other versions of our Orion Platform products or our other products contain those markers. As such, we are not aware that other versions of Orion Platform products have been impacted by this security vulnerability. Other non-Orion Platform products are also not known by us to be impacted by this security vulnerability."

This story will be updated.