Emergency Directive 21-01

See updated supplemental guidance for the latest.

December 13, 2020

Mitigate SolarWinds Orion Code Compromise

This page contains a web-friendly version of the Cybersecurity and Infrastructure Security Agency’s Emergency Directive 21-01, “Mitigate SolarWinds Orion Code Compromise”.

Section 3553(h) of title 44, U.S. Code, authorizes the Secretary of Homeland Security, in response to a known or reasonably suspected information security threat, vulnerability, or incident that represents a substantial threat to the information security of an agency, to “issue an emergency directive to the head of an agency to take any lawful action with respect to the operation of the information system, including such systems used or operated by another entity on behalf of an agency, that collects, processes, stores, transmits, disseminates, or otherwise maintains agency information, for the purpose of protecting the information system from, or mitigating, an information security threat.” 44 U.S.C. § 3553(h)(1)–(2)

Section 2205(3) of the Homeland Security Act of 2002, as amended, delegates this authority to the Director of the Cybersecurity and Infrastructure Security Agency. 6 U.S.C. § 655(3).

Federal agencies are required to comply with these directives. 44 U.S.C. § 3554 (a)(1)(B)(v)

These directives do not apply to statutorily-defined “national security systems” nor to systems operated by the Department of Defense or the Intelligence Community. 44 U.S.C. § 3553(d), (e)(2), (e)(3), (h)(1)(B).


Background

SolarWinds Orion products (affected versions are 2019.4 through 2020.2.1 HF1) are currently being exploited by malicious actors. This tactic permits an attacker to gain access to network traffic management systems. Disconnecting affected devices, as described below in Required Action 2, is the only known mitigation measure currently available.

CISA has determined that this exploitation of SolarWinds products poses an unacceptable risk to Federal Civilian Executive Branch agencies and requires emergency action. This determination is based on:

CISA understands that the vendor is working to provide updated software patches. However, agencies must wait until CISA provides further guidance before using any forthcoming patches to reinstall the SolarWinds Orion software in their enterprise.

Please refer to the MITRE ATT&CK framework for possible tactics the threat actors are using to maintain persistence in the environment.

Required Actions

This emergency directive requires the following actions:

  1. Agencies that have the expertise to take the following actions immediately must do so before proceeding to Action 2. Agencies without this capability shall proceed to Action 2.

    a. Forensically image system memory and/or host operating systems hosting all instances of SolarWinds Orion versions 2019.4 through 2020.2.1 HF1]. Analyze for new user or service accounts, privileged or otherwise.

    b. Analyze stored network traffic for indications of compromise, including new external DNS domains to which a small number of agency hosts (e.g., SolarWinds systems) have had connections.

  1. Affected agencies shall immediately disconnect or power down SolarWinds Orion products, versions 2019.4 through 2020.2.1 HF1, from their network. Until such time as CISA directs affected entities to rebuild the Windows operating system and reinstall the SolarWinds software package, agencies are prohibited from (re)joining the Windows host OS to the enterprise domain. Affected entities should expect further communications from CISA and await guidance before rebuilding from trusted sources utilizing the latest version of the product available. Additionally:

    a. Block all traffic to and from hosts, external to the enterprise, where any version of SolarWinds Orion software has been installed.

    b. Identify and remove all threat actor-controlled accounts and identified persistence mechanisms.

  1. By 12pm Eastern Standard Time on Monday December 14, 2020 agencies shall report as an incident to CISA (at https://us-cert.cisa.gov/report) the existence of any of the following:

    a. [SolarWinds.Orion.Core.BusinessLayer.dll] with a file hash of [b91ce2fa41029f6955bff20079468448]

    b. [C:\WINDOWS\SysWOW64\netsetupsvc.dll]

    c. Other indicators related to this issue to be shared by CISA

  1. After (and only after) all threat actor-controlled accounts and identified persistence mechanisms have been removed:

    a. Treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed.

    b. Rebuild hosts monitored by the SolarWinds Orion monitoring software using trusted sources.

    c. Reset all credentials used by or stored in SolarWinds software. Such credentials should be considered compromised.

    d. Take actions to remediate kerberoasting, including, as necessary or appropriate, engaging with a 3rd party with experience eradicating APTs from enterprise networks. For Windows environments, refer to the following:

  2. By 12pm Eastern Standard Time on Monday December 14, 2020, submit a report to CISA using the provided template. Department-level Chief Information Officers (CIOs) or equivalents must submit completion reports attesting to CISA that the affected devices were either disconnected or powered down.

These requirements apply to any agency network utilizing the SolarWinds Orion product. This includes any information system used or operated by another entity on behalf of an agency, that collects, processes, stores, transmits, disseminates, or otherwise maintains agency information.

CISA Actions

Duration

This emergency directive remains in effect until all agencies have applied the forthcoming patch or the directive is terminated through other appropriate action.

Additional Information

Frequently Asked Questions

Answers to common questions appear below.

What does the directive mean by “expertise”?

By “expertise”, we mean that you have staff or supporting personnel that are properly trained in taking a forensic image of system memory and have tooling readily-available to immediately do so.

What does the supplemental guidance mean by “disconnected”?

By “disconnected” we mean disconnected from the network and powered on if the agency has the capability- or is seeking a capable service provider- to collect forensics images (system memory, host storage, network) off of the host or virtual machine, or disconnected from the network and powered off if there is no such capability.


Supplemental Guidance v3

January 6, 2021

This guidance supersedes the Emergency Directive (ED) 21-01 Supplemental Guidance v1 issued on December 18, 2020 and ED 21-01 Supplemental Guidance v2 issued on December 30, 2020. This version also supersedes Required Action 4 of ED 21-01. All other provisions of ED 21-01 remain in effect.

For reference, see older Emergency Directive 21-01 supplemental guidance.

Summary of Required Actions

This supplemental guidance v3 requires (1) agencies that ran affected versions conduct forensic analysis, (2) agencies that accept the risk of running SolarWinds Orion comply with certain hardening requirements, and (3) reporting by agency from department-level Chief Information Officers (CIOs) by Tuesday, January 19, and Monday, January 25, 2021.

Table Summarizing Conditions for Operating SolarWinds Orion

SolarWinds Orion Platform Version Continued use of SolarWinds Orion permitted Hardening required Rebuild or upgrade If rebuilding or continuing use of SolarWinds, configurations can be restored from backups
Affected versions: 2019.4 HF5, 2020.2 RC1, 2020.2 RC2, 2020.2, 2020.2 HF1 Yes, if network falls into Category 1 or 21. If network is in Category 3, consult CISA before continuing use Yes (Except SAML-based authentication on MS AD FS) Full rebuild2 of SolarWinds Orion infrastructure and reset of all accounts that are currently—or have been—used by the system Yes, but only following the SolarWinds restore guidance 3
Unaffected versions: All other versions Yes Yes–may require rebuild or reinstallation of SolarWinds components Upgrade to latest version of SolarWinds Orion (at least version 2020.2.1HF2) and host OS (Windows Server 2016 or later) Yes

Background

This document provides supplemental guidance v3 on the implementation of CISA Emergency Directive (ED) 21-01, to include an update on affected versions; guidance for ensuring all federal agencies operating unaffected platforms are using at least SolarWinds Orion platform version 2020.2.1HF2; guidance for agencies using third-party service providers; and additional clarity on required actions. CISA provides this guidance as the minimum required guidance for Federal Executive Branch Agencies subject to CISA’s emergency directive authority.

This v3 supplemental guidance, which supersedes both v1 and v2 of the supplemental guidance and Required Action 4 of ED 21-01, is provided pursuant to ED 21-01.4 All other provisions specified in ED 21-01 remain in effect.

ED 21-01 directed agencies to immediately disconnect or power down certain SolarWinds Orion platform versions from their network. Based on developing information, on December 18, 2020, CISA provided supplemental guidance listing a subset of versions that have been identified as containing a malicious backdoor AKA TEARDROP or SUNBURST (“affected versions”). All other versions of the SolarWinds Orion platforms, regardless of whether included in the original range identified in ED 21-01, have been identified as not containing that malicious backdoor (“unaffected versions”).

Affected Versions

The following versions of SolarWinds Orion software are considered affected versions:

Network Categorization

As it pertains to activity related to ED 21-01, federal networks fall into one of three categories, as defined in Activity Alert AA20-352A and briefly restated here for convenience.

For the purposes of ED 21-01 and associated supplemental guidance, a network is defined as any computer network with hosts that share either a logical trust or any account credentials with SolarWinds Orion.

Networks with Affected Versions (Category 2 and 3)

Agencies that were using affected versions at any time prior to the issuance of ED 21-017 must comply with the following requirements:

  1. For Category 3 networks, agencies shall keep hosts that ran affected versions disconnected,8 as required by ED 21-01, and not rebuild or reimage the affected platforms and host operating systems (OS) pending consultation with CISA. This direction to keep such hosts disconnected also prohibits (re)joining the host OS to the enterprise domain. CISA’s approval of the implementation of SolarWinds Orion may be conditioned on agencies observing safeguards that are specifically tailored to the unique architecture/threat profile of the agency’s network. CISA will provide additional mitigation instructions to agencies in this category.

  2. For Category 2 and 3 networks, take appropriate actions (e.g., labeling and isolating, and retaining as appropriate in accordance with applicable record retention requirements/with other cyber investigative records) with backups of affected versions to prevent accidental re-introduction of malicious code to the production environment.

  3. For Category 2 and 3 networks, conduct forensic analysis as outlined in Appendix A - Required Forensics Investigation Actions.

  4. For Category 2 networks, provide an update on the incident to CISA before returning affected versions of SolarWinds Orion to service. Rebuilding SolarWinds Orion shall occur only after a thorough forensic investigation has been conducted and the agency has confirmed that there is no observed adversary activity or secondary actions on objectives (AOOs),9 confirming Category 2 activity only. The update to CISA shall list the mechanisms used to validate the absence of activity beyond Category 2 and recommend that the ticket be marked for closure. When resuming use of SolarWinds Orion in the environment after meeting these requirements, follow “Conditions for Operating SolarWinds Orion,” below (including Appendix B).

Conditions for Operating SolarWinds Orion

All agencies that accept the risk of running SolarWinds Orion in their enterprises (regardless of whether they were required to disconnect their instance(s) pursuant to ED 21-01 and regardless of “Category”) must run at least version 2020.2.1 HF2 and meet additional conditions outlined in Appendix B - Specific Conditions for Operating SolarWinds Orion.10 The National Security Agency (NSA) has examined this version and verified that it eliminates the previously identified malicious code. This version also includes updates to fix vulnerabilities unrelated to this malicious code, including vulnerabilities that SolarWinds has publicly disclosed.

Operating even version 2020.2.1 HF2 of the SolarWinds Orion platform may still carry some risk. The adversary enjoyed longstanding, covert access to the build process that SolarWinds uses for Orion, including to the code underlying the Orion platform. While the immediate known consequence of this access was the insertion of the malicious code into the affected versions of SolarWinds Orion, there may be other unknown consequences as well. The adversary can be presumed to be familiar with at least some aspects of the SolarWinds development and coding practices, as well as the SolarWinds Orion code itself (CISA is unable to assess the level of access the adversary may have had to other SolarWinds [non-Orion] code). Consequently, it is likely that the adversary is in a strong position to identify any potential (and as yet unknown) vulnerabilities in the SolarWinds Orion code that are unrelated to the inserted malicious code and may therefore survive its removal. This adversary has demonstrated the capability and willingness to exploit SolarWinds Orion to compromise U.S. government agencies, critical infrastructure entities, and private organizations. Agencies considering the use of the SolarWinds Orion platform should balance these risks against benefits of using these products to support agency network visibility.

As noted in CISA’s Activity Alert AA20-352A, CISA has evidence that the threat actor that inserted the SolarWinds backdoor also utilized initial access vectors that are unrelated to the SolarWinds Orion platform. Agencies should therefore also hunt for the tactics, techniques, and procedures (TTPs) as well as indicators of compromise (IOCs) related to this activity published in Activity Alert AA20-352A. Agencies should also consult any additional guidance related to this activity published by CISA or provided by the information security community. After completing the requirements of ED 21-01 and this supplemental guidance, agencies should focus on identifying potential account access abuse as well as identity impersonation as outlined in Activity Alert AA20-352.

Reporting Agency Status

For each agency, department-level Chief Information Officers (CIOs) or equivalents shall submit two additional status reports to CISA using the provided template by Tuesday, January 19, and Monday, January 25, 2021. Given the threat actor’s interest in compromising identity, CISA is requiring agencies to provide additional details in order to map the possible threat space that was impacted as part of the compromise.

Federal Information Systems Hosted in Third-Party Environments (such as Cloud)

CISA is working closely with FedRAMP to coordinate the response to ED 21-01 with FedRAMP Authorized cloud service providers (CSPs). FedRAMP Authorized CSPs have been informed to coordinate with their agency customers. CISA is also aware of third parties providing services for federal information systems subject to ED 21-01 that may not be covered by a FedRAMP authorization.

Each agency is responsible for inventorying all their information systems hosted in third-party environments (FedRAMP Authorized or otherwise) and contacting service providers directly for status pertaining to, and to ensure compliance with, ED 21-01. If instances of affected versions have been found in a third-party environment, reporting obligations will vary based on whether the provider is another federal agency or a commercial provider.

Incident reports must identify:

a) Category, per Mitigations section of CISA Activity Alert AA20-352A;

b) Name of affected third-party service (FedRAMP Authorized or otherwise);

c) Name(s) of affected FISMA information systems; and

d) Additional details on what data was exposed to the third-party service provider.

All other provisions specified in ED 21-01 remain in effect.

CISA Actions

Additional Information


Appendix A – Required Forensics Investigation Actions

Agencies that ran affected versions of SolarWinds Orion platform (Category 2 and Category 3) at any time prior to the issuance of ED 21-01 must conduct system memory, host storage, network, and cloud forensic analysis and hunt for indicators of compromise (IOCs) or other evidence of threat actor activity, such as secondary actions on objectives (AOO)11 as outlined in AA20-352A, such as user impersonation, privilege escalation, and data exfiltration.

  1. Agencies running affected versions that have no capability to conduct forensic analysis (system memory, host storage, network, and cloud) shall, at minimum, hunt for IOCs or other evidence of threat actor activity published in ED 21-01, Activity Alert AA20-352A, and future associated guidance. Agencies that, through hunting and/or forensic analysis, find these IOCs or evidence of threat actor activity, such as secondary AOO, shall assume breach and must report it as an incident to CISA through https://us-cert.cisa.gov/report. If a reporting agency already submitted incident information to CISA, please send updates to CISA as you discover new evidence.

  2. Agencies running affected versions that have no capability to conduct forensic analysis and no capability to hunt for IOCs shall assume breach, report the incident to CISA through https://us-cert.cisa.gov/report, and contact Central@cisa.dhs.gov to coordinate finding a qualified service provider capable of conducting forensics. Agencies whose forensics service provider’s analysis finds IOCs or evidence of threat actor activity, such as secondary AOO, shall update the incident report to CISA through https://us-cert.cisa.gov/report.


Appendix B - Specific Conditions for Operating SolarWinds Orion

Agencies that decide to run SolarWinds Orion platform may continue or resume doing so only if each of the following conditions are met:

  1. The agency assesses the risk of operating the SolarWinds Orion platform in agency production environments, and the agency accepts the residual risk.

  2. All incoming and outgoing communications outside of the agency device network management enclave are denied, with additional assurance that communications to the public internet to and from hosts running SolarWinds Orion products has been blocked (as required by ED 21-01).

  3. Cloud instances of Orion can only monitor cloud resources in that cloud infrastructure.

  4. On premises instances of Orion must not be permissioned with any cloud/hosted identity accounts.

  5. In cases where a rebuild is required, restoration of SolarWinds configuration may be done from the legacy database by following the SolarWinds restore guidance. Restoration for affected versions will differ from restoration for unaffected versions—agencies must ensure that they are following the correct restoration guidance.

  6. Networks using affected versions of SolarWinds Orion, where permitted to rebuild, must take the following additional steps before rebuilding their SolarWinds Orion platform:

    a) Change all account credentials, or other shared secrets (such as SNMP strings) that are—or had been—utilized by the affected SolarWinds Orion device being rebuilt.

    i. Enable multi-factor authentication (MFA) for these credentials whenever possible;

    ii. Provide service accounts with the minimum level of privilege necessary for the role performed whenever possible; and

    iii. For accounts where MFA is not possible, require use of randomly generated long and complex passwords (greater than 25 characters) and implement a maximum 90-day rotation policy for these passwords.

    b) Remove all inbound trust relationships to the SolarWinds Orion device being rebuilt.

  7. The agency updates the SolarWinds Orion platform to at least version 2020.2.1 HF2 and install and update the host to the latest supported build of Windows Server 2019 (preferred) or Windows Server 2016, hardened to agency standards.

  8. The SolarWinds Orion server, the web server, and the database server instances must be installed on separate and dedicated hosts.

  9. Agencies must follow the SolarWinds secure configuration (hardening) guidelines provided by the vendor, which can be found at: https://documentation.solarwinds.com/en/Success_Center/orionplatform/content/core-secure-configuration.htm, EXCEPT agencies shall not configure the SolarWinds software to implement SAML-based authentication that relies on Microsoft’s Active Directory Federated Services. This configuration is currently being exploited by the threat actor associated with this activity.

  10. The agency configures logging to ensure that all logs from the host OS, SolarWinds platform, and associated network logs are being captured and stored for at least 180 days in a separate, centralized log aggregation capability.

  11. The agency ensures that the SolarWinds logs are being actively monitored by the agency SOC.

  12. The agency implements subsequent SolarWinds Orion platform updates and security advisories within 48 hours of release.


Footnotes for Supplemental Guidance

  1. For category definitions see Mitigations Section of https://us-cert.cisa.gov/ncas/alerts/aa20-352a. 

  2. A new OS (the latest supported build, at least Windows 2016) and fresh installation of Orion (at least version 2020.2.1 HF2) from trusted media must be used for all affected versions. 

  3. https://solarwinds.com/upgrading-your-environment/ 

  4. On December 13, 2020, CISA issued ED 21-01 to mitigate the SolarWinds Orion code compromise. As noted in ED 21-01, CISA continues to work with our partners to monitor for active exploitation associated with this compromise and will continue providing updated guidance to agencies as new information becomes available. ED 21-01 also indicated that agencies must wait until CISA provides further guidance before using any forthcoming patches to reinstall the SolarWinds Orion software in their enterprise. CISA subsequently issued supplemental guidance v1 and v2, which, respectively, updated ED 21-01 to provide guidance on affected versions and clarify requirements for agencies using third-party service providers, and directed agencies continuing to use unaffected versions of SolarWinds Orion to update to version 2020.2.1 HF2 by December 31, 2020. 

  5. V1 of this guidance included a single bullet that listed two platform versions for the same DLL. For clarity, v3 lists these platform versions that share the same DLL version number separately, as both are considered affected versions. 

  6. See Appendix A for additional information 

  7. This includes instances that may have been rolled back, rebuilt, or reimaged to unaffected version but that, at one time prior to the issuance of ED 21-01, used an affected version. 

  8. By “disconnected” we mean disconnected from the network and powered on if the agency has the capability—or is seeking a capable service provider—to collect forensics images (system memory, host storage, network) off of the host or virtual machine, or disconnected from the network and powered off if there is no such capability. 

  9. https://www.lockheedmartin.com/content/dam/lockheed-martin/rms/documents/cyber/Seven_Ways_to_Apply_the_Cyber_Kill_Chain_with_a_Threat_Intelligence_Platform.pdf 

  10. Per v2 of the guidance, agencies continuing to operate unaffected versions of SolarWinds Orion as of December 30, 2020 were required to update to version 2020.2.1HF2 by December 31, 2020. All agencies resuming use of SolarWinds Orion after this date are required to update before reintroducing to the environment. 

  11. https://www.lockheedmartin.com/content/dam/lockheed-martin/rms/documents/cyber/Seven_Ways_to_Apply_the_Cyber_Kill_Chain_with_a_Threat_Intelligence_Platform.pdf