SolarWinds, a network-management software maker, has had one of the worst hacker breaches in U.S. history. The attack was sophisticated, broad in scope, and marred the trust placed in tech providers. SolarWinds’ new chief executive is still trying to determine how his company became the hacker’s main avenue of attack. From SolarWinds, the hackers managed to penetrate federal government networks.

The attack is the subject of an ongoing investigation, and investigators say there is concrete evidence that the suspected Russian espionage, lasting at least nine months, extended beyond SolarWinds. Hackers apparently broke into the systems by taking advantage of known software bugs, guessing online passwords, and making use of the way Microsoft’s cloud software is configured.

Sudhakar Ramakrishna, SolarWinds’ Chief Executive Officer, said, “Some email accounts were compromised. That led them to compromise other email accounts and as a result our broader [Office] 365 environment was compromised.” Evidence is mounting that the company’s Office 365 email system was host to the hackers for months.

SolarWinds and investigators are still trying to determine how the hackers got in and when that happened. One of the theories being investigated is that hackers may have gotten into the company’s Office 365 accounts even earlier than December 2019, as is currently thought. Then the hackers would have used that initial point of entry to gain access to other accounts in the company.

The investigation has continued for two months so far. Investigators are trying to ascertain how much damage has been done and how far the hackers’ reach has been. Only several dozen victims have been identified, but the attack could have ultimately affected nearly 18,000 SolarWinds customers. Lawmakers have labeled this hack as a national security emergency. Among those customers affected were huge tech companies like Microsoft and Cisco Systems, as well as the departments of State, Treasury, Justice, Energy, Labor and Energy, Commerce, and Homeland Security.

People familiar with the investigation said that a China-linked group of hackers exploited an unrelated and less serious flaw in SolarWinds software to target the Agriculture Department prior to this attack.

The internal investigation has required searching through thousands and thousands of log files and other data to retrace the hacking operation. Ramakrishna said the hacking was undetected for more than a year.

Hackers turned SolarWinds’ software update into a delivery system for their hacks. Hackers ran tests on SolarWinds’ internal build systems used to assemble the company’s software updates. The build systems were then used to create a malware patch that was shipped to 18,000 customers.

SolarWinds will spend millions of dollars in response to the incident. Ramakrishna, who took over on January 4, 2021, said, “My attitude was to come in and assess first and figure out what we needed to do.” He has revamped SolarWinds’ software development processes and brought in outside cybersecurity experts.

The U.S. government has blamed Russia, which denied responsibility. Biden instructed Avril Haines, director of national intelligence, to conduct a review of Russian aggression against the U.S., and the SolarWinds hack is to be included in that investigation.

SolarWinds’ management software Orion was a major avenue of the hacking effort. However, the acting director of the Cybersecurity and Infrastructure Security Agency said last week that about 30% of the hackers' victims had no direct connection with SolarWinds.

Adam Meyers of CrowdStrike Holdings, Inc, a security company SolarWinds hired to investigate the hack, said, “This is a pretty significant incident. Frankly I don’t even know that we’ve scratched the surface on this thing.”