Folks,
This blog entry impacts 85% of all organizations worldwide.
It merits your immediate, highest, undivided attention.
It provides
rare, high-value insight into how perpetrators most likely stole sensitive PII of 21.5 M U.S. individuals from the OPM.
[ You're most likely not going to get such insight from the Microsofts, Intels, Ciscos, IBMs, HPs, Dells, Symantecs, McAfees, FireEyes, CrowdStrikes, CAs, Tripwires, ArcSights, Centrifys, BeyondTrusts, Booz Allen Hamiltons etc. etc. of the world. ]
Summary
This blog entry is almost 10,000 words long. If you only have 2 minutes,
here's the short of it -
- Privileged access is the new holy-grail for malicious perpetrators. 100% of all major recent cyber security breaches (Snowden, Target, JP Morgan, Sony, Anthem, OPM) involved the compromise of single privileged user account.
- In all likelihood, the OPM breach could have been prevented if OPM had only minimized and adequately protected the number of privileged users in its network i.e. the number of Active Directory administrators in its Active Directory.
- Here's what most likely happened at the OPM: "Perpetrators gained authenticated network access to OPM's network by compromising a non-privileged OPM Active Directory domain user account belonging to a KeyPoint contractor. They then used this authenticated non-privileged access to identify the list of all privileged users in OPM's Active Directory deployment, and subsequently engaged in Active Directory Privilege Escalation (using one of these two attack vectors) to compromise a single one of these privileged user accounts, then used it to obtain access to the SF-86 and SF-85 databases. They subsequently exfiltrated the data."
- Today, most organizations in the world, including most U.S. Federal agencies, most national governments, and most of the world's software, hardware, Internet, financial, manufacturing, retail, medical, transportation, education, media, energy, defense and cyber security companies, are all at risk of compromise and a proficient Advanced Persistent Threat (APT) could potentially compromise them within hours, if not within days, using this unmitigated attack vector.
- In the event that a government agency or a company is compromised, its stakeholders (shareholders, customers, employees, partners and others) will hold its leadership accountable. (The Director of the OPM had to resign.)
If this has your attention and piques your curiosity, and you want to get into the details and gain from our unique
insight and
threat intelligence, I'd recommend spending 10 -15 minutes to read this blog entry in its entirety. Its worth a proverbial $ Trillion.
Quick Roadmap
Here's a quick roadmap (it has17 sections) -
- The Office of Personnel Management (OPM), United States Federal Government
- Single Biggest Breach of Data, But Hardly Surprising
- House Oversight and Government Reform Committee’s 2nd Hearing on OPM Breach (Video Excerpts)
- The Attack Surface at OPM seems to have been LARGE
- The Defining Step
- A Quick Word on Privileged Access and Privilege Escalation
- Subtle Clues
- White House Orders Rapid Cyber Security Fixes
- Some Helpful Recommendations for the White House and all U.S. Federal Agencies
- A Caveat when using Two-Factor Authentication for Active Directory Accounts
- Oh, one other thing - Bootstrapping Trust
- The $ Billion Difference between Audit and Auditing
- Oh, and Speaking of Attribution
- On Accountability - Middle and Senior Management Seem Clueless
- The Corporate World is Equally Vulnerable
- 5 Simple and Timely Recommendations for CEOs Worldwide
- PS: Here's an After Thought: Why is Cyber Security Always an After Thought?
+ U.S. Federal Agency IT staff may find this helpful -
How to Identify and Minimize Privileged Users/Accounts in Active Directory
I signed-off my last blog entry (
Is the Whole World Sitting on a Ticking Bomb) on the
Sony Hack, with a question:
Who's Next?
Well, it turns out, it is none other than ...
The Office of Personnel Management (OPM), United States Federal Government
(Well, strictly speaking it was Anthem, sometime in Q1, 2015, but that's a dwarf in comparison.)
As the world knows by now, in June 2015, the U.S Government announced possibly the biggest cyber security breach ever.
Specifically, on June 04, 2015, the U.S. Office of Personnel Management (OPM), the United States Federal Government's chief human resources agency,
announced that it had identified a cyber security incident potentially affecting personnel data for current and former federal employees, including personally identifiable information (PII).
It also announced that it has partnered with the U.S. Department of Homeland Security’s Computer Emergency Readiness Team (US-CERT) and the Federal Bureau of Investigation (FBI) to determine the full impact to Federal personnel.
In days to follow, it additionally announced that "through the course of the ongoing investigation into the cyber intrusion that compromised personnel records of current and former Federal employees announced on June 4, OPM has recently discovered that additional systems were compromised. These systems included those that contain information related to the background investigations of current, former, and prospective Federal government employees, as well as other individuals for whom a Federal background investigation was conducted."
It has since been reported that in a recent closed-door briefing to U.S. Senators, the Director of the FBI, James Comey indicated that the number of people impacted could be close to 18 million.
In essence, the perpetrators purportedly obtained unauthorized access to and exfiltrated a large volume of SF (Standard Form) 86 forms, the OPM questionnaire for national security positions. (The SF-86 is a 127-page form that contains detailed info such as date of birth, addresses, physical features, phone numbers, e-mail addresses, schools, passport numbers, etc. etc.)
In summary, a single cyber security breach at a single U.S. Federal Agency exposed highly sensitive, confidential personnel records and personally identifiable information (PII) of almost all U.S. Federal Government employees and contractors.
July 09 Update: Today the OPM
released a statement communicating that stated that they have concluded with high confidence that the PII of 21.5 Million individuals was stolen.
- Quoting OPM - "While investigating this incident, in early June 2015, OPM discovered that additional information had been compromised: including background investigation records of current, former, and prospective Federal employees and contractors. OPM and the interagency incident response team have concluded with high confidence that sensitive information, including the Social Security Numbers (SSNs) of 21.5 million individuals, was stolen from the background investigation databases."
This undoubtedly has to be one of the biggest, if not the biggest, losses incurred by any organization in any country, ever.
July 10 Update: Katherine Archeluta, Director of the OPM resigned, effective today.
Single Biggest Breach of Data, But Hardly Surprising
“
It may be the single biggest breach of data that our government has ever had,” Rep. Jason Chaffetz, a Utah Republican who chairs the House Oversight and Government Reform Committee, said recently, calling the stolen data “
the most sensitive information we have.”
“
This is not the end of American intelligence, but, it is a significant blow,” said Joel Brenner, a former NSA Senior Counsel.
The loss is colossal indeed, because the fact that this data is in who knows whose and how many hands by now, potentially jeopardizes U.S. National Security. It does so because it could be used to target large numbers of U.S. personnel who work in sensitive positions and who could be subject to blackmail, entrapment, extortion, bribery and other methods of espionage.
Some have characterized it as a Catastrophe. Others have asked if this is a (Cyber) Pearl Harbor. (See below.)
However, it is hardly surprising. I say so because
the likelihood of occurrence of such a breach was rather high, considering the large attack surface the perpetrator had to enact the
defining step that was instrumental in providing him/her/them the access that he/she/they needed to steal this data.
That defining step is the step that made this breach possible. Unfortunately, you're hardly going to find any details in any official statements from the OPM or the U.S. Government. You'll find many articles and opinions but hardly any substantive details. At best, what you'll find is the following description - "
The adversary leveraged a compromised KeyPoint user credential to gain access to OPM's network."
If I were the U.S. Government, or any Corporation in the world, I'd want to know what that
defining step was, so that I could swiftly enact adequate risk mitigation/reduction measures to prevent the occurrence of such a breach in our IT infrastructure.
Although you won't find any mention of this defining step in official statements,
I'll tell you what that defining step most likely was, and why I believe that the attack surface was LARGE, which is why it seems like it was merely going to be a matter of time before something like this happened.
However, before I reveal the defining step, I'd like to share a few valuable snippets from a hearing with you, because the second and definitive clue to the defining step is in a 10 second snippet of this 4 hour hearing.
(If you just want to get to the heart of it, you can skip to
The Defining Step section below.)
House Oversight and Government Reform Committee’s 2nd Hearing on OPM Breach
(This section can be skipped if you're short of time.)
On June 24, 2015 the House Oversight and Government Reform Committee, chaired by U.S. Representative Jason Chaffetz had its second hearing on the OPM data security breach to provide its members an opportunity to gain additional information on the security of the U.S. Office of Personnel Management (OPM) information systems and the data it is entrusted to protect.
If you have an earnest interest in this topic, I highly suggest that you make the time to view this hearing. I did. It was almost 4 hours long, and I heard every minute of it. In fact, I heard it two times over.
Here is the full video of this hearing -
At the very least, I'd recommend viewing the following fascinating/insightful snippets. The most insightful ones are in
red font. -
0:00:30 - Jason Chaffetz, U.S. Representative, [R] Utah
"
US $ 529 Billion is how much the Federal Government has spent on IT since 2008. Roughly US $ 577 million has been spent at the Office of Personnel Management.... We're in a situation here where, the Hurricane has come and gone and just now the OPM is wanting to board up the windows. That's what it feels like! This is a major, major security breach. One of the biggest if not the biggest we have ever seen."
0:01:30 - Jason Chaffetz, U.S. Representative, [R] Utah
"
The uncertainty is very disconcerting to a host of people, and its unacceptable to this committee and the Congress."
00:14:30 - Elijah Cummings, U.S. Representative, [D] Maryland
Appearing before the Senate Appropriations Committee, Director Archuleta testified that "
The adversary leveraged a compromised KeyPoint user credential to gain access to OPM's network."
0:42:27 - Ann Barron-DiCamillo, Director, Department of Homeland Security U.S. CERT
"
We especially need private companies to continue to work with government and to share information about cyber threats and incidents, so that through greater shared awareness, we can all be more secure from those who seek to do us harm."
0:55:23 - Jason Chaffetz, U.S. Representative, [R] Utah
Question: "
Miss Seymour, do you have a complete inventory of servers, database, network devices and people that have access to that information? Do you have a complete inventory of that?"
Answer: "
We have as complete an inventory as we can have."
...
Response: "
The Inspector General does not believe you."
2:15:25 - Ron DeSantis, U.S. Representative, [R] Florida
"
It seems to me that we have bureaucratic paralysis. Nobody is really accountable" ...
Question: "
Miss Archuleta, do you still believe you should remain in your position?"
Answer: "
I am more committed than ever to serve the employees of this administration."
Question: "
Do you accept responsibility?"
Answer: "
I accept the responsibilities that are given to the Director of the OPM, and I have fulfilled those responsibilities"
Question: "
You are not committing that anybody will be fired or held accountable because of this, correct?"
Answer: "
I am committing to you that we are going to do the best job we can."
2:17:44 - Ron DeSantis, U.S. Representative, [R] Florida
Question to Ann Barron-DiCamillo, Director, DHS, U.S. CERT: "
Does this constitute a Cyber Pearl Harbor?"
Answer: "
We use a severity scale, and based on the impact ...., we would consider this to be a medium to high severity level event..."
...
Ron DeSantis: "
I think the damage is very, very severe."
2:19:58 - Gerry Connolly, U.S. Representative, [D] Virginia
“
We are facing a systematic, organized, financed pernicious campaign by the Chinese government in the form of the people's liberation army with a trained unit to penetrate weak spots in our cyberworld. And that includes the federal government and it may include retail and commercial enterprises, certainly banks among them." ... "Whether we want to acknowledge it or not, we now are engaged in a low level, but intense new kind of cold war, a cyberwar, with certain adversaries, including China and Russia. And it is every bit as much a threat to the security and stability of this country, and we need to gird ourselves for this battle.”
2:26:00 - Will Hurd, U.S. Representative, [R] Texas
"
I also got a letter from the Chief Information Officer of OPM and it read: "Dear Mr. Hurd, the U.S. Office of personnel management recently became aware of a cyber security incident affecting its data and you may have been exposed. We have determined the data compromised in this incident may have included your personal information such as your name, social security number, date and place of birth and current or former address." ... "
However, nothing in this letter should be construed as the OPM or the U.S. Government accepting liability for any of the matters covered by this latter, or for any other purpose. We regret this incident."
2:28:30 - Will Hurd, U.S. Representative, [R] Texas
"
If you were in the private sector, the head of a publicly traded company and Ernst & Young was doing your yearly audit and you had at least five years of audit information saying that your digital infrastructure had some high risk to it and needed to be immediately fixed, the Board of Directors would be held accountable for criminal activity, by multiple years."
2:31:56 - Eric A. Hess, President and CEO, KeyPoint Government Solutions
"
There was an individual who had an OPM account that happened to be a KeyPoint employee, and that the credentials of that individual were compromised to gain access to OPM."
2:54:50 - Glenn Grothman, U.S. Representative, [R] Wisconsin
"
It surprises me you folks are not more contrite over what happened. It seems like you don't understand the enormity of the disaster that's happened here."
3:08:00 - Barbara Comstock, U.S. Representative, [R] Virginia
Question to Miss Archuleta: "
In the last 18 months, how many meetings have you had yourself, personally, where it has been exclusively about Cyber Security, and whom have they been with?" ... "Have you visited private sector, such as a data center, and seen what the private sector does?"
3:10:50 - Barbara Comstock, U.S. Representative, [R] Virginia
"
The person at the very top has to take that role. When Target had this breach, it wasn't just their CIO that lost their job, it was the CEO who lost their job. That's how that was responded to in the Private Sector."
3:18:10 - Mark DeSaulnier, U.S. Representative, [D] California
"
Sometimes you can feel passionate about things but not be capable of doing what you desire to do and I think we need to have a serious conversation. I know the Chairman has these concerns about -- to be perfectly honest -- whether the current administration is competent enough to protect this information from people who would hack us."
3:23:10 -
Jason Chaffetz, U.S. Representative, [R] Utah
“
The good people in the Inspector General's office have been warning about this since the '90s. and it was never taken care of.”
3:23:25 -
Jason Chaffetz, U.S. Representative, [R] Utah
“
We don't believe you. I think you're part of the problem. I think if we want different results, we're going to have to have different people. And if you want to refresh the deck and we want to put somebody else in charge, we have to do it. We have a crisis. That hurricane has come and blown this building down. I don't want to hear about putting boards up on windows and it's going to take years to get there. That's why i think it's time for you to go. Ms. Seymour, I'm sorry, but i think you're in over your head. And I think the seriousness of this requires new leadership and a new set fresh of eyes to do this. I wish you the best in life. I'm not out to get you. But you know what, this is as big as it gets."
3:33:49 -
Jason Chaffetz, U.S. Representative, [R] Utah
Question: "Is there anybody in the OPM system, whether they be an employee or a contractor, who is a foreign national?"
...
"
The fact that you two don't know, that's what scares me, that's what really scares me." (3:34:20)
3:34:30 - Jason Chaffetz, U.S. Representative, [R] Utah
Question: How many people have credentials, to become a network administrator?
Answer from Donna K. Seymour, CIO, U.S. OPM: I believe its about 50!
3:35:03 - Jason Chaffetz, U.S. Representative, [R] Utah
"Somebody gained network administrator access, and ..."
3:36:36 - Earl "Buddy" L. Carter, U.S. Representative, [R] Georgia
"
OPM, since 2008 has spent $ 577 Million on IT, ... yet we're still using a legacy system that was built in 1959 ... over 80% of our IT budget is being spent on legacy systems... ?!"
3:38:30 - You've got to see Mr. Carter's reaction!
It is worth mentioning that this hearing was attended by the following distinguished individuals -
- Jason Chaffetz, U.S. Representative, [R] Utah
- Earl "Buddy" L. Carter, U.S. Representative, [R] Georgia
- Elijah Cummings, U.S. Representative, [D] Maryland
- Katherine Archuleta, Director, U.S. Office of Personnel Management
- Donna K. Seymour, Chief Information Officer, U.S. Office of Personnel Management
- Ann Barron-DiCamillo, Director, Department of Homeland Security U.S. CERT
- Patrick E. McFarland, Inspector General, U.S. Office of Personnel Management
- Eric A. Hess, President and CEO, KeyPoint Government Solutions
- Robert "Rob" W. Giannetta, Chief Information Officer, U.S. Investigations Services
- Matt Cartwright, U.S. Representative, [D] Pennsylvania
- Bonnie Watson Coleman, U.S. Representative, [D] New Jersey
- Barbara Comstock, U.S. Representative, [R] Virginia
- Gerry Connolly, U.S. Representative, [D] Virginia
- Ron DeSantis, U.S. Representative, [R] Florida
- Mark DeSaulnier, U.S. Representative, [D] California
- Trey Gowdy, U.S. Representative, [R] South Carolina
- Michelle Lujan Grisham, U.S. Representative, [D] New Mexico
- Glenn Grothman, U.S. Representative, [R] Wisconsin
- Will Hurd, U.S. Representative, [R] Texas
- Ted Lieu, U.S. Representative, [D] California
- Stephen F. Lynch, U.S. Representative, [D] Massachusetts
- Carolyn Maloney, U.S. Representative, [D] New York
- Mark Meadows, U.S. Representative, [R] North Carolina
- John Mica, U.S. Representative, [R] Florida
- Eleanor Holmes Norton, Congressional Delegate, [D] District of Columbia
- Gary Palmer, U.S. Representative, [R] Alabama
- Stacey Plaskett, Congressional Delegate, [D] Virgin Islands
- Michael "Mike" R. Turner, U.S. Representative, [R] Ohio
- Tim Walberg, U.S. Representative, [R] Michigan
I also found it interesting that Tony Scott, the CIO for the U.S. Federal Government did not attend.
The Attack Surface at OPM seems to have been LARGE
Folks, before I share details of the defining step that enabled perpetrators to engage in the possibly the largest cyber security data theft ever at the OPM, it is important to establish some background.
Like the rest of the U.S. Federal Government, the IT infrastructure of the OPM too operates on the Microsoft Windows Server platform, and at the very foundation of its cyber security lies its foundational Microsoft Active Directory deployment.
In a Microsoft Windows Server based IT infrastructure, the proverbial "
Keys to the Kingdom" reside in Active Directory, since all privileged (administrative) user accounts and security groups are stored in, managed in and protected by Active Directory.
(I only know this given my
background. I also happen to be the author of Microsoft's
Bible on how organizations should delegate administrative responsibilities in Active Directory to establish and maintain least-privileged access (LPA) in their networks.)
In my estimate, there are between 6000 to 10,000 domain user accounts in OPM's Active Directory. (It appears that OPM may have around 30 Domain Controllers, 10+ DHCP and DNS Servers, and 10+ remote sites. Anyway, I digress)
In contrast, a 5-second snippet at the 3:34:30 mark (i.e. the 3 hours, 34 minutes, 30 seconds) reveals that there were in fact about 50 individuals with what OPM refers to as "Network Administrators". That's informal terminology for "System Administrator" and what she is alluding to is Active Directory Administrators.
So, there are/were at least 50 (i.e. fifty) individuals who had privileged-user access ("Keys to the Kingdom") in OPM's network!
Specifically, there were at least 50 domain user accounts that had administrative access in OPM's Active Directory, most likely by virtue of membership in Active Directory administrative security groups (e.g. Domain Admins, Enterprise Admins etc.)
(By the way, I doubt this number includes the number of individuals who may be granted varying levels of administrative access in the ACLs protecting the objects that represent their administrative accounts and groups in Active Directory. Also, I don't think that number includes the number of additional individuals that may have had varying levels of restricted (delegated) privileged access in OPM's Active Directory.)
Anyway, I'll give them the benefit of doubt and assume that the total number of privileged users in their network is 50.
In other words, about 0.5% of the entire user account base had privileged access in OPM's network.
That might seem low but it is rather high, considering that the compromise of a single privileged (i.e. administrative) user can cause colossal damage. After all, the higher the number of such individuals in an organization, the larger the attack surface.
(Some of
our customers have 25,000+ accounts in Active Directory, yet have less than 10 individuals who possess privileged (unrestricted administrative) access i.e. Enterprise/Domain Admin Level Access.)
The OPM had around 50 personnel who possessed privileged (i.e. administrative) access in their network, so the perpetrators had 50 potential targets to choose from, and that's a LARGE attack surface, considering that when you have 50 targets to choose from and you only need to compromise 1, that's relatively easy, compared to if you only had 5 targets to choose from.
This is exactly why it is so important, and in fact
paramount to minimize the number of
privileged users, and do so immediately. This is also why the White House too issued exactly this recommendation, It's attack surface reduction 101. (Details below.)
The Defining Step - Active Directory Privilege Escalation
Yet another 5-second snippet at the 3:35:03 mark (i.e. the 3 hours, 35 minutes, 03 seconds) reveals that in fact "
someone gained Network Administrator access and ...".
Based on our research, here is the most likely sequence of events involved in this cyber security breach at OPM (highly simplified) -
- Perpetrator gained non-privileged access to OPM's internal network
- Perpetrator identified the list of all privileged users in OPM's network
- Perpetrator targeted and compromised one of these 50 privileged user accounts
- Perpetrator used this privileged access to obtain access to the SF-86 database
- Perpetrator exfiltrated the data
Now, of the 5 steps listed above, steps 1, 2, 4 and 5 are rather easy to carry out.
It is step 3 above that is the defining step in which the perpetrator escalated his/her/their privilege from a non-privileged Active Directory user account to that of a privileged Active Directory administrative account, and this is exactly what is referred to as
Active Directory Privilege Escalation.
In fact,
we have been saying it for years now that Active Directory Privilege Escalation is the world's #1 cyber security risk, because, and I'll quote this from our corporate website (;
source) -
- "It is the world's #1 cyber security risk because it directly impacts the foundational security of every organization whose IT infrastructure is powered by Active Directory, and in these organizations it let's anyone with a domain user account identify and potentially exploit privilege escalation paths in Active Directory to obtain complete, unrestricted administrative privileges, within minutes, and subsequently use these privileges to bypass/disable all other security controls and obtain access to, compromise, steal, divulge and/or destroy virtually any or all organizational IT resources."
Now, in the interest of objectivity, I should mention that there are 2 main attack-vectors that hackers employ to elevate their privilege (i.e. gain privileged/administrative access) in Active Directory -
- Pass-the-Hash (PtH) - This is a well-established technique that has been used for many years. It involves the capture and replay of password hashes by perpetrators. Albeit effective, its Achilles' Heel is that it requires the target administrator to logon to a machine owned by the attacker. As organizations become aware about it, secure administrative practices substantially reduce the likelihood of this attack vector being used to compromise security.
- Reset-the-Password (RtP) - This is the next frontier in privilege escalation, and it will likely be a highly potent attack vector for years to come, because it simply relies on identifying a kill-chain of password resets by performing simple read-only Active Directory effective permissions analysis. A salient aspect of this attack vector is that unlike PtH, it does NOT require the target administrator to logon to any machine, let alone one owned by the attacker. It can thus be carried out from any domain-joined machine, because all it requires is read access to Active Directory and the ability to perform password-reset analysis in Active Directory environments. It is 100% mitigatable though, and mitigation involves ensuring that there are no privilege escalation paths in Active Directory.
Additional information on these two attack vectors can be found
here.
Only OPM, CERT and the FBI may know which of the two attack vectors
Pass-the-Hash or
Reset-the-Password attack vectors was used to escalate their privilege in OPM's Active Directory.
In essence, to summarize, here is what
most likely happened during the OPM data breach -
- "Malicious perpetrators gained authenticated network access to OPM's network by compromising a non-privileged OPM domain user account. They then used this authenticated non-privileged access to identify the list of all privileged access users in OPM's Active Directory deployment, and subsequently engaged in Active Directory Privilege Escalation to compromise one of these privileged access accounts, which they then used to obtain access to the SF-86 database. They subsequently exfiltrated the data. (They could potentially have used it to access just about anything else as well.)"
I'm saying "most likely" because only OPM, CERT and the FBI may know exactly what happened.
[ July 09 Update:
I penned this entry on July 07, and as you can see above, I had mentioned that "They (the perpetrators) could potentially have used it (privileged access) to access just about anything else as well." Sure enough. As you may know, on July 09, the OPM
announced that in fact the total number of individuals impacted now stands at 21.5 Million, because, and I quote "
While investigating this incident, in early June 2015, OPM discovered that additional information had been compromised: including background investigation records of current, former, and prospective Federal employees and contractors. OPM and the interagency incident response team have concluded with high confidence that sensitive information, including the Social Security Numbers (SSNs) of 21.5 million individuals, was stolen from the background investigation databases."
Specifically, here's the
anything else part, and I quote again - "
if you underwent a background investigation through OPM in 2000 or afterwards (which occurs through the submission of forms SF-86, SF-85, or SF-85P for either a new investigation or a reinvestigation), it is highly likely that you are impacted by the incident involving background investigations."
There it is. In addition to SF-86 forms, those who perpetrated this data theft also accessed and stole SF-85 and SF-85P forms.
]
A Quick Word on Privileged Access and Privilege Escalation
IT IS PARAMOUNT TO NOTE THAT NON-PRIVILEGED ACCESS IS ALMOST NEVER* SUFFICIENT TO PROVIDE THE PERPETRATOR, ACCESS TO SENSITIVE/CONFIDENTIAL DATA.
(*This assertion assumes that organizations do not allow
Authenticated Users read access to confidential data, but rather that they provision least-privileged access (LPA) to their IT resources via the use of specific Active Directory security groups.)
In other words, in most cases, simply having access to a non-privileged user account cannot by itself give a perpetrator access to sensitive access-controlled IT resources, as access to such resources is restricted i.e. only granted to specific individuals.
In almost every case though, if a perpetrator can escalate his/her privilege to that of a privileged user (e.g. a Domain Admin) from that of a non-privileged user, he/she can almost always obtain access to any IT resource, even if additional security controls are in place, because he/she can either directly login as an Administrator on the target machine (that hosts the IT resources) and use administrative privileges to take ownership of the resource, and/or simply modify the membership of the domain (Active Directory) security group gating access to the resource to effortlessly gain access to the IT resource, or/and in rare cases, with sufficient expertise, even decrypt encrypted data to gain access.
However, all of this requires privileged access.
Now it turns out that in Active Directory environments, by default, all authenticated users have unrestricted read access to all Active Directory content, so every insider, including all non-privileged users, and every perpetrator who may have compromised a non-privileged user account, can easily and instantly enumerate the list of all privileged users.
By the way, the default read access to Authenticated Users cannot be turned off or locked down, because if you do so, who knows how many components of the IT infrastructure might stop working and/or get impacted.
In other words, anyone with an Active Directory (domain user) account can instantly identify the list of all privileged users and groups in the network (i.e. in the Active Directory deployment.) So, within minutes of compromising a non-privileged domain user account, a perpetrator can find out exactly how many individuals have the "
Keys to the Kingdom" and who they are.
Once the perpetrator, has this list, he/she only needs to compromise any ONE of these privileged user accounts to gain access to the "
Keys to the Kingdom." Then, once you have the
Keys to the Kingdom, you can obtain access to whatever you desire.
Of course, an advanced perpetrator or a highly skilled and determined adversary would most likely identify and exploit an Active Directory Privilege Escalation path, enabling him/her to simply identify and compromise the weakest link in the system
In all likelihood, what happened at the OPM was that perpetrator successful engaged in Active Directory Privilege Escalation (via
PtH or RtP) to gain privileged access, and once he/she/they had privileged access, he/she/they subsequently effortlessly gained access to the SF-86 database.
By the way, for someone smart enough to be able to get in to an organization's network and then engage in Active Directory privilege escalation, exfiltrating the data out is of course, child's play.
Subtle Clues
Here are some clues that help paint a picture of what we believe most likely happened at the OPM -
Claim 1: At the foundation of OPM's network lies Active Directory
> "SBM uses Active Directory authentication and all access is tied to LAN/WAN accounts.” (Page 2 of Appendix 1)
> "OCIO uses the 1665 form for both AD access and access to specific applications." (Page 9)
+
Clue 2: Information from another publicly searchable source (Undisclosed)
Claim 2: Perpetrators compromised a privileged user account at the OPM
> The fact that the directive from the White House to all U.S. Federal agencies focused on privileged user accounts was sufficient to infer that a privileged user account compromise led to the data theft at OPM -
"Agencies were told to scan systems and check logs for indicators of threats, patch critical vulnerabilities "without delay," as well as tighten policies and practices for privileged users, including minimizing the number of people in this category and limiting the duration a privileged user can be logged in."
+
Clue 2: During the House Oversight and Government Reform Committee’s 2nd Hearing on OPM Breach, a 10-second snippet at the 3:35:03 mark (i.e. the 3 hours, 35 minutes, 03 seconds) reveals that "somebody gained Network Administrator access and..."
Claim 3: There were/are at least 50 privileged user accounts in OPM's
> During the House Oversight and Government Reform Committee’s 2nd Hearing on OPM Breach, a 10-second snippet at the 3:34:30 mark (i.e. the 3 hours, 34 minutes, 30 seconds) reveals that there were in fact about 50 individuals with what OPM refers to as "Network Administrators".
White House Orders Rapid Cyber Security Fixes
In light of this colossal security breach, in early June, the White House
ordered all federal agencies to take specific immediate steps to bolster their cyber security defenses.
- "Agencies were told to scan systems and check logs for indicators of threats, patch critical vulnerabilities "without delay," as well as tighten policies and practices for privileged users, including minimizing the number of people in this category and limiting the duration a privileged user can be logged in." In addition, the White House also wants to "dramatically accelerate implementation" of multi-factor authentication to "significantly reduce the risk of adversaries."
A quick piece-meal analysis of these recommendations tells us a lot.
- "scan systems and check logs for indicators of threats, patch critical vulnerabilities "without delay," : Based on the findings, it must not have been unreasonable to assume that perhaps other U.S. Federal agencies could potentially have been targeted as well. This recommendation is aimed at determining whether or not additional agencies may have in fact been targeted and compromised.
- "as well as tighten policies and practices for privileged users, including minimizing the number of people in this category" - The fact that very next thing the White House is asking all U.S. Federal agencies to act upon is to minimize the number of privileged users and tighten policies and practices for privileged users indicates that most likely, the compromise of a privileged user was at the heart of the OPM breach.
Most importantly,
the directive to minimize the number of privileged users is essential and in fact paramount, because it is the #1 risk reduction measure that not just Federal agencies but all organizations can enact to reduce their attack surface today.
After all, if the compromise of a single privileged-access (i.e. administrative) user can cause such colossal damage, then of course, the higher the number of such individuals in an organization, the larger the attack surface. The OPM had around 50 personnel who possessed privileged-access in their network, so the perpetrators had 50 potential targets to choose from.
When you're have 50 targets to choose from and you only need to compromise 1, that's relatively easy, compared to if you only had 5 targets to choose from. This is why it is so important, and fact paramount to minimize the number of privileged users.
Some Helpful Recommendations for the White House and all U.S. Federal Agencies
Given that the entire U.S. Federal Government (and virtually 85% of the world) runs on Active Directory, as former Microsoft Program Manager for Active Directory Security, I'd like to offer some valuable advice so as to help them swiftly and measurably reduce their attack surface.
Here are 6 steps all U.S. Federal Agencies can take today to help them swiftly and measurably reduce their attack surface -
- Immediately correctly identify all privileged users in your Active Directory deployment. This is the number to minimize.
- Understand the impact of compromise of a privileged Active Directory user account
- Familiarize yourself with the attack surface (; it is your entire Active Directory)
- Familiarize yourself with the attack vectors that can be used to compromise privileged users
- Identify sources of threat to privileged users, since APTs often compromise insider accounts first
- Finally, swiftly establish and enact strategic risk-mitigation measures to bolster your cyber security defenses
Source -
Paramount Defenses Shares Advanced Cyber Security Insight for U.S. Government and Organizations Worldwide
It is also helpful to know that when you're talking about privileged users in your network, you're primarily referring to Active Directory administrative accounts that might be members of various default Active Directory administrative groups (such as but not limited to Domain Admins, Enterprise Admins, Built-In Admins, etc.) as well as any custom delegated administrative groups that may be in use. In essence, instead of using vague terms like "
privileged users" and "
network administrators", it would be incredibly helpful to start using the standardized and correct terminology i.e.
Active Directory Administrators that have unrestricted and delegated administrative access.
A Caveat when using Two-Factor Authentication for Active Directory Accounts
The White House has also directed the accelerated deployment of multi-factor authentication for all privileged users.
When you implement two-factor authentication in an Active Directory environment, it can be invaluable to know there is a setting on each user account in Active Directory that indicates to the system that two-factor authentication is in use for that account.
(As show below on the domain user account for Ted Schlein, a Junior Cyber Security operator, the setting is referred to as
Smartcard is required for interactive logon and exposed in the Active Directory Users and Computer (ADUC)
Account Tab.
Due to the default and custom delegations in Active Directory, at any point in time, there are numerous individuals who are delegated varying levels of access on each of these accounts, and some of these individuals may have sufficient effective permissions/access to be able to disable this setting on one or more accounts, including possibly on privileged accounts.
As soon as one disables this setting, Active Directory will assign a random password to that account and anyone will be able to try and guess, or reset this user's password, and attempt to logon to that account with a password.
Thus, after you have implemented two-factor authentication, it will be imperative to ensure on an on-going basis that you know at all times, exactly how many individuals can disable this setting on every smart-card enabled domain user account.
When you have 5000 domain user accounts, and you need to make this determination on 5000 accounts, a manual attempt to determine effective permissions on 5000 Active Directory objects could take 5000 hours, each time you need to do so.
A viable alternative is to use report #7
Who can enable/disable smart card requirement for interactive logon by user accounts, of this
Privileged Access Audit Tool, scoped at the domain level (e.g. "dc=opm,dc=net"), and be done with it in 5 minutes.
Oh, one other thing - Bootstrapping Trust
Anytime you have a situation wherein a privileged user account is compromised, strictly speaking, you have to assume that somewhere in your network, the perpetrator has left a backdoor. It could be a single service running on any one of your 10,000+ computers, an administrative delegation on any of your privileged domain user-accounts, a malicious script on some machine that when run would open a backdoor for the perpetrator, etc. etc.
The point is that, in such situations, the only way to get back to a known trustworthy state again is to completely rebuild the entire network from the ground up.
However, such an undertaking can easily cost a proverbial $ Billion per occurrence, and if you extrapolate that across hundreds of U.S. Federal Agencies, you're easily looking at a proverbial $ Trillion. And that's just to get back to a known trustworthy state.
In light of this, you might perhaps find this blog entry from August 2013,
Bootstrapping Trust – A Billion Dollar Cyber Security Problem (Responding to a Domain Admin Account Compromise) to be meaningful. Prevention is always better than cure.
All said and done, in hindsight, if the OPM had minimized the number of Active Directory administrative accounts in 2013/4, today, the United States Government would most likely not have lost such valuable data. Sadly, the impact of this loss may be felt for years to come.
The $ Billion Difference between Audit and Auditing
It is very likely that numerous organizations will be seeking to acquire solutions that can help them identify privileged users in their environments. After all, how can you minimize the number of privileged users without being able to precisely identify them in the first place.
In their quest for a
Privileged User Access Audit Tool, they are very likely to encounter a host of Active Directory Auditing solutions, because Active Directory Auditing is a commodity today. The vendors of these solutions will most likely attempt to convince these organizations that what they need is an Auditing solution.
While Auditing undoubtedly helps fulfill logging requirements, it cannot and does not fulfill the need to be able to identify the users that have privileged access in Active Directory today. That need can only be fulfilled by an
Active Directory Effective Access Audit Tool (an
example), because the only way to identify privileged users in Active Directory is to determine the identities of all users who have sufficient effective access so as to be able to can enact privileged (administrative) tasks in Active Directory.
Thus, although an Auditing solution can help fulfill a logging requirement, it cannot help identify all the privileged users in your Active Directory. I just wanted to mention this because there is a huge misconception out there that Active Directory Auditing is sufficient to fulfill this need.
The difference is perhaps best understood with a simple scenario....
Imagine a hypothetical scenario in which a VIP (a Head of State, a CEO or a celebrity) is going to be presenting a speech in an outdoor arena, say in Central Park, Manhattan, New York. As you may know, Central Park is surrounded by almost a 100 high-rise buildings.
Now assume that the Secret Service has been informed of a credible threat that one or more snipers could pose a lethal threat to the VIP. Given that there are almost a 100 high-rise buildings, one or more snipers could be positioned in and take the shot from any one of over 10,000 windows (100 high-rise buildings x 100 windows per high-rise building.)
In light of this, would you rather have a security system that would alert you (and create an entry in a log)
AFTER a sniper had fired a shot (so you have a record of a fired shot), considering that the shot was successful in hitting and taking the target (the VIP) out
OR would you rather have a security system that could instantly and precisely identify the identity and location of each sniper
BEFORE any shot is fired, so you could send a SWAT out to neutralize the threat before he/she inflicts damage?
Most people I know would choose the latter, because it is ALWAYS better to prevent a catastrophic event from happening in the first place, rather than letting it happen, and collecting forensic evidence to find how it happened and who the perpetrator was.
Once the damage is done, it doesn't really matter that much as to who did it. What matters is that the damage was done.
In this example, Active Directory Auditing is the reactive (AFTER the event) security system that will provide you with a log-entry when a privileged user engages in unauthorized access AFTER he/she has done so, and an
Active Directory Effective Privileged Access Audit is the proactive (BEFORE the event) security system that will help you identify all the individuals in your Active Directory that possess privileged access today, and reveal exactly what all they are capable of doing, thus empowering you to instantly identify and revoke the access of any and every individual whom you believe should not have privileged access in your environment.
You see,
if you're in a situation where you're having to look at logs, the shot's already been taken, and the target may already have been hit (compromised), and the damage already done.
In essence, both Audit and Auditing are valuable,
but an Audit is almost always substantially more valuable than a record in a log, because it can help you identify and neutralize a potential threat
BEFORE it has the opportunity to inflict damage.
Simply speaking, would an organization rather have no idea as to how many privileged users there exist in its network, and come to know that an APT successfully compromised and used a privileged user's account to obtain access to valuable data AFTER the fact, or would an organization rather proactively identify and minimize the number of privileged users in its environment, so that it could better defend a known small number of accounts, and in all likelihood, PREVENT the compromise from happening, in the first place. (Even the directive issued by the White House clearly suggests the latter.) That is the $ Billion difference between Audit and Auditing.
Oh, and Speaking of Attribution
As we all know by now, the cyber security breach at the OPM is being attributed to the Chinese.
Quoting, Gerry Connolly, U.S. Representative, [D] Virginia -
“
We are facing a systematic, organized, financed pernicious campaign by the Chinese government in the form of the people's liberation army with a trained unit to penetrate weak spots in our cyberworld. And that includes the federal government and it may include retail and commercial enterprises, certainly banks among them." ... "Whether we want to acknowledge it or not, we now are engaged in a low level, but intense new kind of cold war, a cyberwar, with certain adversaries, including China and Russia. And it is every bit as much a threat to the security and stability of this country, and we need to gird ourselves for this battle.”
There is also much talk about
Determined Adversaries and
Advanced Persistent Threats (APTs), and many a cyber security expert will claim that numerous indicators (IP addresses, malware etc.) point to the Chinese.
At the end of the day, it doesn't really matter that much as to who took you out. What matters is that you were taken out, and that your high-value, sensitive data is now in wrong and potentially dangerous hands.
By the way, if a perpetrator is smart enough to compromise a high-value, high-security target, he/she is also smart enough to be able to plant false evidence that could implicate another perpetrator. For instance, potentially this attack could very well have been carried out by the Russians.
The Russians could easily have setup the exfiltration destination to point to an IP address in China, and simply sniffed or redirected the payload en-route such as by using a network sniffer or a prism on a apriori known specific route(r) en-route.
(Speaking of prisms, now that's something even the U.S. Government might know a thing or two about.)
(By the way, there is already code written in St. Petersburg, Russia running as
System on Domain Controllers (DCs) across large parts of the U.S. Federal Government, so I doubt the Russians would go through all this trouble; unless of course, they too don't have a clue about it.)
The point is that when you are a U.S. Federal Agency, it is not sufficient to say that we got hacked because the adversary was advanced and highly motivated.
When you are a U.S. Federal Agency, you have to assume that a 100 advanced adversaries are attempting to simultaneously breach you 365-24-7, and you have to accordingly have adequate defenses in place to protect yourself, especially when you are entrusted with and responsible for safeguarding valuable information such as that contained in the millions of SF-86 forms that were stolen.
I'm not saying attribution is not important. What I'm saying is that its not nearly as important as protecting yourself from getting compromised in the first place. So yes, it may very well be that the Chinese were behind this, and if so, I'm confident and I hope that the U.S. Government will adequately respond in time, but for now, the focus should be on learning from this incident and ensuring that all other U.S. Federal agencies bolster their defenses, and that's what the White House rightfully seems to be tactically focused on.
On Accountability - Middle and Senior Management Seem Clueless
In light of, and in the aftermath of this colossal breach, there have been numerous calls for the resignation of the Director of the OPM, Katherine Archuleta. She however maintains that she cannot be held responsible for the breach, since highly advanced perpetrators targeted the OPM and that the breach was in part caused due the presence of legacy systems.
I'm not about to comment on whether or not the Director of the OPM is ultimately accountable. That is a matter best left to Congressmen, Senators and U.S. Government officials.
(July 10 Update: Katherine Archeluta, Director of the OPM resigned, effective today.)
Here's what I can tell you.
Over the last half decade, almost 10,000 organizations from across 150 countries worldwide have knocked at our doors (unsolicited) requesting assistance, and in our experience in assisting these organizations, here's what we have found.
We have found that in many organizations worldwide,
middle and senior management seem to be blissfully ignorant about specific details involving privileged access management and the risks associated with the compromise of a privileged account.
(In a few cases, even the troops in the trenches don't seem to have a clue. But that's generally rare.)
In fact, there's a huge disconnect between the troops in the trenches and middle and senior command. And while the troops may know a thing or two about the subject, they seldom escalate their thoughts, needs and requirements upstream (apparently due to fear) and as a consequence, the troops in the trenches seldom get the weaponry they need to adequately defend the Kingdom.
For instance, an IT administrator might identify the need for a tool that can help him/her identify and minimize privileged users in the organization's IT environment. However, he/she does not have any budgetary authority to procure the tools, and he/she is seldom able to communicate the requirement upstream, mostly due to fear,
and in cases where he/she does communicate it upstream, because middle and senior management do not seem to know much about the topic, his/her requests are often turned down, and the most common reasons cited are -
Not a Priority and/or
No Budget.
Well, if the reduction and adequate protection of privileged user accounts in an organization is not a priority, I'd love to know what IS a priority. After all, what could be more important than
identifying and protecting the
Keys to the Kingdom?!
And I can assure you that
no CISO/CIO/CFO/CEO in his right mind will deny a budget request for the protection of the Keys to the Kingdom. However, sadly, in most situations such requests never make it all the way up, because some middle manager decides that this is not important.
A few weeks later the organization has a Cyber Security incident, and then everyone from the top, down, wants to know exactly how this happened, and why the organization's
Keys to the Kingdom were not adequately protected? (Incidentally, here's one cyber security enthusiast's attempt at putting together an unofficial timeline of the OPM breach.)
If that middle manager had only kept his ignorance (and in some cases, ego) aside, and done what was required of him (which is to have communicated the request upstream), the CEO and the organization would not end up in such a distressed situation.
You see, the devil is in the details. The troops in the trenches see it everyday. However, as you go up the management chain, detailed knowledge exponentially decreases. CISOs and CIOs are operating at the 50,000 feet level with fancy dashboards and briefings to attend, while the IT admins are struggling to identify and minimize the number of privileged user accounts in their foundational Active Directory.
In fact, most IT admins are having to break their heads and spends countless hours trying to solve the problem manually, even though
fully-automated tools that can solve this problem within minutes exist today, only because some middle-level manager (one or two levels above the IT admin) decided that this wasn't important and thus there was no budget for it.
If I were the CEO,
to begin with, I'd possibly fire the middle-level manager (beyond whom such an important request that impacted the entire organization's security never made it up), and have my CISO/CFO speak directly to my Enterprise Admins and Domain Admins to get an unequivocally clear picture of what my troops are seeing, and what they need to get the job done.
In the Corporate World, there has to be accountability, and consequently, at the end of the day, the CEO might have to resign. IF only the request from the troops in the trenches would have made it up all the way up to him, not only would such a breach have been avoided, but the CEO could have kept his job.
The keyword above was "many" ; this scenarios plays out in "many" organizations worldwide.
The Corporate World is Equally Vulnerable
What happened at the OPM could happen in virtually any business organization in the world today, because the operating environment and the attack surface are essentially the same.
Virtually every organization in the world operates on Active Directory. Large numbers of individuals have varying levels of privileged access provisioned in these Active Directory environments. No one knows exactly who has what privileged access, even though, with the right tools, any non-privileged account can be used to identify and subsequently compromise any one of these privileged accounts.
Getting non-privileged access in an organization is easy these days, because determined adversaries can skillfully use various simple social engineering techniques to target naive non-techie organizational users and compromise their accounts to get a foot in the door.
This foot in the door gets them non-privileged but authenticated access to Active Directory. The execution of a simple 10-second script gets them a list of many privileged users. In most organizations, this list is dangerously long. The perpetrator need only compromise any ONE of these privileged user accounts to gain unrestricted access to virtually every IT resource in the organization. Once that's done, its game over. He/she/they will access and exfiltrate whatever they want.
A few weeks later, someone in the organization might uncover that they've been compromised. A public announcement will follow, resulting in a huge PR debacle, loss of credibility, market value etc.
Depending on the asset that was stolen, the organization may or may not survive the breach.
All of it made possible, by the compromise of a
single privileged account i.e. a single Active Directory administrative account.
5 Simple and Timely Recommendations for CEOs Worldwide
In light of the above, I'd humbly like to offer 5 simple recommendations to CEOs worldwide, because in the Corporate World, it is the CEO who would be held accountable for such a breach -
- If not from us, please take a cue from the White House, and make it a top corporate priority to identify and minimize the number of privileged users in your foundational Active Directory deployment.
- Tactically, immediately establish a direct communication channel with the troops in the trenches (your top-level (Enterprise Admins, Domain Admins etc.) administrative personnel), understand their requirements and challenges, and empower them to help you succeed in implementing recommendation #1.
- Strategically, establish and implement maintainable, adequate risk mitigation measures as soon as possible to empower the organization to deal with such threats for the long-haul. As a part of the plan, also establish a direct chain of accountability from the very top to the very bottom.
- Get specific and demand specifics. Invest in helping all involved personnel learn more about the specifics involved in these attack vectors, ask specific questions, and demand fact-based precise answers. (E.g. The answer to the question - How many privileged users do we have today must not be "I believe about 50." It should be an exact, provable number (e.g. "2", "3", "10" etc.)
- Please take this VERY seriously because this impacts business continuity, and because in the corporate world, as the Captain of the Ship, they are going to hold you accountable.
(If you need help, let me know.)
By the way, here's one of the best ways to break the ice. In about 2 minutes, you can use
this free tool to find out who can reset your password, and or that of your CIO/CFO/CISO, and then request an explanation for why so. Its the perfect ice-breaker.
Alright, my time's up.
In summary, the OPM hack, like the Sony Hack, was yet another situation, wherein perpetrators got a foot in the door, then engaged in Active Directory Privilege Escalation to gain privileged access, and then steal valuable data. In the case of Sony, it was Sony's confidential IT assets. In the case of OPM, it turned out to be highly sensitive PII of millions of U.S. Federal Employees & Contractors.
At the heart of the OPM breach, was the compromise of a single Active Directory Administrative (Privileged User) account.
(To conclude, in today's digital world, all business and government organizations worldwide must assume that there is always someone out there trying to breach them, and understand that today cyber security is a matter of paramount defenses.)
Best wishes,
Sanjay
Founder and CEO
Paramount Defenses
PS: Here's an After Thought: Why is Cyber Security Always an After Thought?
Folks, over the last 24 months, we've witnessed so many high-profile breaches (Snowden, Target, JP Morgan, Sony, Anthem) and now OPM. In each case, you'll find that PRIOR to the breach, Cyber Security was almost always never a top organizational priority, and AFTER the breach, Cyber Security suddenly becomes a top priority.
For instance, consider the OPM ...
On February 02, 2015, Katherine Archuleta, Director of the OPM released its annual "
Summary Performance and Financial Information for Fiscal Year 2014". This
30-page report touches upon numerous aspects including OPM's Strategic Goals. It lists
9 goals, and Cyber Security is NOT on that list anywhere. In fact, in the entire
30-page document, there is NO mention of the phrase "Cyber Security" anywhere! Nada, not ONE mention. ZERO.
As we all know, on June 04, 2015, OPM announced that it had discovered the security breach, and only AFTER the breach did it release an 8-page report titled "
Actions to Strengthen Cybersecurity and Protect Critical IT Systems" enumerating all the steps the OPM was taking to strengthen security.
Even after so many recent high-profile breaches, it took a breach at the OPM to make Cyber Security a priority at the OPM.
(I hope my little after-thought (PS Note) on
Cyber Security being an After Thought makes it to Boardrooms across the world. Please pay heed - today, just about every organization
is at risk.)
PS2: If you found this to be valuable, please consider sharing it with others.
Kindly Note: All pictures are licensed, and all content is copyrighted.