Today Cyber Security plays a paramount role in global security. On this blog, the CEO of Paramount Defenses shares rare insights on issues related to Cyber Security, including Privileged Access, Organizational Cyber Security, Foundational Security, Windows Security, Active Directory Security, Insider Threats and other topics.

August 29, 2013

Bootstrapping Trust – A Billion Dollar Cyber Security Problem (Responding to a Domain Admin Account Compromise)


A few days ago I posed a seemingly simple question to the world's foremost global community of Active Directory Security Professionals How does one respond to a Domain Admin Account compromise?

I say "seemingly simple" because in fact it is a "very difficult" question. (Strictly speaking, it is not the answer that is difficult, but rather the implementation of the measures suggested by the "right" answer that is "very difficult".) 
Before I can answer the question, perhaps I should establish context.

Establishing Context - What does a Domain Account Compromise have to do with Cyber Security?
You see, at the very foundation of cyber security at 85+% of all organizations worldwide lies Microsoft’s Active Directory.

Consequently, in these organization, the entirety of their IT resources are ultimately protected by Active Directory. 
I think Microsoft's CISO, Bret Arsenault stated this most eloquently in a recently released white paper titled Best Practices for Securing Active Directory. Quoting Bret - "Active Directory plays a critical role in the IT infrastructure, and ensures the harmony and security of different network resources in a global, interconnected environment.
In other words, Active Directory plays a critical role in ensuring the proper functioning of and the security afforded to every account, group, laptop, desktop, server, application, etc. deployed in the IT infrastructure.

Consequently, it also follows that the compromise of an organization's Active Directory could potentially be used to compromise any account, group, laptop, desktop, server, any file stored on any server, any application, LOB server etc.

(By the way, given the substantially high potential impact of an Active Directory compromise, it is increasingly becoming a very lucrative target for malicious entities. This point too is made in that white paper.)
NOW, in every organization's Active Directory deployment, there are designated Domain Admin accounts that are all-powerful and have virtually complete and unrestricted access to Active Directory and by extension to all computers joined to the Active Directory, and thus realistically, by extension to the entire IT infrastructure.


By default, a Domain Admin account holder has complete unrestricted access to all resources in the entire IT infrastructure. This is because every system needs at least one admin account for administration purposes.
A consequence of this is that a rogue / coerced Domain Admin could also access, tamper, divulge and destroy virtually any organizational IT resource (i.e. he/she can also subvert virtually any existing security policy as well as disable / control / destroy auditing evidence of virtually any action on any domain-joined machine/system, directly / indirectly.) 
In other words Domain Admins hold the proverbial "Keys to the Kingdom" and thus control the very foundation of cyber security in the IT infrastructures of their respective organizations.


Impact of a Domain Admin Account Compromise
In light of the above, its worth considering the scenario wherein a Domain Admin account was either compromised, or a Domain Admin turned rogue, or whether his/her account was hijacked without his/her knowledge.

In such a situation, as indicated above, the Domain Admin account could basically be used to access, tamper, divulge and destroy virtually any organizational IT resource. His/her account could be used to create any number of accounts and grant them administrative access on any resource in the Active Directory, such as on OUs, admin accounts/groups, GPOs etc.

ONE MORE THING. A Domain Admin's account could also be used to potentially place time-bombed malicious software on ANY domain-joined machine (i.e. any laptop, desktop or server) that could do anything from letting the perpetrator back in from the outside, to automating the destruction of  large parts of the IT infrastructure from the inside, or allowing him/her to easily regain administrative control in the Active Directory itself at a later point in time.

As a result, ONCE a Domain Admin account has been compromised, the entire fabric of TRUST across the IT infrastructure is potentially compromised.

Again, I'm not the only one saying this. Here's a snippet from Microsoft IT's white paper -

"It is generally well-accepted that if an attacker has obtained SYSTEM, Administrator, root, or equivalent access to a computer, regardless of operating system, that computer can no longer be considered trustworthy, no matter how many efforts are made to “clean” the system. Active Directory is no different. If an attacker has obtained privileged access to a domain controller or a highly privileged account in Active Directory, unless you have a record of every modification the attacker makes or a known good backup, you can never restore the directory to a completely trustworthy state."

It continues to read -

"Even if you reinstall every domain controller in the forest, you are simply reinstalling the hosts on which the AD DS database resides. Malicious modifications to Active Directory will replicate to freshly installed domain controllers as easily as they will replicate to domain controllers that have been running for years."

In other words, a Domain Admin account compromise is potentially tantamount to a system-wide compromise.

Responding to a Domain Admin Account Compromise

In light of the above, you'll hopefully see that the ONLY WAY to re-establish TRUST, which is PARAMOUNT to cyber security is to completely re-build the entire IT infrastructure, because that is the only way to ensure that any and every potential machine where the malicious Domain Admin could have potentially left a back door has been wiped clean.

Anything less than a complete re-build will not GUARANTEE the elimination of that risk.

As you can imagine completely re-building an entire IT infrastructure costs a proverbial BILLION DOLLARS. Meaning it is VERY expensive and virtually impossible to do so, even in medium organizations let alone large ones. Not to mention that most organizations will not be able to afford the downtime involved in rebuilding an entire IT infrastructure.

Bootstrapping Trust – A Billion Dollar Cyber Security Problem

Bootstrapping trust, both proverbially and realistically, is thus a billion dollar cyber security problem today. (Hypothetically speaking, just the cost of organization-wide downtime to rebuild the IT infrastructure of a large organization could be a $B.)

The scenario painted above applies to over 85% of all organizations worldwide today, including to almost a 100% of business and government organizations that are household names today. This is a very real global challenge today.

Is There a Million Dollar Fix to this Issue?

As I indicated, although a complete re-build is the only guaranteed way to bootstrap trust a 100%, realistically speaking, it is simply not possible/feasible (without it considerably impacting organization wide business operation).

The following snippet from the same Microsoft IT white paper corroborates this -

"A compromise to the degree described earlier is effectively irreparable, and the standard advice to “flatten and rebuild” every compromised system is simply not feasible or even possible if Active Directory has been compromised or destroyed. Even restoring to a known good state does not eliminate the flaws that allowed the environment to be compromised in the first place."

If the ONLY way to re-establish a 100% TRUST is to rebuild the entire IT infrastructure, which costs a proverbial BILLION dollars, and is not really feasible, is there a proverbial MILLION dollar (i.e cheaper) way to address this challenge?

Yes, and No.

The No part first. Technically, if you're looking to establish a 100% trust, that is the only way to go. I mean you can establish 99% trust for a proverbial MILLION dollars, but NOT 100%. But for many organizations, 99% may be sufficient, especially given where they're at today. (You'd be surprised if I told you just how many organizations have 100s of Domain Admin accounts holders that they don't even know of.)

The Yes part now. There's an old saying - "Prevention is better than cure". Every organization will always need a handful of Domain Admins, because every system needs an Administrator. But NO organization needs or should have more than a handful of Domain Admins, because as indicated above, with each additional Domain Admin you have, you increase the likelihood of the compromise of a very high value IT asset.

So, the Yes in essence relies on prevention, and involves the following -
  1. Immediately, REDUCE the number of Domain Admin accounts to a bare minimum
  2. Ensure that you can impose the HIGHEST levels of trust in the few Domain Admin accounts
  3. Disable their use. Enable them on a need basis, and when used ensure that 2 PAIRS of eyes are involved
  4. Establish policies that communicate in no uncertain terms, that should a willful or abetted misuse of administrative authority be found, there will undoubtedly be legal repercussions for the perpetrator
  5. Proactively audit administrative access in Active Directory on a weekly/fortnightly basis
  6. Implement 24-7 Active Directory Auditing to monitor the management and use of Domain Admin accounts 
(This assumes, of course, that one does not have already a compromised Domain Admin account, today.)

The Yes, is thus not strictly a Yes, but is an approach that can be used to substantially reduce risk realistically. Protecting 3 Domain Admin accounts is far easier (and has a higher likelihood of success) than protecting 30 or 300 Domain Admin accounts.

Why do Organizations have so many Domain Admin Accounts?

Like Microsoft, in our collective experience at Paramount Defenses, over the last half decade, we have had the opportunity to help 1000s of organizations worldwide, and we often come across organizations that have an alarming large number of Domain Admin accounts.

In my humble opinion, the root of this problem has to do with a lack of consistent diligence by IT groups in establishing and enforcing a stringent policy that ensures that access is ALWAYS provisioned based on the principle of least privilege, even when it may be inconvenient to do so.

Most organizations end up trading off comfort for security in this regard, by giving away Domain Admin access to so many individuals just because it is an easy way to not have to deal with the hardship of determining and provisioning least privileged access.

I will also add that many times, IT groups are under-staffed because they just don't have sufficient budget, so they often end up taking the convenient road. It is thus equally important for Executive Management to make this a priority, and adequately fund their IT groups to address this challenge.

Lastly, let me also state unequivocally there there is no shortcoming in Active Directory in regards to being able to precisely authorize/specify access. In fact, Active Directory offers arguably the world's most highly securable security model, wherein access can be very precisely specified based on the principle of least privilege.

Reducing the Number of Domain Admin Accounts

In regards to reducing the number of Domain Admins to facilitate access to resources stored on files servers, and/or to LOB applications etc, there is absolutely NO need to have Domain Admin access. A combination of least privileged local machine-specific access and least privileged network access to the appropriate resources should be sufficient.

As it pertains to reducing the number of Domain Admins to facilitate access to resources within the Active Directory, I'd recommend IT groups to seriously consider leveraging the Delegation of Administration capability to its fullest potential.

Properly implemented, Delegation of Administration, based on the principle of least privilege can be used to reliably delegate all administrative responsibilities (tasks) that do not require Domain Admin access, and virtually eliminate the need for requiring Domain Admin access for all but the most critical of all admin activities (e.g. DCPROMOing a new DC.)

In fact, by leveraging secure (verifiable) least-privilege, role-based delegation in Active Directory, most organizations should be able to reduce their number of Domain Admin accounts by at least 90%, thus substantially reducing risk.

A Word of Caution

Some organizations choose to deploy 3rd party solutions for identity and access management delegation. This approach is an alternative to directly delegating control in Active Directory, and while it may appear to be a secure/efficient alternate, it may actually potentially be laden with high security risks.

For instance, such a proxy solution, may have a requirement that it needs to be run on a Domain Controller, or that it itself needs complete unrestricted access to Active Directory to create and manage objects.

Before deploying ANY solution on a Domain Controller, or ANY Solution that requires UNRESTRICTED access to the Active Directory, organizations must (in their own best interest) ensure that any such solution is at least as TRUSTWORTHY as is the Active Directory itself, because otherwise, it could easily become their weakest link.

When evaluating the TRUSTWORTHINESS of any such solution, one should think about factors like -
  1. Is the vendor offering this solution at least as trustworthy as Microsoft?
  2. Is this solution developed by highly proficient developers?
  3. Is it developed in a trustworthy country, or in some foreign country, where hackers or other malicious entities could easily (physically or via network security compromise) obtain access to its code-base?
Why is this important? It is important, because this solution will most likely either be running on a DC or require complete admin access to your Active Directory, so ensuring its utmost TRUSTWORTHINESS is paramount.

(I'm told that there is at least one 3rd party "Active Directory Delegation Solution" which is developed in / supported from Russia. In light of the Snowden affair, when it comes to cyber security, a Russian solution may or may not be a wise choice. In contrast, Active Directory's trustworthiness is unparalleled, which is why so many organizations worldwide choose to natively delegate administrative access in Active Directory.)

Time's Up

My 10 minute timer just rang, so I will have to end this here. I hope you now see why Bootstrapping Trust is a Billion Dollar Cyber Security Problem today, and why organizations MUST make it a high priority to reduce the number of Domain Admin accounts.  Using secure (verifiable) least-privilege, role-based delegation in Active Directory is one of the easiest ways in which organizations can reduce their number of Domain Admin accounts by at least 90%, thus substantially reducing risk.

Best wishes,

PS: 2016 Update: If you liked this post, perhaps you'll like this too - Defending Active Directory Against Cyberattacks