Bootstrapping Trust – A Billion Dollar Cyber Security Problem (Responding to a Domain Admin Account Compromise)

Folks,

A few days ago I posed a seemingly simple question to the world's foremost global community of Active Directory Security Professionals – How does one respond to a Domain Admin Account compromise?

 

I say "seemingly simple" because in fact it is a "very difficult" question. (Strictly speaking, it is not the answer that is difficult, but rather the implementation of the measures suggested by the "right" answer that is "very difficult".) 

 

Before I can answer the question, perhaps I should establish context.

Establishing Context - What does a Domain Account Compromise have to do with Cyber Security?

 

You see, at the very foundation of cyber security at 85+% of all organizations worldwide lies Microsoft’s Active Directory.

 

Consequently, in these organization, the entirety of their IT resources are ultimately protected by Active Directory. 

 

I think Microsoft's CISO, Bret Arsenault stated this most eloquently in a recently released white paper titled Best Practices for Securing Active Directory. Quoting Bret - "Active Directory plays a critical role in the IT infrastructure, and ensures the harmony and security of different network resources in a global, interconnected environment.”

 

In other words, Active Directory plays a critical role in ensuring the proper functioning of and the security afforded to every account, group, laptop, desktop, server, application, etc. deployed in the IT infrastructure.

Consequently, it also follows that the compromise of an organization's Active Directory could potentially be used to compromise any account, group, laptop, desktop, server, any file stored on any server, any application, LOB server etc.

(By the way, given the substantially high potential impact of an Active Directory compromise, it is increasingly becoming a very lucrative target for malicious entities. This point too is made in that white paper.)

 

NOW, in every organization's Active Directory deployment, there are designated Domain Admin accounts that are all-powerful and have virtually complete and unrestricted access to Active Directory and by extension to all computers joined to the Active Directory, and thus realistically, by extension to the entire IT infrastructure.

 

By default, a Domain Admin account holder has complete unrestricted access to all resources in the entire IT infrastructure. This is because every system needs at least one admin account for administration purposes.

 

A consequence of this is that a rogue / coerced Domain Admin could also access, tamper, divulge and destroy virtually any organizational IT resource (i.e. he/she can also subvert virtually any existing security policy as well as disable / control / destroy auditing evidence of virtually any action on any domain-joined machine/system, directly / indirectly.) 

 

In other words Domain Admins hold the proverbial "Keys to the Kingdom" and thus control the very foundation of cyber security in the IT infrastructures of their respective organizations.

 

Impact of a Domain Admin Account Compromise

 

In light of the above, its worth considering the scenario wherein a Domain Admin account was either compromised, or a Domain Admin turned rogue, or whether his/her account was hijacked without his/her knowledge.

In such a situation, as indicated above, the Domain Admin account could basically be used to access, tamper, divulge and destroy virtually any organizational IT resource. His/her account could be used to create any number of accounts and grant them administrative access on any resource in the Active Directory, such as on OUs, admin accounts/groups, GPOs etc.

ONE MORE THING. A Domain Admin's account could also be used to potentially place time-bombed malicious software on ANY domain-joined machine (i.e. any laptop, desktop or server) that could do anything from letting the perpetrator back in from the outside, to automating the destruction of  large parts of the IT infrastructure from the inside, or allowing him/her to easily regain administrative control in the Active Directory itself at a later point in time.

As a result, ONCE a Domain Admin account has been compromised, the entire fabric of TRUST across the IT infrastructure is potentially compromised.

Again, I'm not the only one saying this. Here's a snippet from Microsoft IT's white paper -

"It is generally well-accepted that if an attacker has obtained SYSTEM, Administrator, root, or equivalent access to a computer, regardless of operating system, that computer can no longer be considered trustworthy, no matter how many efforts are made to “clean” the system. Active Directory is no different. If an attacker has obtained privileged access to a domain controller or a highly privileged account in Active Directory, unless you have a record of every modification the attacker makes or a known good backup, you can never restore the directory to a completely trustworthy state."

It continues to read -

"Even if you reinstall every domain controller in the forest, you are simply reinstalling the hosts on which the AD DS database resides. Malicious modifications to Active Directory will replicate to freshly installed domain controllers as easily as they will replicate to domain controllers that have been running for years."

In other words, a Domain Admin account compromise is potentially tantamount to a system-wide compromise.

Responding to a Domain Admin Account Compromise

In light of the above, you'll hopefully see that the ONLY WAY to re-establish TRUST, which is PARAMOUNT to cyber security is to completely re-build the entire IT infrastructure, because that is the only way to ensure that any and every potential machine where the malicious Domain Admin could have potentially left a back door has been wiped clean.

Anything less than a complete re-build will not GUARANTEE the elimination of that risk.

As you can imagine completely re-building an entire IT infrastructure costs a proverbial BILLION DOLLARS. Meaning it is VERY expensive and virtually impossible to do so, even in medium organizations let alone large ones. Not to mention that most organizations will not be able to afford the downtime involved in rebuilding an entire IT infrastructure.

Bootstrapping Trust – A Billion Dollar Cyber Security Problem

Bootstrapping trust, both proverbially and realistically, is thus a billion dollar cyber security problem today. (Hypothetically speaking, just the cost of organization-wide downtime to rebuild the IT infrastructure of a large organization could be a $B.)

The scenario painted above applies to over 85% of all organizations worldwide today, including to almost a 100% of business and government organizations that are household names today. This is a very real global challenge today.

Is There a Million Dollar Fix to this Issue?

As I indicated, although a complete re-build is the only guaranteed way to bootstrap trust a 100%, realistically speaking, it is simply not possible/feasible (without it considerably impacting organization wide business operation).

The following snippet from the same Microsoft IT white paper corroborates this -

"A compromise to the degree described earlier is effectively irreparable, and the standard advice to “flatten and rebuild” every compromised system is simply not feasible or even possible if Active Directory has been compromised or destroyed. Even restoring to a known good state does not eliminate the flaws that allowed the environment to be compromised in the first place."

If the ONLY way to re-establish a 100% TRUST is to rebuild the entire IT infrastructure, which costs a proverbial BILLION dollars, and is not really feasible, is there a proverbial MILLION dollar (i.e cheaper) way to address this challenge?

Yes, and No.

The No part first. Technically, if you're looking to establish a 100% trust, that is the only way to go. I mean you can establish 99% trust for a proverbial MILLION dollars, but NOT 100%. But for many organizations, 99% may be sufficient, especially given where they're at today. (You'd be surprised if I told you just how many organizations have 100s of Domain Admin accounts holders that they don't even know of.)

The Yes part now. There's an old saying - "Prevention is better than cure". Every organization will always need a handful of Domain Admins, because every system needs an Administrator. But NO organization needs or should have more than a handful of Domain Admins, because as indicated above, with each additional Domain Admin you have, you increase the likelihood of the compromise of a very high value IT asset.

So, the Yes in essence relies on prevention, and involves the following -

1. Immediately, REDUCE the number of Domain Admin accounts to a bare minimum
2. Ensure that you can impose the HIGHEST levels of trust in the few Domain Admin accounts
3. Disable their use. Enable them on a need basis, and when used ensure that 2 PAIRS of eyes are involved
4. Establish policies that communicate in no uncertain terms, that should a willful or abetted misuse of administrative authority be found, there will undoubtedly be legal repercussions for the perpetrator
5. Proactively audit administrative access in Active Directory on a weekly/fortnightly basis
6. Implement 24-7 Active Directory Auditing to monitor the management and use of Domain Admin accounts 

(This assumes, of course, that one does not have already a compromised Domain Admin account, today.)

The Yes, is thus not strictly a Yes, but is an approach that can be used to substantially reduce risk realistically. Protecting 3 Domain Admin accounts is far easier (and has a higher likelihood of success) than protecting 30 or 300 Domain Admin accounts.

Why do Organizations have so many Domain Admin Accounts?

Like Microsoft, in our collective experience at Paramount Defenses, over the last half decade, we have had the opportunity to help 1000s of organizations worldwide, and we often come across organizations that have an alarming large number of Domain Admin accounts.

In my humble opinion, the root of this problem has to do with a lack of consistent diligence by IT groups in establishing and enforcing a stringent policy that ensures that access is ALWAYS provisioned based on the principle of least privilege, even when it may be inconvenient to do so.

Most organizations end up trading off comfort for security in this regard, by giving away Domain Admin access to so many individuals just because it is an easy way to not have to deal with the hardship of determining and provisioning least privileged access.

I will also add that many times, IT groups are under-staffed because they just don't have sufficient budget, so they often end up taking the convenient road. It is thus equally important for Executive Management to make this a priority, and adequately fund their IT groups to address this challenge.

Lastly, let me also state unequivocally there there is no shortcoming in Active Directory in regards to being able to precisely authorize/specify access. In fact, Active Directory offers arguably the world's most highly securable security model, wherein access can be very precisely specified based on the principle of least privilege.

Reducing the Number of Domain Admin Accounts

In regards to reducing the number of Domain Admins to facilitate access to resources stored on files servers, and/or to LOB applications etc, there is absolutely NO need to have Domain Admin access. A combination of least privileged local machine-specific access and least privileged network access to the appropriate resources should be sufficient.

As it pertains to reducing the number of Domain Admins to facilitate access to resources within the Active Directory, I'd recommend IT groups to seriously consider leveraging the Delegation of Administration capability to its fullest potential.

Properly implemented, Delegation of Administration, based on the principle of least privilege can be used to reliably delegate all administrative responsibilities (tasks) that do not require Domain Admin access, and virtually eliminate the need for requiring Domain Admin access for all but the most critical of all admin activities (e.g. DCPROMOing a new DC.)

In fact, by leveraging secure (verifiable) least-privilege, role-based delegation in Active Directory, most organizations should be able to reduce their number of Domain Admin accounts by at least 90%, thus substantially reducing risk.

A Word of Caution

Some organizations choose to deploy 3rd party solutions for identity and access management delegation. This approach is an alternative to directly delegating control in Active Directory, and while it may appear to be a secure/efficient alternate, it may actually potentially be laden with high security risks.

For instance, such a proxy solution, may have a requirement that it needs to be run on a Domain Controller, or that it itself needs complete unrestricted access to Active Directory to create and manage objects.

Before deploying ANY solution on a Domain Controller, or ANY Solution that requires UNRESTRICTED access to the Active Directory, organizations must (in their own best interest) ensure that any such solution is at least as TRUSTWORTHY as is the Active Directory itself, because otherwise, it could easily become their weakest link.

When evaluating the TRUSTWORTHINESS of any such solution, one should think about factors like -

1. Is the vendor offering this solution at least as trustworthy as Microsoft?
2. Is this solution developed by highly proficient developers?
3. Is it developed in a trustworthy country, or in some foreign country, where hackers or other malicious entities could easily (physically or via network security compromise) obtain access to its code-base?
 

Why is this important? It is important, because this solution will most likely either be running on a DC or require complete admin access to your Active Directory, so ensuring its utmost TRUSTWORTHINESS is paramount.

(I'm told that there is at least one 3rd party "Active Directory Delegation Solution" which is developed in / supported from Russia. In light of the Snowden affair, when it comes to cyber security, a Russian solution may or may not be a wise choice. In contrast, Active Directory's trustworthiness is unparalleled, which is why so many organizations worldwide choose to natively delegate administrative access in Active Directory.)

Time's Up

My 10 minute timer just rang, so I will have to end this here. I hope you now see why Bootstrapping Trust is a Billion Dollar Cyber Security Problem today, and why organizations MUST make it a high priority to reduce the number of Domain Admin accounts.  Using secure (verifiable) least-privilege, role-based delegation in Active Directory is one of the easiest ways in which organizations can reduce their number of Domain Admin accounts by at least 90%, thus substantially reducing risk.

Best wishes,

Sanjay

PS: 2016 Update: If you liked this post, perhaps you'll like this too - Defending Active Directory Against Cyberattacks

NSA Contractor Edward Snowden Leaked Secrets - A Classic Example of Cyber Security Risks Posed by Trusted Insiders

Folks,

Edward Snowden needs no introduction, and I'm not about to opine on his actions.  What I would like to share my 2 cents on is the nature of this "security incident", and what government and business organizations worldwide can learn from it.



A Trusted Insider

This incident was a classic case of "unauthorized information disclosure" by "a trusted insider" with unrestricted access.

In this case, the "insider" seemingly had virtually "unrestricted" access to information, and the nature of information he accessed and divulged was so highly "sensitive" that the impact of its disclosure was colossal enough to cause a national government and a clandestine agency, potentially substantial harm, and embarrassment.


Risks to Cyber Security from Trusted Insiders

Unlike a traditional cyber security incident, involving an attack from an outsider, such a security incident is much harder, but not impossible, to protect against, because it involves a "trusted insider."

The threat of a security compromise from an insider always exists. However, few organizations take it seriously, perhaps because they perceive the "likelihood" of it to be low, or because they "perceive" the damage to being usually manageable, in that your average insider does not have administrative access and thus the extent of confidential information to which they could obtain access is usually limited.

However, in situations, wherein a highly trusted IT/Systems Administrator is involved, the damage can be substantial, as was the case here, because such admins almost always have unrestricted access to virtually the entire IT infrastructure, and are trusted with the great responsibility of safeguarding the organization's information assets.

So, when a highly trusted administrator turns malicious, there is very little you can do to stop him/her from inflicting substantial damage to the organization. That is because he/she can access, tamper, divulge and destroy virtually any organizational information asset he/she likes at will.

For example, should an accountant at a defense company leak the earnings numbers before their scheduled disclosure time, the impact would be limited to legal fall outs, but should a systems administrator leak the entire set of confidential blue-prints of the next supersonic plane the company was working on, such a breach could effectively put the company out of business.

This is why it is of paramount importance to ensure that organizations minimize the number of highly trusted administrators to an ABSOLUTE bare minimum. The importance of this elemental cyber security measure cannot be over-stated.

A Trusted Administrator

I know a thing or two about this, because I authored Microsoft's 400-page official white paper on delegating administration in Active Directory deployments, which deals with this very subject i.e. how to minimize the number of highly privileged administrative personnel to a minimum by delegating administrative authority based on the principle of least privilege.

Just one more thing. The method/system that NSA (and 20K+ organizations worldwide) would most likely have to use to find out who has what administrative powers in their IT infrastructures is protected by a patent, that I happen to be assigned.

(But I digress.)


Managing Risk Posed by Trusted Insiders with Unrestricted Administrative Access

The risk posed by a privileged trusted insider can almost never be completely eliminated because you will always have at least ONE person who will need to have (/ be able to obtain) unrestricted administrative access across the organization's IT infrastructure.

 

However, in most cases, the "likelihood" of this risk being materialized can be substantially minimized by reducing the number of highly privileged administrators, and by ensuring (to the extent possible) that those who do possess such unrestricted access are highly trustworthy and understand the serious implications of the misuse of their unrestricted administrative power.

Practically speaking though, if I were to share with you just how dismal the state of excessive administrative access entitlements is in most business and government organizations worldwide, you might fall out of your chair!

For instance, you'd be surprised if I told you just how many companies out there have 100s of Domain Admin accounts. In fact, in one company we came across, over 700 individuals had the ability to reset the password of the CEO's account, and login as the CEO on-demand within seconds. The only thing more scary is that no one including the CEO or these 700 admins knew about this. (Interestingly, one of their employees used Gold Finger Mini to figure this out in 30 seconds.)

(Anyway, I digress again, so back to the point at hand...)


What Can Organizations Do To Minimize The Risk Posed by A Malicious Trusted Insider?

The #1 thing organizations can do to minimize this risk is to understand and acknowledge just how serious and damaging a single such security incident can be for the organization.  (ONE such incident is all it takes to inflict substantial damage.)

Executive Management

Specifically, what is needed is for executive management to require and demand the enactment of adequate security risk management measures aimed at reducing the number of insiders who have unrestricted access to the IT infrastructure i.e. Domain Admins, Enterprise Admins and the like, i.e. folks whose job titles read "Infrastructure Consultant" etc.

Without executive support, this problem can almost never be adequately addressed.

Executive support is necessary because without it, the organization's IT group may not be able to drive the changes necessary to accomplish the reduction in the number of administrative accounts.

The #2 thing that organizations can do once executive support is in place, is to assign a high-priority IT project aimed at identifying the list of all individuals who have unrestricted or widespread access across their IT infrastructure.

Administrative Access Audit

This list should then be vetted out to understand the business requirements that drive/necessitate the provisioning of such unrestricted access for the identified individuals.

The vetting process must involve an analysis of why each of the identified individuals currently possess and require unrestricted administrative/system-wide access, and for each case wherein such access is not actually required, actionable steps must be identified to reduce/revoke such unrestricted administrative access, such that individuals only possess the least amount of access they need to fulfill their responsibilities.

The #3 thing organizations can do, is enact the steps identified in #2 above to minimize unrestricted administrative access to a bare minimum, by leveraging delegation of administrative responsibilities based on the principle of least privilege.

In other words, administrative access should be locked down based on the principle of least privilege.


Maintaining Security Post Initial Risk Reduction

It is not sufficient to minimize the number of privileged account holders, and then forget about it, because, unchecked, business requirements will invariably cause this number to get out of control again.

Thus it is imperative that all subsequent access provisioning requests be fulfilled in adherence to the principle of least privilege. This takes effort and time, but it is the harder right.

Also, to maintain security, on an ongoing basis, organizations should also periodically audit administrative access to ensure that the number of folks with unlimited /unrestricted system-wide access (as well as delegated access) is in line with what is expected, approved and authorized (i.e. not in violation of established business policy.)

It is also important to institute additional protection and monitoring measures to protect all accounts that have all-powerful administrative / unrestricted / system-wide access. In addition, it is equally important to establish policies that clearly state the ramfications of abuse of administrative power, and to communicate them to all powerful administrators. This deterrence measure is necessary.

If organizations enact just these 3 simple measures listed above, they could substantially reduce their attack surface, and thus reduce the likelihood of a successful "security breach" by a trusted insider.

For instance, you could use these measures to reduce the number of individuals who have unlimited administrative access from say 400, down to 40. Now, 40 is still 36 too many, but it is 360 less than the existing and unacceptable level of 400. (The number 400 is arbitrary, albeit representative of many large organizations, and primarily used to make the point.)


Time's Up

Given additional time, I could elaborate further, and provide additional and detailed guidance, but for now my 10 minutes are almost up, so this will have to be it.

My apologies if my 2c above is not proof-read by an editorial staff. Given my role at Paramount Defenses, I only have a few minutes each month to spend on "blogging", so this will have to be it.

Best,
Sanjay.

PS: There's no dearth of commercially motivated advice out there that seems to suggest the deployment of certain access management solutions in such situations. I'll add just this much - no software solution "in and by itself" can reduce this risk as much as the single fundamental step of actually reducing the number of individuals who possess unrestricted privileges can, because you cannot protect a system from the administrator of the system, because the administrator is, by definition, a part of the system's TCB (Trusted Computing Base.)

PS2: Here's something to think about in light of Mr Snowden's actions - http://www.sanjaytandon.com/integrity.html