A Massive Cyber Breach at a Company Whilst it was Considering the 'Cloud'

(A Must-Read for all CEOs, CFOs, CIOs, CISOs, Board Members & Shareholders Today)


Folks,

Today was supposed to be an exciting Friday morning at a Multi-Billion $ organization since the world's top Cloud Computing companies were going to make their final pitches to the company's C-Suite today, as it was considering moving to the "Cloud."

With Cloud Computing companies spending billions to market their latest Kool-Aid to organizations worldwide (even though much of this may actually not be ready for mission-critical stuff), how could this company too NOT be considering the Cloud?



The C-Suite Meeting

Today was a HUGE day for this multi-billion dollar company, for today after several months of researching and evaluating their choices and options, the company's leadership would finally be deciding as to which Cloud Computing provider to go with.

This meeting is being chaired by the Chairman of the Board and attended by the following organizational employees -

Chief Executive Officer (CEO) Chief Financial Officer (CFO)

Chief Information Officer (CIO) Chief Information Security Officer (CISO)

 Also in attendance are about a dozen Vice Presidents, representing Sales, Marketing, Research and Development etc.

Meeting In-Progress

After breakfast, the presentations began at 9:00 am. The organization's CIO kicked off the meeting, rattling off the numerous benefits that the company could enjoy by moving to the Cloud, and minutes later the Vice President of Cloud Computing from the first Cloud Computing company vying for their business started his presentation. His presentation lasted two hours.

The C-Suite then took a break for lunch.

The next presentation began at 1:00 pm and was expected to last till about 4:00 pm. The Vice President of Cloud Computing from the second Cloud Computing company had started her presentation and was almost an hour into it, when all of a sudden this happened...

... the CISO's assistant unexpectedly entered the room, went straight to the CISO and whispered something into his ear.

Everyone was surprised, and all eyes were on the CISO, who grimly asked his assistant - "Are you 100% sure?"  He said "Yes."





Houston, We Have a Problem

The CISO walked up to the CIO and whispered something into his ear. The CIO sat there in complete shock for a moment!

He then gathered himself and proceeded to request everyone except the C-Suite to immediately leave the conference room.

He told the Vice President of this Cloud Computing company - "Hopefully, we'll get back to you in a few weeks."

He then looked at the CEO and the Chairman of the Board, and he said - "Sir, we have a problem!"




Its Over

The CEO asked the CIO - "What's wrong? What happened?"

The CIO replied - "Sir, about 30 minutes ago, an intruder compromised the credentials of each one of our 20,000 employees!"

The CEO was almost in shock, and just couldn't believe what he had just heard, so he asked - "Everyone's credentials?!"

The CIO replied - "I'm afraid yes Sir, yours, mine, literally everyone's, including that of all our privileged users!"

The CEO could sense that there was more bad news, so he asked - "Is there something else I should know?"

The CIO replied - "Sir, 15 minutes ago, the intruder logged on as an Enterprise Admin, disabled the accounts of each one of our privileged users, and used Group Policy to deploy malicious software to each one of our 30,000 domain-joined computers! By now, he could have stolen, exfiltrated and destroyed the entirety of our digital assets! We may have lost literally everything!"

The CEO was shocked! They'd just been breached, and what a massive breach it was - "How could this have happened?"

Mimikatz DCSync 

The CIO turned to the CISO, who stepped in, and answered the question - "Sir, an intruder used a tool called Mimikatz DCSync to basically request and instantly obtain the credentials of every single user from our foundational Active Directory deployment."

The CEO asked - "What is Active Directory?"

The CISO replied - "Sir, simply put, it is the very foundation of our cyber security"

The CEO then asked - "Wait. Can just anyone request and extract credentials from Active Directory?"

The CISO replied - "Sir, not everyone can. Only those individuals whose have sufficient access to do so, and by that I mean, specifically only those who have Get-Replication-Changes-All effective-permissions on the domain root object, can do so."

The CEO then said - "This does not sound right to me. I'm no technical genius, but shouldn't we have known exactly who all have this, whatever you just said, er yes that Get-Replication-Changes-All effective permissions in our Active Directory?!"

The CISO replied - "Sir, it turns out that accurate determination of effective permissions in Active Directory is actually very difficult, and as a result it is almost impossible to figure out exactly who has this effective permissions on our domain root!"

The CEO figured it out - "So you're saying that the intruder had compromised the account of someone who was not on your radar and not supposed to have this access, but actually did, and the intruder used that access to steal everyone's credentials?"

The CISO replied - "That's right. It appears we did not know that this someone had sufficient access (i.e. effective permissions) to be able to replicate secrets from Active Directory, because it is very difficult to accurately figure this out in Active Directory."

The CEO was furious! - "You're kidding right?! Microsoft's spent billions on this new fad called the "Cloud", yet it doesn't even have a solution to help figure out something as vital as this in Active Directory? How long has Active Directory been around ?!

The CISO replied - "Seventeen years."

The CEO then said in disbelief - "Did you just 17 years, as in S-E-V-E-N-T-E-E-N years?!  Get Satya Nadella on the line now! Perhaps I should #REFRESH his memory that we're a customer, and that we may have just lost a few B-I-L-L-I-O-N dollars!"

This is for Real

Make NO mistake about it. As amusing as it might sound, the scenario shared above is very REAL, and in fact today, most business and government organizations worldwide that operate on Active Directory have no idea as to exactly who has sufficient effective permissions to be able to replicate secrets out of their Active Directory. None whatsoever!

We can demonstrate the enactment of this exact scenario, and its underlying cause, to any organizations that wishes to see it.

This Could've Been (and Can Be) Easily Prevented 

This situation could easily have been prevented, if this organization's IT personnel had only possessed the ability to adequately and accurately determine effective permissions in their foundational Active Directory deployments.

Sadly, since Microsoft apparently never educated its customers about the importance of Active Directory effective permissions, most of them have no clue, and in fact have no idea as to exactly who can do what across their Active Directory deployments!

Unfortunately, Mimikatz DCSync is just the Tip of the Iceberg. Today most organizations are likely operating in the dark and have no idea about the actual attack surface, and thus about exactly who can create, delete and manage the entirety of their domain user accounts, domain computer accounts, domain security groups, GPOs, service connection points (SCPs), OUs etc. even though every insider and intruder could try and figure this out and misuse this insight to compromise their security.

Technically speaking, with even just minimal education and the right tooling, here is how easy it is for organizations to figure this out and lock this down today, i.e. to lock this down before an intruder can exploit it to inflict colossal damage - RIGHT HERE.

Oh, and you don't need to call Microsoft for this, although you certainly can and should. If you do, they'll likely have no answer, yet they might use even this to pitch you their latest toy, Microsoft ATA, and of course, their Cloud offering, Microsoft Azure.

Wait, weren't these C*O discussing the Cloud (and likely Microsoft Azure) just a few hours (and a few billion dollars) ago?!

Fast-Forward Six Months

Unfortunately, given the massive scale of this breach, the company did not survive the attack, and had to declare bankruptcy. The C*Os of this company are still looking for suitable employment, and its shareholders ended up losing billions of dollars.

All of this could've been prevented, if they only knew about something as elemental as this, and had the ability to determine this.

Summary

The moral of the story is that while its fine to fall for the latest fad, i.e. consider moving to the "Cloud" and all, but as AND while you consider and plan to do so, you just cannot let you on-prem cyber defenses down even for a moment, because if you do so, you may not have a company left to move to the Cloud. A single excessive effective permission in Active Directory is all it takes.

I'll say this one more time and one last time - what I've shared above could easily happen at almost any organization today.

Best wishes,

Sanjay

CEO, Paramount Defenses

PS: If this sounds too simple and high-level i.e. hardly technical, that is by intent, as it is written for a non-technical audience. This isn't to showcase our technical depth; examples of our technical depth can be found here, here, here, here, here  etc.  etc.



PS2: Note for Microsoft - This may be the simplest example of "Active Directory Access Control Lists - Attack and Defense."

Here's why - Mimikatz DCSync, which embodies the technical brilliance of a certain Mr. Benjamin Delpy, may be the simplest example of how someone could attack Active Directory ACLs to instantly and completely compromise Active Directory. On the other hand, Gold Finger, which embodies the technical expertise of a certain former Microsoft employee, may be the simplest example of how one could defend Active Directory ACLs by being able to instantly identify/audit effective permissions/access in/across Active Directory, and thus lockdown any and all unauthorized access in Active Directory ACLs, making it impossible for an(y) unauthorized user to use Mimikatz DCSync against Active Directory.



PS3: They say to the wise, a hint is enough. I just painted the whole picture out for you. (You may also want to read this & this.)

PS4: If you liked this, you may also like - How To Easily Identify & Thwart Sneaky Persistence in Active Directory

A Letter to Benjamin Delpy regarding Mimikatz & Active Directory Security

Dear Mr. Delpy,

Hello. I hope this finds you doing well. Although we don't personally know each other, I've heard about your work, and the impact it's had on improvements in Windows Security, so thank you for your contributions to the field of Windows Security.

To begin with, I'd like to apologize for not having been able to tune sooner into the incredible circus caused by your interesting hobbies over these past few years :-), as I was away, silently working on the world's most powerful cyber security weapon ...

Gold Finger 007G

... you know, one that can be used to go from breach to 0wned (AD) in about 5 minutes in any Active Directory deployment in the world, even and particularly in environments (such as the one described below) wherein an intruder couldn't accomplish much with Mimikatz. We've built it for the good guys though, so that they can prevent such scenarios from occurring.


By the way, like you, I too am a programmer. I personally wrote about 10% of the code for all of Gold Finger and 007G.
(The rest of it was written and tested by a team of < ~ 50 software dev and QA engineers, all of whom are ex-Microsoft.)

However, given its sheer power, I'm not too big a fan of releasing our work in the wild, because I believe in an old saying - "Along with great power, comes great responsibility." (As for this, it is merely temporary and controlled.)


(Oh I digress. I have a bad habit of digressing often, so my apologies. End of digression.)


I did however briefly hear about Microsoft scrambling to try and save its face by buying a little start-up whose founder (, and you too might find this funny,) not too long ago, one day, sat thinking for two hours (as to) how to build an attack path, and then realized that everyone has access to Active Directory! (Hilarious! - that video's here (2:10 onwards.))

Oh, by the way, I forgot to introduce myself. I'm the one who did a risk assessment of Microsoft Corporation's global Active Directory more than a decade ago and showed them...

 ...well, I'm not about to reveal what I showed them. They know it, and that's all that matters.

Oh, I'm also the one who amongst other things, wrote Microsoft's 400-page whitepaper on how to delegate privileged access in Active Directory, only about 10 years before the world had heard of Mimikatz or an Aorato.


Speaking of privileged access in Active Directory, as you probably know, that cool little recent DCSync feature of yours, would be rendered completely USELESS if the perpetrator didn't have sufficient rights to replicate secrets out of Active Directory.

Replicating Directory Changes All Extended Right on Domain root in Active Directory

And while that feature may wow some, to us it very simply boils down to this - if you already possess sufficient privilege to be able to replicate secrets out of Active Directory, you're already virtually God in that environment any way. But thank you nonetheless for showing the world just how easy it can be. (Actually its not that super easy too, so I respect your work.)

Interesting side note question - Do you think anyone in the world knows exactly who has the Get Replication Changes All extended right effectively granted to whom in any production Active Directory environment, wherein the domain root ACL has been changed even once? (I don't think so.)

Oh, and by the way, as you may well know, you can't generate a Kerberos Golden Ticket without having the NTLM hash for the krbtgt account, and if you don't have the Get-Replication-Changes-All extended right to request secrets from a DC, you can't have that hash until you can log on as an Admin on a Domain Controller, BUT, if you can log in as an Admin on a Domain Controller, buddy, its ALREADY GAME OVER RIGHT THERE!


So, why bother going a step or five forward (LSASS dump, hash extraction etc.), when its already game over! Just use the task scheduler to launch a cmd prompt as System, and you're playing God in no time ;-)


I mean, seriously, if you're telling me that its that easy for a perpetrator to be able to logon as Admin on a Domain Controller at hundreds or thousands of organizations worldwide, then I must say, the $400+ Billion Microsoft hasn't done enough to educate its customers about the importance of protecting its Domain Controllers, and probably ought to spend a petty $100 million to do so right away (because if that's that easy, forget ATA, nothing's going to be able to protect these organizations from getting 0wned in minutes.)


[Again, a (not so) small digression...


Active Directory / Cyber Security 101 for the World

If your Domain Controller has been compromised, you're DONE. Period.

Domain Controllers in a Data Center

From the AD Admin to the CEO, it's time to pack up, go home, and find another job, because believe you me, until you completely re-build the forest from scratch, he who knows what to do will have you for ETERNITY without needing any ticket, let alone a Golden Ticket, at all! (To whose who would like to see a demo of this, please let us know.)

Let me repeat that. If an unauthorized individual can login to a Domain Controller as Admin, you are done! D-O-N-E! 

Only ignorant organizations will continue to operate that Active Directory forest, because you'd be operating on a compromised foundation, and if you do, those who truly understand Active Directory security, will know (and could divulge) everything there is to know about you, tamper with anything, destroy anything at will etc. etc. And yes, even Microsoft's latest toy won't be able to stop them.

... end of digression.]



Mr. Delpy, if you'll allow me, I'd like to most respectfully paint a very simple picture for you, one wherein, one could possibly grow old and maybe even have grand-children, and yet not succeed with Mimikatz. For brevity, I'll keep it simple.

Consider a forest with one or more domains wherein -

1. Domain Controllers: All physically secured in data-centers, requiring triple-factor authentication to physically access. Interactive logon denied to everyone except an empty AD security group. Of course, fully patched (so no low-hanging fruit etc.), plus all default User Rights and Privilege assignments severely locked-down. Soldered USB drives and keyboards i.e. no chance of applying a keystroke logger or USB-anything. No 3rd party software of any kind on these machines. Etc. etc.

2. Get Replication Changes All Extended Right: Only granted to Domain Controllers.

3. Domain Admins: None. All responsibilities for Active Directory service and data management (i.e. account and group management) natively delegated in Active Directory. In fact, the Domain Admins group SID has been removed from across Active Directory by a simple execution of a baby of a tool called dsrevoke.

4. Active Directory Admins and Designated Workstations: Less than 5 AD admins, each one with alt accounts, and separate workstations for every day use and admin use. Admin use workstations are as secure as the DCs, and interactive and network logons only allowed to the designated admin. Soldered USB drives and keyboards i.e. no chance of applying a keystroke logger or USB-anything. No 3rd party software of any kind on these machines. Minimum 25-character passwords, changed frequently, never written or stored anywhere.

5. Active Directory Backups: Stored encrypted in a physical bank vault, physical access to which requires the CISO to escort you to them. Only the CISO can get into bank vault.

6. An Educated Active Directory Admin Team: These aren't you ordinary admins. They know NEVER to logon to any system they do not OWN and CONTROL. They're also bound by a simple policy - you logon to any other system and you will be instantly terminated. Besides, Group Policy prevents them from logging on anywhere else.

7. Zero Services Using Domain Admin Creds: Not a single service in the enterprise uses Domain Admin creds, because if you know how to do security, you know that nothing strictly requires running as Domain Admin.

8. No DCs in the Cloud: Yes, you heard it right. I don't care how iron-clad the SLA is, we're not putting a single DC in the cloud with any provider yet. Not just yet. (More on that in days to come.)

9. Microsoft ATA:  Microsoft ATA is NOT deployed just yet, because if you think about it, its still just some rudimentary technology acquired from a start-up, and rolled out in just months. Not sure how robust and reliable it is in code and its art to protect multi-$ Billion companies.

I could go on and on, but you get the drift. (BTW, if you want to make it harder, throw in 2-factor authentication, at least for all admin accounts at a minimum, and a bunch of other latest Band-aids, none of which can stop a knowledgeable perpetrator.)

In such an environment, you might be able to compromise a domain-joined machine, but you're likely not going to get anywhere close to getting Domain Admin type credentials, because a Domain Admin type account will NEVER logon to a system 0wned by an intruder, and the intruder will almost NEVER be able to logon to an admin workstation or on to a DC as admin. Period.

In such a situation, I doubt an intruder's going to have a field day harvesting or reusing credentials to gain administrative access to Active Directory. In all likelihood, he/she'd sit and wait, and wait and wait, and wait, and grow old, and still not get anywhere.

Disgruntled intruder not getting any success with Mimikatz :-(

Hmm. Not a fun day with Mimikatz anymore, is it :-( 

NOW, I'll be the first to admit that the environment described above is a rarity, but not an impossibility by any stretch. In fact, in time, the world will get there, and we already are there. For instance, our Active Directory environment is only about 50 times tighter than I have described above, so at least one such environment exists, and we operate it most easily. But in reality (, and 8,000+ organizations from 160+ countries have knocked at our doors, completely unsolicited of course, so) we know just how bad the situation is out there.

End of digression. Back on track...

Hmm. I wonder if there might be a way for an intruder to compromise a relatively more secure Active Directory environment, specifically, such as the one described above? Well, it doesn't seem very likely, at least not using credential theft based attack vectors, since no Active Directory admins seem to want to give the bad guys even a chance at getting them.




Duh!

But wait, if this is all about Active Directory Security, then it doesn't take a brilliant mind to figure out the following - hey let me actually look INTO Active Directory, and I mean it literally! Look INTO it! -

Inside Active Directory

Wow, now that I'm looking in, it is an ocean in there!

And since we're talking about security here, perhaps its worth looking at something called an Access Control List (ACL), you know, that little thing that protects each object in Active Directory, including every administrative account, administrative group, domain user account, domain computer account (and of course the domain controller's domain computer account), the domain root, AdminSDHolder, the System Container, the entire Configuration container, the Schema container, every GPO, etc. etc.-

Active Directory ACL on some random user, say an Alex Simon's domain user account.

Hmm. I wonder why nobody has looked INTO the Active Directory that much? And by that, I don't mean looking at basic kid-stuff enumeration of accounts, groups, GPOs, DCs etc. I mean looking a little deeper, and a little more intelligently.


You know, like trying to figure out who can actually reset a Domain Admin account's password, or who can change the Domain Admins group membership, or perhaps who can change the ACL on the AdminSDHolder object, etc. such as and/or ...

... speaking of say, Mimikatz, you know who can modify the ACL protecting the Domain root to grant someone or themselves the Get Replication Changes All extended right so they could replicate secrets (password hashes) out from Active Directory, or say, grant themselves Full Control over the entire Active Directory domain. (You see, intelligent and (very) powerful stuff.)

[Quick Digression -  

Now, a novice might say - "You don't need an effective permissions tool to assess REPL perms on NC." Well, if no one has ever changed the ACL on the NC head, and you're just dealing with default permissions, then its his lucky day and a simple group enumeration should do it, BUT, if the permissions on the NC head have changed even once, and there's even ONE deny permission there (e.g. Deny All extended rights to Group X), well, then tough luck, because if you really want an accurate picture, you're going to have to determine effective permissions on the NC head ;-)"

End of Digression]

Hmm. I wonder. Let me see. Well, to begin with, what could I figure out by looking into these ACLs?

Not much, other than exactly who has the keys to every door in the kingdom, and the Keys to the Kingdom of course!

Hmm. Wonder how I do it? Yes, I know! I could use a tool like dsacls from Microsoft or I could pick up several amateur scripts from TechNet which claim to help find out who has what permissions -

dsacls

But wait, I see all kinds of permissions :-( > Allow permissions, Deny permissions, Explicit permissions, Inherited permissions, over a dozen basic permissions, five dozen plus special permissions (Extended rights), Validated writes, permissions with IO flags, etc. etc.

Whoa! Where do I even begin?

And does "who has what permissions" even equal "who actually has what permissions"?

Well, maybe I should read Microsoft's bible on all this Active Directory permissions stuff, and perhaps in a year or so, get my head around all this, and could possibly get somewhere. Perhaps, that's when I'll realize that what matters is who has what effective permissions, not who has what permissions!

Oh wait, yes, I've seen an Effective Permissions Tab in Active Directory Users and Computers, but I never did give it much thought! Hmm, its one out of four tabs in there, so it must be important -

Why don't I just use that to figure out who can do what within the Active Directory?

Sure, go ahead ... ...consider an environment consisting of thousands of users and you'll know why it is almost virtually useless.

Oh, and if you come across this, RUN, because its dangerously inaccurate. Same is true for the advice on TechNet forums.

Well, perhaps, if I had something like this, I could actually answer that question accurately and quickly -

Then ask yourself a simple question - what if I wanted to find out who can create a user account in Active Directory? Hmm - would I even know where to begin? Hey, this Active Directory has 100s of OUs, 100s of containers, etc. etc. - how many objects am I going to have to analyze effective access on to answer this one simple question. It'll be 100s if not 1000s.

Again, you could try for a year, or be done in seconds ...

An Effective Privileged Access Audit in Active Directory

But, you should absolutely try it yourself first, so you can get a sense of just how monumentally difficult it is do so. I insist!

Years may go by, but you'll eventually gain the knowledge to comprehend the profoundness of this, this, this and this.

When you're there, please let me know, because then you too will have learnt how the (really) smart guys find ways to the Keys to the Kingdom, or to any door they want in to, in minutes, without having to wait for someone to logon to a machine you 0wn!


When you get there, please let me know...

 

... I'm already there.


Until then, I wish you well, and hope you continue to enjoy hashes and tickets! Oh, and if the intriguing world of Active Directory Security (which literally lies within every Active Directory) has piqued your curiosity, here's some recommended 101 reading.

Personally though, as a fellow programmer who too has spent 1000s of hours on an ocean of technical detail that most would never get into, my humble unsolicited advice to you would be to also enjoy life because there's a lot more to it than just work. You've already proven how incredibly smart you are, and made a big (enough) difference, so thank you also for helping make Windows more secure for the world.

Sir, for your valued contributions to the field of cyber security, you have my respect, and that of the world.

I most sincerely wish you well.

Thank you.

Best wishes,
ST aka TS.

PS: Some RESPECK for you in a simple ($100B) question posed to Alex Simons at Microsoft - here

PS 2: For those who may be new to the subject, and wish to share your thoughts on Twitter- here + Closure and Clarity - here

PS 3: For those who may not yet understand the depth of what I just shared above, you may want to take a shot at answering any 1 of the 10 simple questions concerning Active Directory Security that I've posed to Sean Metcalf and Microsoft - here

PS 4: October 21, 2016 Update - Defending Active Directory Against Cyber Attacks - here

PS 5: May 22 2017: Taking Microsoft to Active Directory Security School - here

PS 6: June 01, 2017: How Well Does Microsoft Really Understand Cyber Security - here

PS 7: July 25, 2017 Update: You'll want to read this because its paramount to security - Active Directory Effective Permissions