Today Cyber Security plays a paramount role in global security. On this blog, the CEO of Paramount Defenses shares rare insights on issues related to Cyber Security, including the World's Top Cyber Security Risk, Advanced Persistent Threats (APT), Cyber Warfare, Corporate Espionage, Insider Threats and other topics.


Gold Finger The Paramount Brief Gold Finger Mini World Peace

June 18, 2018

Alarming! : Windows Update Automatically Downloaded and Installed an Untrusted Self-Signed Kernel-mode Lenovo Driver on New Surface Device

Folks,

Given what it is I do, I don't squander a minute of precious time, unless something is very important, and this is very important.


Let me explain why this is so alarming, concerning and so important to cyber security, and why at many organizations (e.g. U.S. Govt., Paramount Defenses etc.), this could've either possibly resulted in, or in itself, be considered a cyber security breach.

Disclaimer: I'm not making any value judgment about Lenovo ; I'm merely basing this on what's already been said.


As you know, Microsoft has been brazenly leaving billions of people and thousands of organizations worldwide with no real choice but to upgrade to their latest operating system, Windows 10, which albeit is far from perfect, is better than Windows Vista, Windows 8 etc., even though Windows 10's default settings could be considered an egregious affront to Privacy.

Consequently, at Paramount Defenses, we too felt that perhaps it was time to consider moving on to Windows 10, so we too figured we'd refresh our employees' PCs. Now, of the major choices available from amongst several reputable PC vendors out there, Microsoft's Surface was one of the top trustworthy contenders, considering that the entirety of the hardware and software was from the same vendor (, and one that was decently trustworthy (considering that most of the world is running their operating system,)) and that there seemed to be no* pre-installed drivers or software that may have been written in China, Russia etc.

Side-note: Based on information available in the public domain, in all likelihood, software written in / maintained from within Russia, may still likely be running as System on Domain Controllers within the U.S. Government.

In particular, regardless of its respected heritage, for us, Lenovo wasn't  an option, since it is partly owned by the Chinese Govt.

So we decided to consider evaluating Microsoft Surface devices and thus purchased a couple of brand-new Microsoft Surface devices from our local Microsoft Store for an initial PoC, and I decided to personally test-drive one of them -


The very first thing we did after unsealing them, walking through the initial setup and locking down Windows 10's unacceptable default privacy settings, was to connect it to the Internet over a secure channel, and perform a Windows Update.

I should mention that there was no other device attached to this Microsoft Surface, except for a Microsoft Signature Type Cover, and in particular there were no mice of any kind, attached to this new Microsoft surface device, whether via USB or Bluetooth.


Now, you're not going to believe what happened within minutes of having clicked the Check for Updates button!



Windows Update
Downloaded and Installed an Untrusted
Self-Signed Lenovo Device Driver on Microsoft Surface! -

Within minutes, Windows Update automatically downloaded and had installed, amongst other packages (notably Surface Firmware,) an untrusted self-signed Kernel-mode device-driver, purportedly Lenovo - Keyboard, Other hardware - Lenovo Optical Mouse (HID), on this brand-new Microsoft Surface device, i.e. one signed with an untrusted WDK Test Certificate!

Here's a snapshot of Windows Update indicating that it had successfully downloaded and installed a Lenovo driver on this Surface device, and it specifically states "Lenovo - Keyboard, Other hardware - Lenovo Optical Mouse (HID)" -


We couldn't quite believe this.

How could this be possible? i.e. how could a Lenovo driver have been installed on a Microsoft  Surface device?

So we checked the Windows Update Log, and sure enough, as seen in the snapshot below, the Windows Update Log too confirmed that Windows Update had just downloaded and installed a Lenovo driver -


We wondered if there might have been any Lenovo hardware components installed on the Surface so we checked the Device Manager, and we could not find a single device that seemed to indicate the presence of any Lenovo hardware. (Later, we even took it back to the Microsoft Store, and their skilled tech personnel confirmed the same finding i.e. no Lenovo hardware on it.)

Specifically, as you can see below, we again checked the Device Manager, this time to see if it might indicate the presence of any Lenovo HID, such as a Lenovo Optical Mouse, and as you can see in the snapshot below, the only two Mice and other pointing devices installed on the system were from Microsoft - i.e. no Lenovo mouse presence indicated by Device Manager -



Next, we performed a keyword search of the Registry, and came across a suspicious Driver Package, as seen below -


It seemed suspicious to us because as can be seen in the snapshot above, all of the other legitimate driver package keys in the Registry had (as they should) three child sub-keys i.e. Configurations, Descriptors and Strings, but this specific one only had one subkey titled Properties, and when we tried to open it, we received an Access Denied message!

As you can see above, it seemed to indicate that the provider was Lenovo and that the INF file name was phidmou.inf, and the OEM path was "C:\Windows\SoftwareDistribution\Download\Install", so we looked at the file system but this path didn't seem to exist on the file-system. So we performed a simple file-system search "dir /s phidmou.*" and as seen in the snapshot below, we found one instance of such a file, located in C:\Windows\System32\DriverStore\FileRepository\.

Here's that exact location on the file-system, and as evidenced by the Created date and time for that folder, one can see that this folder (and thus all of its contents), were created on April 01, 2018 at around 1:50 am, which is just around the time the Windows Update log too confirmed that it had installed the Lenovo Driver -



When we opened that location, we found thirteen items, including six drivers -


Next, we checked the Digital Signature on one of the drivers, PELMOUSE.SYS, and we found that it was signed using a self-signed test Windows Driver certificate, i.e. the .sys files were SELF-SIGNED by a WDKTestCert and their digital signatures were NOT OK, in that they terminated in a root certificate that is not trusted by the trust provider -


Finally, when we clicked on the View Certificate button, as can be seen below, we could see that this driver was in fact merely signed by a test certificate, which is only supposed to be used for testing purposes during the creation and development of Kernel-mode drivers. Quoting from Microsoft's documentation on Driver Testing "However, eventually it will become necessary to test-sign your driver during its development, and ultimately release-sign your driver before publishing it to users." -


Clearly, the certificate seen above is NOT one that is intended to be used for release signing, yet, here we have a Kernel-mode driver downloaded by Windows Update and installed on a brand new Microsoft surface, and all its signed by is a test certificate, and who knows who wrote this driver!

Again, per Microsoft's guidelines on driver signing, which can also be found here, "After completing test signing and verifying that the driver is ready for release, the driver package has to be release signed", and AFAIK, release signing not only requires the signer to obtain and use a code-signing certificate from a code-signing CA, it also requires a cross cert issued by Microsoft.

If that is indeed the case, then a Kernel-mode driver that is not signed with a valid code-signing certificate, and one whose digital signature does not contain Microsoft's cross cert, should not even be accepted into the Windows Update catalog.

It is thus hard to believe that a Windows Kernel-Mode Driver that is merely self-signed using a test certificate would even make it into the Windows Update catalog, and further it seems that in this case, not only did it make it in, it was downloaded, and in fact successfully installed onto a system, which clearly seems highly suspicious, and is fact alarming and deeply-concerning!

How could this be? How could Windows Update (a trusted system process of the operating system), which we all (have no choice but to) trust (and have to do so blindly and completely) have itself installed an untrusted self-signed Lenovo driver (i.e. code running in Kernel-Mode) on a Microsoft Surface device?

Frankly, since this piece of software was signed using a self-signed test cert, who's to say this was even a real Lenovo driver? It could very well be some malicious code purporting to be a Lenovo driver. Or, there is also the remote possibility that it could be a legitimate Lenovo driver, that is self-signed, but if that is the case, its installation should not have been allowed to succeed.



Unacceptable and Deeply Concerning

To us, this is unacceptable, alarming and deeply concerning, and here's why.


We just had, on a device we consider trustworthy (, and could possibly have engaged in business on,) procured from a vendor we consider trustworthy (considering that the entire world's cyber security ultimately depends on them), an unknown, unsigned piece of software of Chinese origin that is now running in Kernel-mode, installed on the device, by this device's vendor's (i.e. Microsoft's) own product (Windows operating system's) update program!

We have not had an opportunity to analyze this code, but if it is indeed malicious in any way, in effect, it would've, unbeknownst to us and for no fault of ours, granted System-level control over a trusted device within our perimeter, to some entity in China.

How much damage could that have caused? Well, suffice it to say that, for they who know Windows Security well, if this was indeed malicious, it would've been sufficient to potentially compromise any organization within which this potentially suspect and malicious package may have been auto-installed by Windows update. (I've elaborated a bit on this below.)

In the simplest scenario, if a company's Domain Admins had been using this device, it would've been Game Over right there!

This leads me to the next question - we can't help but wonder how many such identical Surface devices exist out there today, perhaps at 1000s of organizations, on which this suspicious unsigned Lenovo driver may have been downloaded and installed?

This also leads me to another very important question - Just how much trust can we, the world, impose in Windows Update?

In our case, it just so happened to be, that we happened to be in front of this device during this Windows update process, and that's how we noticed this, and by the way, after it was done, it gave the familiar Your device is upto date message.

Speaking which, here's another equally important question - For all organizations that are using Windows Surface, and may be using it for mission-critical or sensitive purposes (e.g. AD administration), what is the guarantee that this won't happen again?

I ask because if you understand cyber security, then you know, that it ONLY takes ONE instance of ONE malicious piece of software to be installed on a system, to compromise the security of that system, and if that system was a highly-trusted internal system (e.g. that machine's domain computer account had the "Trusted for Unconstrained Delegation" bit set), then this could very likely also aid perpetrators in ultimately gaining complete command and control of the entire IT infrastructure. As I have already alluded to above, if by chance the target/compromised computer was one that was being used by an Active Directory Privileged User, then, it would be tantamount to Game Over right then and there!

Think about it - this could have happened at any organization, from say the U.S. Government to the British Government, or from say a Goldman Sachs to a Palantir, or say from a stock-exchange to an airline, or say at a clandestine national security agency to say at a nuclear reactor, or even Microsoft itself. In short, for absolutely no fault of theirs, an organization could potentially have been breached by a likely malicious piece of software that the operating system's own update utility had downloaded and installed on the System, and in 99% of situations, because hardly anyone checks what gets installed by Windows Update (now that we have to download and install a whopping 600MB patch every Tuesday), this would likely have gone unnoticed!

Again, to be perfectly clear, I'm not saying that a provably malicious piece of software was in fact downloaded and installed on a Microsoft Surface device by Windows Update. What I'm saying is that a highly suspicious piece of software, one that was built and intended to run in Kernel-mode and yet was merely signed with a test certificate, somehow was automatically downloaded and installed on a Microsoft Surface device, and that to us is deeply concerning, because in essence, if this could happen, then even at organizations that may be spending millions on cyber security, a single such piece of software quietly making its way in through such a trusted channel, could possibly instantly render their entire multi-million dollar cyber security apparatus useless, and jeopardize the security of the entire organization, and this could happen at thousands of organizations worldwide.

With full respect to Microsoft and Mr. Nadella, this is deeply concerning and unacceptable, and I'd like some assurance, as I'm sure would 1000s of other CEOs and CISOs, that this will never happen again, on any Surface device, in any organization.

In our case, this was very important, because had we put that brand new Surface device that we procured from none other than the Microsoft Store, into operation (even it we had re-imaged it with an ultra-secure locked-down internal image), from minute one, post the initial Windows update, we would likely have had a potentially compromised device running within our internal network, and it could perhaps have led to us being breached.



If I Were Microsoft, I'd Send a Plane

Dear Microsoft, we immediately quarantined that Microsoft Surface device, and we have it in our possession.


If I were you, I'd send a plane to get it picked up ASAP, so you can thoroughly investigate every little aspect of this to figure out how this possibly happened, and get to the bottom of it! (Petty process note: The Microsoft Store let us keep the device for a bit longer, but will not let us return the device past June 24, and the only reason we've kept it, is in case you'd want to analyze it.)

Here's why. At the very least, if I were still at Microsoft, and in charge of Cyber Security -
  1. I'd want to know how an untrusted Kernel-mode device driver made it into the Windows Catalog
  2. I'd want to know why a Microsoft Surface device downloaded a purportedly Lenovo driver
  3. I'd want to know how Windows 10 permitted and in fact itself installed an untrusted driver
  4. I'd want to know exactly which SKUs of Microsoft Surface this may have happened on
  5. I'd want to know exactly how many such Microsoft Surface devices out there may have downloaded this package 

Further, and as such, considering that Microsoft Corp itself may easily have thousands of Surface devices being used within Microsoft itself, if I were still with Microsoft CorpSec, I'd certainly want to know how many of their own Surface devices may have automatically downloaded and installed this highly suspicious piece of untrusted self-signed software.


In short, Microsoft, if you care as deeply about cyber security as you say you do, and by that I'm referring to what Mr. Nadella, the CEO of Microsoft, recently said (see video below: 0:40 - 0:44) and I quote "we spend over a billion dollars of R&D each year, in building security into our mainstream products", then you'll want to get to the bottom of this, because other than the Cloud, what else could be a more mainstream product for Microsoft today than, Microsoft Windows and Microsoft Surface ?! -



Also, speaking of Microsoft's ecosystem, it indeed is time to help safeguard Microsoft's global ecosystem. (But I digress,)



In Conclusion

Folks, the only reason I decided to publicly share this is because I care deeply about cyber security, and I believe that this could potentially have impacted the foundational cyber security of any, and potentially, of thousands of organizations worldwide.


Hopefully, as you'll agree, a trusted component (i.e. Windows Update) of an operating system that virtually the whole world will soon be running on (i.e. Windows 10), should not be downloading and installing a piece of software that runs in Kernel-mode, when that piece of software isn't even digitally signed by a valid digital certificate, because if that piece of software happened to be malicious, then in doing so, it could likely, automatically, and for no fault of its users, instantly compromise the cyber security of possibly thousands of organizations worldwide. This is really as simple, as fundamental and as concerning, as that. 

All in all, the Microsoft Surface is an incredible device, and because, like Apple's computers, the entire hardware and software is in control of a single vendor, Microsoft has a huge opportunity to deliver a trustworthy computing device to the world, and we'd love to embrace it. Thus, it is vital for Microsoft to ensure that its other components (e.g. Update) do not let the security of its mainstream products down, because per the Principle of Weakest Link, "a system is only as secure as is its weakest link."


By the way, I happen to be former Microsoft Program Manager for Active Directory Security, and I care deeply for Microsoft.

For those may not know what Active Directory Security is (i.e. most CEOs, a few CISOs, and most employees and citizens,) suffice it to say that global security may depend on Active Directory Security, and thus may be a matter of paramount defenses.

Most respectfully,
Sanjay


PS: Full Disclosure: I had also immediately brought this matter to the attention of the Microsoft Store. They escalated it to Tier-3 support (based out of New Delhi, India), who then asked me to use the Windows Feedback utility to share the relevant evidence with Microsoft, which I immediately and dutifully did, but/and I never heard back from anyone at Microsoft in this regard again.

PS2: Another small request to Microsoft - Dear Microsoft, while at it, could you please also educate your global customer base about the paramount importance of Active Directory Effective Permissions, which is the ONE capability without which not a single object in any Active Directory deployment can be adequately secured! Considering that Active Directory is the foundation of cyber security of over 85% of all organizations worldwide, this is important. Over the last few years, we've had almost 10,000 organizations from 150+ countries knock at our doors, and virtually none of them seem to know this most basic and cardinal fact of Windows Security. I couldn't begin to tell you how shocking it is for us to learn that most Domain Admins and many CISOs out there don't have a clue. Can you imagine just how insecure and vulnerable an organization whose Domain Admins don't even know what Active Directory Effective Permissions are, let alone possessing this paramount capability, could be today?

June 14, 2018

Hello World, We Are Paramount Defenses

Folks,

Hello again. I know its been 6 months since I blogged, and considering that I penned 60+ posts last year, it feels like an eternity.

Perhaps I should introduce ourselves again ;-)

Hello World, We are …



I've been busy, but its finally time to help safeguard Microsoft's global ecosystem, so we'll start on June 18, 2018.

Best wishes,
Sanjay

December 31, 2017

2017 - The Year The World Realized the Value of Active Directory Security

Folks,

As we get ready to bid farewell to 2017, it may be fitting to recap notable happenings in Active Directory Security this year.

This appears to have been the year in which the mainstream Cyber Security community finally seems to have realized just how important and in fact paramount Active Directory Security is to cyber security worldwide, in that it appears that they may have finally realized that Active Directory is the very heart and foundation of privileged access at 85% of organizations worldwide!


I say so only because it appears to have been in this year that the following terms seem to have become mainstream cyber security buzzwords worldwide - Privileged User, Privileged Access, Domain Admins, Enterprise Admins, Mimikatz DCSync, AdminSDHolder, Active Directory ACLs, Active Directory Privilege Escalation, Sneaky Persistence in Active Directory, Stealthy Admins in Active Directory, Shadow Admins in Active Directory, Domain Controllers, Active Directory Botnets, etc. etc.



Active Directory Security Goes Mainstream Cyber Security

Here are the 10 notable events in Active Directory Security that helped it get mainstream cyber security attention this year -


  1. Since the beginning on the year, i.e. January 01, 2017, Mimikatz DCSync, an incredibly and dangerously powerful tool built by Benjamin Delpy, that can be used to instantly compromise the credentials of all Active Directory domain user accounts in an organization, including those of all privileged user accounts, has been gaining immense popularity, and appears to have become a must-have tool in every hacker, perpetrator and cyber security penetration-tester's arsenal.

  2. On May 15, 2017, the developers of BloodHound introduced version 1.3, with the objective of enhancing its ability to find privilege escalation paths in Active Directory that could help find out "Who can become Domain Admin?"  From that point on, Bloodhound, which is massively inaccurate, seems to have started becoming very popular in the hacking community.

  3. On June 08, 2017, CyberArk a Billion+ $ cyber-security company, and the self-proclaimed leader in Privileged Account Security, introduced the concept of Shadow Admins in Active Directory, as well as released a (massively inaccurate) tool called ACLight to help organizations identify all such Shadow Admins in Active Directory deployments worldwide.

  4. On June 14, 2017, Sean Metcalf, an Active Directory security enthusiast penned an entry-level post "Scanning for Active Directory Privileges and Privileged Accounts" citing that Active Directory Recon is the new hotness since attackers, Red Teamers and penetration testers have realized that control of Active Directory provides power over the organization!

  5. On July 11, 2017, Preempt, a Cyber Security announced that they had found a vulnerability in Microsoft's implementation of LDAP-S that permits the enactment of an NTLM relay attack, and in effect could allow an individual to effectively impersonate a(n already) privileged user and enact certain LDAP operations to gain privileged access. 

  6. On July 26, 2017, the developers of (massively inaccurate) BloodHound gave a presentation titled An ACE Up the Sleeve - Designing Active Directory DACL Backdoors at the famed Black Hat Conference USA 2017. This presentation at Black Hat likely played a big role in bringing Active Directory Security to the forefront of mainstream Cyber Security.

  7. Also on July 26, 2017, a second presentation on Active Directory Security at the Black Hat Conference titled The Active Directory Botnet introduced the world to a new attack technique that exploits the default access granted to all Active Directory users, to setup command and control servers within organizations worldwide. This too made waves.

  8. On September 18, 2017, Microsoft's Advanced Threat Analytics (ATA) Team penned a detailed and insightful blog post titled Active Directory Access Control List - Attacks and Defense, citing that recently there has been a lot of attention regarding the use of Active Directory ACLs for privilege escalation in Active Directory environments. Unfortunately, in doing so Microsoft inadvertently ended up revealing just how little its ATA team seems to know about the subject.

  9. On December 12, 2017, Preempt, a Cyber Security announced that they had found a flaw in Microsoft's Azure Active Directory Connect software that could allow Stealthy Admins to gain full domain control. They also suggested that organizations worldwide use their (massively inaccurate) tooling to find these Stealthy Admins in Active Directory.

  10. From January 26, 2017 through December 27, 2017, Paramount Defenses' CEO conducted Active Directory Security School for Microsoft, so that in turn Microsoft could help not just every entity mentioned in points 1- 9 above, but the whole world realize that in fact the key and the only correct way to mitigate each one of the security risks and challenges identified in points 1 - 9  above, lies in Active Directory Effective Permissions and Active Directory Effective Access.





Helping Defend Microsoft's Global Customer Base
( i.e. 85% of  Organizations Worldwide )

Folks, since January 01, 2017, both, as former Microsoft Program Manager for Active Directory Security and as the CEO of Paramount Defenses, I've penned 50+ insightful blog posts to help educate thousands of organizations worldwide about...


...not just the paramount importance of Active Directory Security to their foundational security, but also about how to correctly secure and defend their foundational Active Directory from every cyber security risk/challenge covered in points 1- 9 above.

This year, I ( / we) ...

  1. conducted 30-days of advanced Active Directory Security School for the $ 650+ Billion Microsoft Corporation

  2. showed thousands of organizations worldwide How to Render Mimikatz DCSync Useless in their Active Directory

  3. helped millions of pros (like Mr. Metcalf) worldwide learn How to Correctly Identify Privileged Users in Active Directory

  4. helped the developers of BloodHound understand How to Easily Identify Sneaky Persistence in Active Directory

  5. helped Microsoft's ATA Team learn advanced stuff About Active Directory ACLs - Actual Attack and Defense

  6. showed CyberArk, trusted by 50% of Fortune 100 CISOs, How to Correctly Identify Shadow Admins in Active Directory

  7. helped cyber security startup Preempt's experts learn How to Correctly Identify Stealthy Admins in Active Directory

  8. helped the presenters of The Active Directory Botnet learn How to Easily Solve the Problem of Active Directory Botnets

  9. helped millions of cyber security folks worldwide understand and illustrate Active Directory Privilege Escalation

  10. Most importantly, I helped thousands of organizations worldwide, including Microsoft, understand the paramount importance of Active Directory Effective Permissions and Active Directory Effective Access to Active Directory Security


In fact, we're not just providing guidance, we're uniquely empowering organizations worldwide to easily solve these challenges.





Summary

All in all, its been quite an eventful year for Active Directory Security (, and one that I saw coming over ten years ago.)

In 2017, the mainstream cyber security community finally seem to have realized the importance of Active Directory Security.


Perhaps, in 2018, they'll realize that the key to Active Directory Security lies in being able to accurately determine this.

Best wishes,
Sanjay.

PS: Why I do, What I Do.

December 29, 2017

Why I Do, What I Do

Folks,

I trust you're well. Today, I just wanted to take a few minutes to answer a few questions that I've been asked so many times.


Here are the answers to the Top-5 questions I am frequently asked -

  1. You're the CEO of a company (Paramount Defenses), so why do you blog so often, and how do you have time to do so?

    Good question. This is a bit of a unique situation, in that whilst I am the CEO of a company, I am also a subject matter expert in Active Directory Security (simply by virtue of my background) and thus I feel that it is my civic duty to help organizations understand the paramount importance of securing their foundational Active Directory deployments.

    In fact, over the last 7+ years, I've penned 150+ blog posts on Active Directory Security (here) and Cyber Security (here) on various topics such as Active Directory Privilege Escalation, the OPM Breach, Kerberos Token Bloat, Eff Perms, AdminSDHolder, Mimikatz DCSync, Sneaky Persistence, How to Correctly Identify Stealthy Admins in Active Directory, How to Correctly Identify Shadow Admins in Active Directory etc. and most recently on Active Directory Botnets.

    As to how I have the time to do so, that's actually not that difficult. We have a world-class team at Paramount Defenses, and I've been able to delegate a substantial amount of my CEO-related work amongst our executive leadership team.




  2. Speaking of which, how big is Paramount Defenses?

    At Paramount Defenses, we believe that less is more, so our entire global team is less than a 100 people. For security reasons, 100% of our staff are U.S. Citizens, and to-date, the entirety of our R&D team are former Microsoft employees.

    If by how big we are, you meant how many organizations we impact, today our unique high-value cyber security solutions and insights help adequately secure and defend thousands of prominent organizations across six continents worldwide.




  3. Why is it just you (and why aren't your employees) on Social Media (e.g. LinkedIn, Facebook, Twitter etc.)?

    The simple answer to this question - For Security Reasons.

    At Paramount Defenses, we care deeply about cyber security, so we also strive to lead by example in every way.

    As it pertains to cyber security, we have found that the presence of an organization's employees on social-media almost always results in excessive information disclosure that could be very valuable for hackers and various other entities who may have malicious intent, so our corporate policies do not permit a social media presence.

    Also, we're not huge fans of Twitter, and we certainly don't care about being on Facebook. We do like and appreciate LinkedIn, and in fact, we lead the world's largest community of Active Directory Security Professionals on LinkedIn.




  4. What do you intend to accomplish by blogging?

    The intention is to help organizations worldwide understand just how profoundly important Active Directory Security is to organizational cyber security, and how paramount Active Directory Effective Permissions are to Active Directory Security.

    That's because this impacts global security today, and here's why -




    You see, the Crown Jewels of cyber security reside in Active Directory, and if they're compromised, its Game Over. By Crown Jewels, I'm referring to privileged access, or as commonly known, Domain Admin equivalent accounts.

    It is a fact that 100% of all major recent cyber security breaches (except Equifax) involved the compromise of a single Active Directory privileged user account. Such accounts are Target #1 for hackers, which is why it is so very important that organizations be able to exactly identify and minimize the number of such privileged accounts in Active Directory.

    Now, when it comes to identifying privileged user accounts in Active Directory, most organizations focus on enumerating the memberships of their default administrative groups in Active Directory, and that's it. Unfortunately, that's just the Tip of the Iceberg, and we have found that most of them do not even seem to know that in fact there are FAR many more accounts with varying levels of elevated admin/privileged access in Active Directory than they seem to know about.

    This isn't a secret; its something you know if you've ever heard about Active Directory's most powerful and capable cyber security feature - Delegation of Administration. The truth is that at most organizations, a substantial amount of delegation has been done over the years, yet no one seems to have a clue as to who has what privileged access. Here's why.

    In fact, Active Directory privileged access accounts have been getting a lot of attention lately, because so many cyber security experts and companies are starting to realize that there exists a treasure-trove of privileged access in Active Directory. Thus, recently many such cyber security expert and companies have started shedding light on them (for example, one, two, three etc.), and some have even started developing amateur tools to identify such accounts.

    What these experts and companies may not know is that their amateur tools are substantially inaccurate since they rely on finding out "Who has what Permissions in Active Directory" WHEREAS the ONLY way to correctly identify privileged user accounts in Active Directory is by accurately finding out "Who has what Effective Permissions in Active Directory?"

    On a lighter note, I find it rather amusing that for lack of knowing better, most cyber security experts and vendors that may be new to Active Directory Security have been referring to such accounts as Stealthy Admins, Shadow Admins etc.

    To make matters worse, there are many prominent vendors in the Active Directory space that merely offer basic Active Directory Permissions Analysis/Audit Tooling, yet they mislead organizations by claiming to help them "Find out who has what privileged access in Active Directory," and since so many IT personnel don't seem to know better, they get misled.

    Thus, there's an imperative need to help organizations learn how to correctly audit privileged users in Active Directory.

    Consequently, the intention of my blogging is to HELP thousands of organizations and cyber security experts worldwide UNDERSTAND that the ONLY correct way to identify privileged users in Active Directory is by accurately determining effective permissions / effective access in Active Directory. There is only ONE correct way to accomplish this objective.




  5. Why have you been a little hard on Microsoft lately?

    Let me begin by saying that I deeply love and care for Microsoft. It may appear that I may have been a tad hard on them, but that is all well-intentioned and only meant to help them realize that they have an obligation to their global customer base to adequately educate them about various aspects of cyber security in Windows, particularly the most vital aspects.

    In that regard, if you truly understand cyber security in Windows environments, you know that Active Directory Effective Permissions and Active Directory Effective Access play an absolutely paramount role in securing Windows deployments worldwide, and since Active Directory has been around for almost two decades by now, one would expect the world to unequivocally understand this by now. Unfortunately, we found that (as evidenced above) no one seems to have a clue.

    You may be surprised if I were to share with you that at most organizations worldwide, hardly anyone seems to even know about what Active Directory Effective Permissions are, let alone why they're paramount to their security, and this a highly concerning fact, because this means that most organizations worldwide are operating in the proverbial dark today.

    It is upon looking into the reason for this that we realized that in the last decade, it appears that (for whatever reason) Microsoft may not have educated its global customer based about Active Directory Effective Permissions at all - Proof.

    Thus, it is in the best interest of organizations worldwide that we felt a need to substantially raise awareness.

    As to how on earth Microsoft may have completely forgotten to educate the world about this, I can only guess that perhaps they must've gotten so involved in building their Cloud offering and dealing with the menace of local-machine credential-theft attack vectors that they completely seem to have missed this one paramount aspect of Windows security.

    Fortunately for them and the world, we've had our eye on this problem for a decade know and we've been laser-focused. Besides, actions speak louder than words, so once you understand what it is we do at Paramount Defenses, you'll see that we've done more to help secure Microsoft's global customer base than possibly any other company on the planet.

    Those who understand what we've built, know that we may be Microsoft's most strategic ally in the cyber security space.


Finally, the most important reason as to why I do, what I do is because I care deeply and passionately about cyber security.

Best wishes,

December 12, 2017

Paramount Privileged Account Security Guidance (101) for CyberArk

Shadow Admins - The Stealthy Accounts That You Should Fear The Most, but Needn't Anymore


Folks,

Today's post concerns CyberArk's guidance on Privileged Account Security, a subject that is paramount to cyber security today, and it likely impacts Trillions of $, as it impacts the foundational cyber security of 85% of all organizations worldwide. I pen this as former Microsoft Program Manager for Active Directory Security, and thus as the world's top expert in privileged access.



An Intro to CyberArk

I shouldn't have to provide an intro to CyberArk (CYBR), a $ Billion+ cyber security company, because according to its website, CyberArk is the (self-proclaimed) leader in Privileged Account Security, with more than 3450 global companies, including more than 50% of the Fortune 100 companies, relying on its solutions to protect their most critical and high-value assets.

Image Attribution: CyberArk's Website
According to CyberArk's website - HALF OF FORTUNE 100 CISOs RELY ON CYBERARK.

If that is the case, then recent guidance provided by CyberArk's experts on a very important topic is a bit concerning.

Specifically, on June 08, 2017 CyberArk's researchers penned a blog post on their Threat Research Blog, which is presumably read by thousands, titled Shadow Admins - The Stealthy Accounts that You Should Fear The Most. In it, they've shed light on a category of privileged accounts they called Shadow Admin accounts, and introduced and recommended tooling that they have developed, and that according to them, could help organizations discover these Shadow Accounts in their networks.

It is concerning because as a subject matter expert, i.e. as former Microsoft Program Manager for Active Directory Security, it is my professional opinion that though its premise is accurate, the guidance and tooling provided in that post are inaccurate, and consequently any reliance upon it by organizations could result in a false sense of security, and leave them vulnerable.

Note: The specific details of the various inaccuracies are provided below in the section titled The Inaccuracy and a link to two demos that illustrate these inaccuracies is also provided in the section titled Accurate Guidance.

The remainder of this well-intentioned blog post is meant to help CyberArk and organizations worldwide understand this esoteric yet paramount aspect of organizational cyber security i.e. the so-called "Shadow Admins" and how to correctly discover them.





Privileged Account Security

Before I share why CyberArk's guidance may be inaccurate, its important to say a few words on Privileged Account Security.




The importance and value of Privileged Accounts is perhaps best summarized in line #1 of CyberArk's data-sheet -
"Privileged accounts represent the largest security vulnerability an organization faces today. These powerful accounts are used in nearly every cyber-attack, and they allow anyone who gains possession of them to control organization(al) resources, disable security systems, and access vast amounts of sensitive data."
CyberArk is 100% right. The compromise of even just 1 (i.e. ONE) such privileged account could easily grant perpetrators complete command and control over the entire IT infrastructure and empower them to swiftly enact a devastating cyber attack.

In fact, 100% of all major recent high-impact cyber security breaches (E.g. Snowden, Target, JP Morgan, Sony, Anthem, OPM) involved the compromise and subsequent misuse of a single, i.e. just ONE Active Directory Privileged User Account.

In that regard, CyberArk's focus on helping organizations adequately protect privileged accounts is spot-on and appreciated.


That said, and as you'll hopefully agree, "one can't protect what one can't identify" which is why the accurate discovery of all privileged accounts in an organization's network, especially of all Active Directory Privileged Access Accounts, is paramount.

In fact, it is exactly these privileged access accounts in Active Directory that CyberArk's well-intentioned blog sought to shed light on. Further, they likely felt this was very important (and they're right), which is why they even proceeded to develop tooling to help organizations identify all such accounts. Its just that perhaps CyberArk's experts too may not yet understand the intricate details of Active Directory Security well enough, and thus their well-intentioned guidance may have turned out to be inaccurate.

Speaking of which, this makes for a perfect segue, so please allow me to shed light on where CyberArk's well-intentioned guidance is inaccurate, and how organizations can correctly discover all such "Shadow Accounts" in Active Directory.




The so-called Shadow Admin Accounts

CyberArk's famous post on Shadow Admins, titled Shadow Admins - The Stealthy Accounts that You Should Fear The Most begins by describing what these so-called "Shadow Admin Accounts" are, and I quote -
"Shadow Admin accounts are accounts in your network that have sensitive privileges and are typically overlooked because they are not members of a privileged Active Directory (AD) group. Instead, Shadow Admin accounts were granted their privileges through the direct assignment of permissions (using ACLs on AD Objects)"

I've been working on Active Directory Security for almost two-decades know and may have personally clocked over 30,000 hours on the subject, and yet the first time I came across the term "Shadow Admins" was when I read CyberArk's blog post.

If you Google/Bing "Shadow Admins" you're likely not going to find many references to it, other than to CyberArk's post on their blog, and then all the places wherein numerous people who may have read their blog have shared this across the Web.

Ah! What CyberArk's researchers are referring to as "Shadow Admin" accounts are actually Active Directory user accounts that may not belong to any privileged Active Directory groups, yet may have been directly granted various security permissions at various locations (i.e. on various Active Directory objects) within Active Directory, SUCH THAT the permissions they've been granted effectively provide them with access that is tantamount to possessing privileged access in Active Directory.

The rest of us, who have been doing Active Directory Security for years, and by that I also mean and include thousands of Active Directory admins at organizations worldwide, typically refer to such accounts as "Delegated Admins" in Active Directory.

By the way, I only know this because while at Microsoft, I wrote the Bible on Privileged Account Security in Windows - i.e. back in 2004, I authored Microsoft's official 400-page whitepaper titled "Best Practices for Delegating Active Directory Administration."

For instance, here are 3 quick examples -
  1. James has Write-Property Member permissions specified in the ACL of the Domain Admins group.
  2. Emily has Reset Password permissions specified in the ACL of a Domain Admin's user account.
  3. John has Get-Replication Changes All permissions granted in the ACL of the domain root.
In each case above, even though Emily, James and John may not be a member of any one of the many default Active Directory admins groups, their access is*  tantamount to Domain-Admin equivalent access; this discovery might be startling for novices.

Like other accomplished cyber security folks who may have recently taken a keen interest in Active Directory Security (e.g. one, two, three, etc.), CyberArk's experts too may be new to Active Directory Security, and may have come to realize that indeed there likely possibly exist hundreds of such "Delegated Admin" accounts in Active Directory, many of whom may have what is tantamount to unrestricted privileged access in Active Directory, yet neither these account holders nor the organization's privileged users may know about them, BECAUSE it is very difficult to accurately identify/discover/audit these accounts.

Speaking of which, therein lies the inaccuracy in CyberArk's guidance and tooling, as explained below.




The Inaccuracy

Let me first acknowledge that CyberArk's general recommendation on Privileged Account Security are correct, and I quote -
"To maintain a strong security posture, CyberArk Labs highly recommends that organizations get to know all of the privileged accounts in the network, including those Shadow Admins.
That said, if you read their entire blog post, which I highly recommend every IT and Cyber Security professional and CISO to do, you'll find that CyberArk's experts seem to be making the same classic mistake that so many other have been making for years.


Specifically, here's that classic mistake, and again I quote from their post -
"Searching and analyzing the ACL permissions granted to each account is a more comprehensive method."
You see, searching and analyzing the permissions granted to each account in Active Directory ACLs is NOT the right way to find out exactly what level of access that account holder may actually (i.e. effectively) have in Active Directory.

Here's why - the ONLY CORRECT WAY to find out exactly who actually has what access in Active Directory, including of course any/all privileged access, is by determining "Active Directory Effective Permissions / Active Directory Effective Access."

This cardinal technical fact may be confirmed by contacting Microsoft .

Not only is there a HUGE difference between merely "searching and analyzing the ACL permissions granted to each account" and "determining effective permissions in Active Directory," more importantly the latter is a thousand times more difficult.

Incidentally, for reasons best known to Microsoft, for an entire decade, Microsoft apparently forgot to educate the world about the paramount importance of effective permissions/access in (and to the security of) Active Directory, which is also likely why even the authors of An ACE Up the Sleeve - Designing Active Directory ACL Backdoors, which likely was what prompted CyberArk's experts to look into and pen this post, also seem to have made the same mistake in their approach and tooling.

I find it amazing that based on this limited (and inaccurate) knowledge, CyberArk's experts even procceded to develop tooling, and I say so because unlike the developers of (the inaccurate) Bloodhound, CyberArk is a respected Billion $ company -
"...we have developed a special tool that scans and discovers privileged accounts based on account permissions. The tool, ACLight, is available for free on GitHub and can be used to discover these Shadow Admin acocunts on your network today...

We tested their ACLight tooling and unfortunately it failed even the most basic of tests that one could put such a tool through.

Consequently, it is in light of the above (i.e. their guidance seems to be based on incorrect technical facts and relies upon the use of tooling which too may be based on the same incorrect technical facts, and thus may likely be vastly inaccurate) that my professional opinion leads me to believe that the following guidance from CyberArk's experts is most likely inaccurate -
"...We encourage you to use our Shadow Admins scanning tool, ACLight, to start uncovering these accounts."

To help everyone clearly understand this, I've illustrated this in 2 DEMOS which can be accessed here.




Accurate Guidance

To help CyberArk's experts and the entire world better understand why the naïve approach of "searching and analyzing the ACL permissions granted to each account" is fundamentally flawed, and why it is effective permissions that matter, as well as how to CORRECTLY identify all such Shadow Admins in Active Directory, I've penned a separate blog post on my technical blog, and here's the URL -



I highly recommend that every IT and Cyber Security professional and every CISO, including the CISOs of half of the Fortune 100 that rely on CyberArk today as well as the half that don't rely on CyberArk yet, READ that insightful technical blog post.




Fear, No More

Those who truly understand Active Directory Security, and thus those who truly understand Privileged Account Security in Windows networks know that the ONLY CORRECT WAY to accurately identify all such Delegated Admins (or as CyberArk calls them, "Shadow Admins") in Active Directory is by determining effective permissions / effective access in Active Directory.

As former Microsoft Program Manager for Active Directory Security, let me be the first to tell you that accurately determining effective permissions in Active Directory, on even a single Active Directory object, is very difficult. To then be able to do so on thousands of objects in an Active Directory is almost a herculean task on par with scaling Mount Everest.

That said, if you can click a button, you needn't fear "Shadow Accounts" anymore because this tool can uniquely, instantly and accurately identify all such "Delegated Admins" (or if you prefer to call them "Shadow Admins") accounts in Active Directory -


It is the only tool in the world that can accomplish the herculean feat of being able to accurately identify all such "Delegated Admin" / "Shadow Admin" accounts in Active Directory, and it took over half a decade to build, thoroughly test and deliver.

Today, the world's most powerful government and business organizations across 6 continents worldwide rely on it.

We care deeply about all organizations, including all cyber security companies so I'll also be the first to tell you that it does NOT obviate the need for various privileged account security solutions that respectable companies like CyberArk and others provide.

It ONLY helps accurately discover/identify/audit all such accounts, but as CyberArk too has emphasized in their blog, that in itself is PARAMOUNT because "you cannot protect what you cannot identify" and just ONE such privileged account (of which at most organizations, there likely are hundreds today) is all that perpetrators need to discover and compromise to then be able to easily 0wn the Kingdom.



Summary

Ladies and Gentlemen, in closing, Privileged Account Security is paramount to organizational cyber security, and please don't just take my word for it, for here's CyberArk communicating in effect the same fact -
"Privileged accounts represent the largest security vulnerability an organization faces today. These powerful accounts are used in nearly every cyber-attack, and they allow anyone who gains possession of them to control organization(al) resources, disable security systems, and access vast amounts of sensitive data."
As I've said above, CyberArk is 100% right. The compromise of even just 1 (i.e. ONE) such privileged account could easily grant perpetrators complete command and control over your entire network and empower them to swiftly take over everything.

In fact, 100% of all major recent high-impact cyber security breaches (E.g. Snowden, Target, JP Morgan, Sony, Anthem, OPM) involved the compromise and subsequent misuse of a single, i.e. just ONE Active Directory Privileged User Account.


CyberArk is also 100% right that in most Active Directory deployments worldwide, today there likely exist a dangerously and excessively large number of such "Shadow Admin" accounts, that for all practical reasons possess the same level of privileged access as do members of default Active Directory administrative / privileged access groups, yet because they're not members of these default privileged access groups, these accounts are in fact very difficult to accurately identify.

Consequently, their presence may possibly post a FAR greater risk to organizational cyber security, which is why it is so very important for organizations to be able to accurately discover/identify all such accounts i.e. each and every single one of them.

We also appreciate CyberArk's well-intentioned efforts to offer guidance that could help organizations identify all such accounts. Unfortunately, because this is a rather esoteric subject, and Microsoft has apparently not provided any guidance on how to correctly identify such accounts, CyberArk's experts may not have known how to correctly identify all such accounts.

Thus, we were happy to have shed light on this paramount subject to help them and organizations worldwide better understand how to accurately identify all such "Shadow Admin" accounts. Towards the same, I also shared a pointer to a technical blog wherein we're illustrated the inaccuracy and classic mistake that most organizations make, as well as the correct approach.

Finally, for all such organizations that wish to be able to efficiently and accurately identify all such "Shadow Admin" accounts, I've also shared with you above how the world's most powerful government and business organizations easily do so today.

I hope you've found this to be helpful, and I wish you all, including CyberArk, all the very best.

We're all in this together.

Best wishes,
Sanjay


PS: (Highly) Recommended Reading -
  1. The Entire World runs on Active Directory
  2. Defending Active Directory Against CyberAttacks (Slide 88 alludes to CyberArk)
  3. Active Directory Effective Permissions
  4. Active Directory Privilege Escalation - A Trillion Dollar Example
  5. How to Thwart Sneaky Persistence in Active Directory
  6. How to Discover Stealthy Admins in Active Directory
  7. A Letter to Benjamin Delpy, a Letter to Microsoft, and a Letter to President Donald Trump

December 8, 2017

Time to DEMONSTRATE Thought Leadership in the Cyber Security Space

Folks,

Hope you're all well. Last year I had said that it was time for us to provide Thought Leadership to the Cyber Security space.


Since then, I've penned over 50 blog posts, on numerous important topics,
and helped 1000s of organizations worldwide better understand -

  1. The Importance of Active Directory Security

  2. Insight into Active Directory ACLs - Attack and Defense

  3. How to Defend Active Directory Against Cyber Attacks

  4. How to Mitigate the Risk Posed by Mimikatz DCSync

  5. How to Thwart Sneaky Persistence in Active Directory

  1. How to Identify Stealthy Admins in Active Directory

  2. Understand Windows Elevation of Privilege Vulnerability

  3. Illustrate Active Directory Privilege Escalation

  4. Correctly Identify Privileged Users in Active Directory

  5. Importance of Active Directory Effective Permissions
There's so much more to share, and I will continue to do so.





A Paramount Global Cyber Security Need

Today, I wanted to take a moment to touch upon one (not so) little aspect of cyber security that today profoundly impacts the foundational security of 85% of all business and government organizations worldwide, including most cyber security companies.

Folks, I am talking about empowering organizations worldwide identify exactly who holds the proverbial "Keys to the Kingdom" i.e. helping them accurately identify exactly who actually possesses what privileged access in Active Directory deployments.


The reason this is so important is because 100% of all major recent cyber security breaches (e.g. Snowden, Target, JP Morgan, Sony, Anthem, OPM) involved the compromise and misuse of guess what - just ONE Active Directory Privileged User Account.

Since we've been silently working on this 2006, we've a head start of about a decade. Over the last few months, we've seen several prominent vendors finally realize the importance of doing so, and we've seen them share guidance to this subject.

Unfortunately, just about every piece of advice out there, whether it be from prominent cyber security experts or billion dollar cyber security companies, on how to actually correctly audit privileged access in Active Directory, is dangerously inaccurate.





Thought Leadership

There's an old saying - "Actions Speak Louder Than Words." While there's no dearth of talk by so many big names out there on how to improve cyber security, identify privileged users etc., the key to actually (demonstrably and provably) enhancing cyber security lies in actually helping organizations do so, and we've been silently at work for a decade to help organizations do so.

So, in days to come, right here on this blog, I'm going to (hopefully for one last time), share exactly how organizations worldwide can today accurately and efficiently identify privileged access in their foundational Active Directory deployments worldwide.


In doing so, we will yet again demonstrate Thought Leadership in the Cyber Security space. By the way, this is neither about us, nor about pride. I've already said I'm just a nobody (, whose work possibly impacts everybody.) This is about a desire to help.

So, that post should be out right here on this blog next week, possibly as early as Monday morning.

Best wishes,
Sanjay