Given what it is I do, I don't squander a minute of precious time, unless something is very important, and this is very important.
Let me explain why this is so alarming, concerning and so important to cyber security, and why at many organizations (e.g. U.S. Govt., Paramount Defenses etc.), this could've either possibly resulted in, or in itself, be considered a cyber security breach.
Disclaimer: I'm not making any value judgment about Lenovo ; I'm merely basing this on what's already been said.
As you know, Microsoft's been brazenly leaving billions of people and thousands of organizations worldwide with no real choice but to upgrade to their latest operating system, Windows 10, which albeit is far from perfect, is much better than Windows Vista, Windows 8 etc., even though Windows 10's default settings could be considered an egregious affront to Privacy.
Consequently, at Paramount Defenses, we too felt that perhaps it was time to consider moving on to Windows 10, so we too figured we'd refresh our workforce's PCs. Now, of the major choices available from amongst several reputable PC vendors out there, Microsoft's Surface was one of the top trustworthy contenders, considering that the entirety of the hardware and software was from the same vendor (, and one that was decently trustworthy (considering that most of the world is running their operating system,)) and that there seemed to be no* pre-installed drivers or software that may have been written in China, Russia etc.
Side-note: Based on information available in the public domain, in all likelihood, software written in / maintained from within Russia, may still likely be running as System on Domain Controllers within the U.S. Government.
In particular, regardless of its respected heritage, for us, Lenovo wasn't an option, since it is partly owned by the Chinese Govt.
So we decided to consider evaluating Microsoft Surface devices and thus purchased a couple of brand-new Microsoft Surface devices from our local Microsoft Store for an initial PoC, and I decided to personally test-drive one of them -
 |
| Microsoft Surface |
The very first thing we did after unsealing them, walking through the initial setup and locking down Windows 10's unacceptable default privacy settings, was to connect it to the Internet over a secure channel, and perform a Windows Update.
I should mention that there was no other device attached to this Microsoft Surface, except for a Microsoft Signature Type Cover, and in particular there were no mice of any kind, attached to this new Microsoft surface device, whether via USB or Bluetooth.
Now, you're not going to believe what happened within minutes of having clicked the Check for Updates button!
Windows Update Downloaded and Installed an Untrusted
Self-Signed Lenovo Device Driver on Microsoft Surface! -
Within minutes, Windows Update automatically downloaded and had installed, amongst other packages (notably Surface Firmware,) an untrusted self-signed Kernel-mode device-driver, purportedly Lenovo - Keyboard, Other hardware - Lenovo Optical Mouse (HID), on this brand-new Microsoft Surface device, i.e. one signed with an untrusted WDK Test Certificate!
Here's a snapshot of Windows Update indicating that it had successfully downloaded and installed a Lenovo driver on this Surface device, and it specifically states "Lenovo - Keyboard, Other hardware - Lenovo Optical Mouse (HID)" -
We couldn't quite believe this.
How could this be possible? i.e. how could a Lenovo driver have been installed on a Microsoft Surface device?
So we checked the Windows Update Log, and sure enough, as seen in the snapshot below, the Windows Update Log too confirmed that Windows Update had just downloaded and installed a Lenovo driver -
We wondered if there might have been any Lenovo hardware components installed on the Surface so we checked the Device Manager, and we could not find a single device that seemed to indicate the presence of any Lenovo hardware. (Later, we even took it back to the Microsoft Store, and their skilled tech personnel confirmed the same finding i.e. no Lenovo hardware on it.)
Specifically, as you can see below, we again checked the Device Manager, this time to see if it might indicate the presence of any Lenovo HID, such as a Lenovo Optical Mouse, and as you can see in the snapshot below, the only two Mice and other pointing devices installed on the system were from Microsoft - i.e. no Lenovo mouse presence indicated by Device Manager -
Next, we performed a keyword search of the Registry, and came across a suspicious Driver Package, as seen below -
It seemed suspicious to us because as can be seen in the snapshot above, all of the other legitimate driver package keys in the Registry had (as they should) three child sub-keys i.e. Configurations, Descriptors and Strings, but this specific one only had one subkey titled Properties, and when we tried to open it, we received an Access Denied message!
As you can see above, it seemed to indicate that the provider was Lenovo and that the INF file name was phidmou.inf, and the OEM path was "C:\Windows\SoftwareDistribution\Download\Install", so we looked at the file system but this path didn't seem to exist on the file-system. So we performed a simple file-system search "dir /s phidmou.*" and as seen in the snapshot below, we found one instance of such a file, located in C:\Windows\System32\DriverStore\FileRepository\.
Here's that exact location on the file-system, and as evidenced by the Created date and time for that folder, one can see that this folder (and thus all of its contents), were created on April 01, 2018 at around 1:50 am, which is just around the time the Windows Update log too confirmed that it had installed the Lenovo Driver -
When we opened that location, we found thirteen items, including six drivers -
Next, we checked the Digital Signature on one of the drivers, PELMOUSE.SYS, and we found that it was signed using a self-signed test Windows Driver certificate, i.e. the .sys files were SELF-SIGNED by a WDKTestCert and their digital signatures were NOT OK, in that they terminated in a root certificate that is not trusted by the trust provider -
Finally, when we clicked on the View Certificate button, as can be seen below, we could see that this driver was in fact merely signed by a test certificate, which is only supposed to be used for testing purposes during the creation and development of Kernel-mode drivers. Quoting from Microsoft's documentation on Driver Testing "However, eventually it will become necessary to test-sign your driver during its development, and ultimately release-sign your driver before publishing it to users." -
Clearly, the certificate seen above is NOT one that is intended to be used for release signing, yet, here we have a Kernel-mode driver downloaded by Windows Update and installed on a brand new Microsoft surface, and all its signed by is a test certificate, and who knows who wrote this driver!
Again, per Microsoft's guidelines on driver signing, which can also be found here, "After completing test signing and verifying that the driver is ready for release, the driver package has to be release signed", and AFAIK, release signing not only requires the signer to obtain and use a code-signing certificate from a code-signing CA, it also requires a cross cert issued by Microsoft.
If that is indeed the case, then a Kernel-mode driver that is not signed with a valid code-signing certificate, and one whose digital signature does not contain Microsoft's cross cert, should not even be accepted into the Windows Update catalog.
It is thus hard to believe that a Windows Kernel-Mode Driver that is merely self-signed using a test certificate would even make it into the Windows Update catalog, and further it seems that in this case, not only did it make it in, it was downloaded, and in fact successfully installed onto a system, which clearly seems highly suspicious, and is fact alarming and deeply-concerning!
How could this be? How could Windows Update (a trusted system process of the operating system), which we all (have no choice but to) trust (and have to do so blindly and completely) have itself installed an untrusted self-signed Lenovo driver (i.e. code running in Kernel-Mode) on a Microsoft Surface device?
Frankly, since this piece of software was signed using a self-signed test cert, who's to say this was even a real Lenovo driver? It could very well be some malicious code purporting to be a Lenovo driver. Or, there is also the remote possibility that it could be a legitimate Lenovo driver, that is self-signed, but if that is the case, its installation should not have been allowed to succeed.
Unacceptable and Deeply Concerning
To us, this is unacceptable, alarming and deeply concerning, and here's why.
We just had, on a device we consider trustworthy (, and could possibly have engaged in business on,) procured from a vendor we consider trustworthy (considering that the entire world's cyber security ultimately depends on them), an unknown, unsigned piece of software of Chinese origin that is now running in Kernel-mode, installed on the device, by this device's vendor's (i.e. Microsoft's) own product (Windows operating system's) update program!
We have not had an opportunity to analyze this code, but if it is indeed malicious in any way, in effect, it would've, unbeknownst to us and for no fault of ours, granted System-level control over a trusted device within our perimeter, to some entity in China.
How much damage could that have caused? Well, suffice it to say that, for they who know Windows Security well, if this was indeed malicious, it would've been sufficient to potentially compromise any organization within which this potentially suspect and malicious package may have been auto-installed by Windows update. (I've elaborated a bit on this below.)
In the simplest scenario, if a company's Domain Admins had been using this device, it would've been Game Over right there!
This leads me to the next question - we can't help but wonder how many such identical Surface devices exist out there today, perhaps at 1000s of organizations, on which this suspicious unsigned Lenovo driver may have been downloaded and installed?
This also leads me to another very important question - Just how much trust can we, the world, impose in Windows Update?
In our case, it just so happened to be, that we happened to be in front of this device during this Windows update process, and that's how we noticed this, and by the way, after it was done, it gave the familiar Your device is upto date message.
Speaking which, here's another equally important question - For all organizations that are using Windows Surface, and may be using it for mission-critical or sensitive purposes (e.g. AD administration), what is the guarantee that this won't happen again?
I ask because if you understand cyber security, then you know, that it ONLY takes ONE instance of ONE malicious piece of software to be installed on a system, to compromise the security of that system, and if that system was a highly-trusted internal system (e.g. that machine's domain computer account had the "Trusted for Unconstrained Delegation" bit set), then this could very likely also aid perpetrators in ultimately gaining complete command and control of the entire IT infrastructure. As I have already alluded to above, if by chance the target/compromised computer was one that was being used by an Active Directory Privileged User, then, it would be tantamount to Game Over right then and there!
Think about it - this could have happened at any organization, from say the U.S. Government to the British Government, or from say a Goldman Sachs to a Palantir, or say from a stock-exchange to an airline, or say at a clandestine national security agency to say at a nuclear reactor, or even Microsoft itself. In short, for absolutely no fault of theirs, an organization could potentially have been breached by a likely malicious piece of software that the operating system's own update utility had downloaded and installed on the System, and in 99% of situations, because hardly anyone checks what gets installed by Windows Update (now that we have to download and install a whopping 600MB patch every Tuesday), this would likely have gone unnoticed!
Again, to be perfectly clear, I'm not saying that a provably malicious piece of software was in fact downloaded and installed on a Microsoft Surface device by Windows Update. What I'm saying is that a highly suspicious piece of software, one that was built and intended to run in Kernel-mode and yet was merely signed with a test certificate, somehow was automatically downloaded and installed on a Microsoft Surface device, and that to us is deeply concerning, because in essence, if this could happen, then even at organizations that may be spending millions on cyber security, a single such piece of software quietly making its way in through such a trusted channel, could possibly instantly render their entire multi-million dollar cyber security apparatus useless, and jeopardize the security of the entire organization, and this could happen at thousands of organizations worldwide.
With full respect to Microsoft and Mr. Nadella, this is deeply concerning and unacceptable, and I'd like some assurance, as I'm sure would 1000s of other CEOs and CISOs, that this will never happen again, on any Surface device, in any organization.
In our case, this was very important, because had we put that brand new Surface device that we procured from none other than the Microsoft Store, into operation (even it we had re-imaged it with an ultra-secure locked-down internal image), from minute one, post the initial Windows update, we would likely have had a potentially compromised device running within our internal network, and it could perhaps have led to us being breached.
If I Were Microsoft, I'd Send a Plane
Dear Microsoft, we immediately quarantined that Microsoft Surface device, and we have it in our possession.
If I were you, I'd send a plane to get it picked up ASAP, so you can thoroughly investigate every little aspect of this to figure out how this possibly happened, and get to the bottom of it! (Petty process note: The Microsoft Store let us keep the device for a bit longer, but will not let us return the device past June 24, and the only reason we've kept it, is in case you'd want to analyze it.)
Here's why. At the very least, if I were still at Microsoft, and in charge of Cyber Security -
- I'd want to know how an untrusted Kernel-mode device driver made it into the Windows Catalog
- I'd want to know why a Microsoft Surface device downloaded a purportedly Lenovo driver
- I'd want to know how Windows 10 permitted and in fact itself installed an untrusted driver
- I'd want to know exactly which SKUs of Microsoft Surface this may have happened on
- I'd want to know exactly how many such Microsoft Surface devices out there may have downloaded this package
Further, and as such, considering that Microsoft Corp itself may easily have thousands of Surface devices being used within Microsoft itself, if I were still with Microsoft CorpSec, I'd certainly want to know how many of their own Surface devices may have automatically downloaded and installed this highly suspicious piece of untrusted self-signed software.
In short, Microsoft, if you care as deeply about cyber security as you say you do, and by that I'm referring to what Mr. Nadella, the CEO of Microsoft, recently said (see video below: 0:40 - 0:44) and I quote "we spend over a billion dollars of R&D each year, in building security into our mainstream products", then you'll want to get to the bottom of this, because other than the Cloud, what else could be a more mainstream product for Microsoft today than, Microsoft Windows and Microsoft Surface ?! -
Also, speaking of Microsoft's ecosystem, it indeed is time to help safeguard Microsoft's global ecosystem. (But I digress,)
In Conclusion
Folks, the only reason I decided to publicly share this is because I care deeply about cyber security, and I believe that this could potentially have impacted the foundational cyber security of any, and potentially, of thousands of organizations worldwide.
Hopefully, as you'll agree, a trusted component (i.e. Windows Update) of an operating system that virtually the whole world will soon be running on (i.e. Windows 10), should not be downloading and installing a piece of software that runs in Kernel-mode, when that piece of software isn't even digitally signed by a valid digital certificate, because if that piece of software happened to be malicious, then in doing so, it could likely, automatically, and for no fault of its users, instantly compromise the cyber security of possibly thousands of organizations worldwide. This is really as simple, as fundamental and as concerning, as that.
All in all, the Microsoft Surface is an incredible device, and because, like Apple's computers, the entire hardware and software is in control of a single vendor, Microsoft has a huge opportunity to deliver a trustworthy computing device to the world, and we'd love to embrace it. Thus, it is vital for Microsoft to ensure that its other components (e.g. Update) do not let the security of its mainstream products down, because per the Principle of Weakest Link, "a system is only as secure as is its weakest link."
By the way, I happen to be former Microsoft Program Manager for Active Directory Security, and I care deeply for Microsoft.
For those may not know what Active Directory Security is (i.e. most CEOs, a few CISOs, and most employees and citizens,) suffice it to say that global security may depend on Active Directory Security, and thus may be a matter of paramount defenses.
Most respectfully,
Sanjay
PS: Full Disclosure: I had also immediately brought this matter to the attention of the Microsoft Store. They escalated it to Tier-3 support (based out of New Delhi, India), who then asked me to use the Windows Feedback utility to share the relevant evidence with Microsoft, which I immediately and dutifully did, but/and I never heard back from anyone at Microsoft in this regard again.
PS2: Another small request to Microsoft - Dear Microsoft, while at it, could you please also educate your global customer base about the paramount importance of Active Directory Effective Permissions, which is the ONE capability without which not a single object in any Active Directory deployment can be adequately secured! Considering that Active Directory is the foundation of cyber security of over 85% of all organizations worldwide, this is important. Over the last few years, we've had almost 10,000 organizations from 150+ countries knock at our doors, and virtually none of them seem to know this most basic and cardinal fact of Windows Security. I couldn't begin to tell you how shocking it is for us to learn that most Domain Admins and many CISOs out there don't have a clue. Can you imagine just how insecure and vulnerable an organization whose Domain Admins don't even know what Active Directory Effective Permissions are, let alone possessing this paramount capability, could be today?
