Jump to content
BGags

Windows Update Delivery Primer v2017

Recommended Posts

This document is meant as a successor to and replacement for "How I got to 95% patch efficacy in Eight Easy(?) Steps", mainly because about half of it is obsolete, but also because "efficacy" wasn't used correctly in that context and I know better now. Mostly.

 

My hope is to make this more of a living document. I'd like to update this original front post as changes to it are needed rather than force the reader to descend into seven pages of comments to find the most up-to-date solution for any given problem.

 

DISCLAIMER: I'm still running the "Classic" Patch Manager, based on the advice of so many other Geeks who advise that the LT11 Patch Manager is still problematic as of this writing. Either way, this discussion focuses mostly on the functionality of Windows itself relative to how LabTech delivers updates, so a large portion of this document should remain fairly universal.

 

OTHER DISCLAIMER: I am not responsible for what happens when you follow my advice. Nope.

 

Finally, these are mainly descriptions of what I've found to be useful coupled with a discussion about how Windows Updates actually work. There isn't much meat here on Windows 10, mainly because right now it patches pretty good, and the problem of delivering whole operating system upgrades is slightly beyond the scope of this discussion (for now).

 

Let's dive in.

 

 

I. Master the Windows Update Agent Version.

 

Put simply, the Windows Update Agent is the collection of code that uses Microsoft Update to check the patch status of the OS versus available updates from Microsoft, as well as drive update scheduling and UI behavior. Between the time when the Win7 and Win8/8.1 kernels were released and now, the WUA has changed tremendously to the point where the original WUA code shipped with the OS will no longer effectively patch a system in a reasonable amount of time if you try to use it today. You may find that you get no useful data, a failed hotfix inventory, a pegged CPU, and/or other unpleasant behavior.

 

This means it's imperative that you track the WUA of your endpoints and make sure they're up-to-date, especially when you onboard new clients that come from questionable patching practices. The most common way to do this is simply to set up an EDF for each computer and have a script populate it. Automate 11's Patch Manager has this facility built-in. Once you have that information, you can take action on bringing the Windows Update Agent up-to-date for the various operating systems in question. The WUA version of any Windows system after XP/2003 can be found by checking the Product Version of the wuaueng.dll file in System32. This can be accomplished with a one-line PowerShell command:

 

(Get-Item 'C:\Windows\system32\wuaueng.dll').VersionInfo.ProductVersion

 

This gets you a collection of WUAs that correlate to an operating system. Now what? There are a million ways to maintain your WUA versions. I have it on good authority that Cubert's Patch Remedy plugin is pretty sweet. That said, here's how I do it: Since the actual object of the game is to get your targets to the point where they're reliably checking for and delivering updates, I've found that it's not necessarily important to get any given endpoint up to the very latest WUA to get them to patch. What I do is to figure out the minimum WUA version needed for each OS, get my target to that point, and then let the built-in patching mechanisms take over and figure out the rest for itself.

 

To do this, I've set up multiple Properties (Dashboard -> Config -> Configurations -> Properties) to track the minimum WUA required for reliable update delivery, so I have a single place to update this value for any script that uses it (did you know that Properties in Automate are extensible? Holy cow!). Then, I have a WUA Validator script that runs before any patching-related action which checks the endpoint's WUA version against the relative hard-set minimum property, and sets a variable to PASS / FAIL on whether or not a patching agent may go forward. Any script that wants to patch calls the Validator, checks the variable, and if it gets a FAIL, it doesn't perform the patching command.

 

I've turned off my built-in Hotfix Inventory schedule for desktops and laptops. I throw scripts at them instead. And those scripts run the WUA Validator first. This has been alarmingly successful for me.

 

So how the heck to you validate a damn version number? It's not like you can use a math compare, since versions are strings with multiple dots in them. If you're at 7.6.7601.19161 (horrid), and you wanna be at 7.6.7601.23453 (phew!), you have to basically do an alpha sort with numbers. Here's a neat SQL trick I'll drop for you:

 

SELECT * FROM (SELECT '@WUA_Minimum@' AS WUA UNION SELECT '@WUA_Current@' AS WUA) AS `WUAs` ORDER BY WUA LIMIT 1

 

And here it is in a script step:

 

PYiuR4C.png

 

What's going on here? You're basically making a tiny table with one column and two rows, sorting it, and then taking the first row. This will always give you the "lower" of the two versions we're comparing, which we can then check against our minimum version. If our WUA is in compliance, the endpoint's current WUA should be at or above minimum, so it sorts above the minimum, and our variable check then succeeds and we issue a PASS. If not, screw you, endpoint. Your FAIL means I'm sending you to the WUA Remediation Groups for a 12 Step script.

 

"Hi, my name is DCFS1... and I don't know how to check for updates."

 

"HI DCFS1!!!!"

 

"Thanks, everyone. My last MSP, it's like… it's like they didn't even know how vulnerable I was. I felt so insecure. And now that I can check for updates, well, I KNOW I'm insecure! But I'm getting better. It feels… it feels good. I don't feel so vulnerable anymore."

 

Of course, all these fine updates require a restart, so you'll want to factor that into your scheduling. Make sure you re-populate your EDFs after you restart your targets, as the WUA version won't update until after the restart.

 

A NOTE ABOUT BANDWIDTH

 

With the July 2016 WUA updates for the above operating systems, Microsoft fundamentally changed the way the WUA checks for updates from Microsoft to make them a lot faster. The WUA used to make lots and lots of queries to Windows Update over and over again, encrypting and decrypting everything. This took a long time, and certain versions of the WUA (screw you forever, KB3083324) could even peg your CPU while doing it. Now, the most recent WUA versions download the entire OS update manifest from Microsoft in one big gulp and do the comparisons locally. This is faster and more efficient, but if you've triggered all your Hotfix Inventories during the day for all systems at once in that one client location that's still plugging along with its stupid bonded T1, they'll probably notice a bit of a bandwidth hit.

 

I also try to do WOL wherever possible, do the majority of my hotfix inventories in the evenings, and do my hotfix inventories for laptops additionally during the day around lunchtime because you know those suckers are going straight back into users' bags at the end of the day.

 

 

II. Master Your Update Scheduling.

 

The thing about the Windows Update process is that it can literally only do one thing at a time. If you're trying to deliver an update but then you suddenly shoot a Hotfix Inventory command at your agent, you'll break the process. If you're doing a Hotfix Inventory but then also do an Update Config, you'll break the inventory because, believe it or not, the Update Config command also deliberately stops and restarts the Windows Update service.

 

Ugh. What the hell, Automate. Why would you do that?

 

This tells us that we have to be careful how we schedule things that are Windows Update-related. Make sure your Resend Hotfix Inventory templates are scheduled away from your Update Config templates. Make sure any update delivery is scheduled away from both of the above. Make sure the Windows Update service isn't being restarted by some other process or software (I'm looking at YOU, LogMeIn).

 

Have you ever looked at your patch command history to find a patch that failed with error code 0x8024001E? That's the code you get when the Windows Update service gets stopped while it's attempting to perform a task. There are other causes of that error, but the biggest one (assuming your WUA is up-to-date) is a scheduling conflict.

 

You've also got to look at your templates and schedules. Did you know that any inventory command run by a schedule doesn't actually show up in your Commands history? The inventories run, but they're run by the agents directly without being queued and issued from the database agent first. Raise your hand if you found that out the hard way (hi).

 

 

III. Tracking and Fixing Windows Update Problems

 

Stuff breaks. With Windows Update, it might be a temporary thing (interrupted download, cosmic rays, the usual), or it might be something that requires a little extra abuse. Ideally, you want an automatic solution that

 

1. Fixes stuff automatically where automatic fixing is possible

2. Recognizes intermittent failures and allows endpoints to "try again"

3. Informs the admin when something's wrong that can't be automatically fixed.

 

Because every version of Windows since Vista still uses the Side-by-Side mechanisms (that's what WinSxS stands for, btw) for installing and linking updates, these steps conveniently apply to every supported version of Windows as of this writing. As with so many things with Automate, there are a million ways to do it.

 

I'm a personal fan of the nightly health check. I wrote an offline script that runs through every eligible agent, looking for patterns of failures. Some of the patterns include:

 

1. More than three patch install failures in a row with no success

2. More than three hotfix inventory failures in a row with no success

3. Excessive patching (if patches are detected as missing and re-delivered multiple times per week, something's up)

4. Empty patching history

 

When my health check script finds a problematic pattern, it makes a couple decisions. First, it checks to see if a Windows Update Repair has already been run for that endpoint by looking for an indicative alert. If that alert isn't already present, it sets an EDF that a WU Repair is needed. That EDF does two things. First, it exempts that endpoint from further attempts at patching through various checks, searches, and other conditionals. Second, it adds the endpoint to a group that fires off the Windows Update Repair script on a fairly assertive schedule. After the WU Repair itself gets run, it sets an Informational Alert on that agent indicating a successful run of the repair, and then the script clears its EDF.

 

The Alert alters the behavior of the nightly health check because if it finds Windows Update problems and sees that a Repair has already been run, then throw a ticket because clearly that crap isn't fixing itself.

 

What's in the Windows Update Repair script, you ask? Oh, the usual:

 

1. Stop the Windows Update service and nuke the subfolder contents of %windir%\SoftwareDistribution\

2. Stop the Cryptographic Services service and nuke %windir%\System32\catroot2\

3. Reset the winhttp proxy (can't hurt, right?)

4. Check for and remove any WSUS policies applied (various registry entries)

 

If you feel like getting really clever, you could even try fixing or alerting on the type of patching error that gets spit out in your patching history text by correlating it with the type of patching error you get. Or, you can just look at this handy reference document and see what applies to you:

 

A List of Windows Update error Codes

 

IV. Make Sure You're Patching More Than Just The Operating System.

 

This is a classic from the previous article, but it still applies:

 

You know when you go into the Control Panel and you get that bit on the screen that says "You receive updates: ...For Windows and other products from Microsoft Update"? Did you know you can set that programmatically?

 

It's ugly, but here's a line of PowerShell that you can throw into a script to do this for any supported Microsoft OS:

 

$ServiceManager = New-Object -ComObject "Microsoft.Update.ServiceManager"; $ServiceManager.ClientApplicationID = "My App"; $ServiceManager.AddService2( "7971f918-a847-4430-9279-4a52d1efe18d",7,"")

 

 

Hey, don't blame me, I didn't write the OS.

 

 

V. Clear The "Pushed" Update Status ("Classic" Patch Manager only)

 

Another classic from the previous article! The "Pushed" status for a patch means that LabTech has tried twice to install the patch and it didn't succeed, so it's not going to try to install it anymore. That's literally what it means, straight from the developer's mouth.

 

Are you kidding? LabTech! Don't give up! You can do it!

 

That's the behavior. And for my environment, it doesn't work. It's not effective, it gets in the way, and it lowers my patch efficacy numbers. You can't actually "get rid of" this functionality, but you can certainly reset the Pushed flags. I do it directly with a SQL query every night with a stored procedure. Here's the SQL:

 

UPDATE hotfix SET pushed='0' WHERE installed=0 AND pushed=1 AND approved=1

 

 

Before you do this, if you want to see how many patches you're missing due to the Pushed flag, here's the SQL to get that count:

 

SELECT COUNT(hotfix.`HotFixID`) FROM hotfix WHERE installed=0 AND pushed=1 AND approved=1

 

 

VI. Patch Early, Patch Often.

 

Once you have your Windows Update agents tamed and responsive, there's usually no reason not to deliver updates during the day. You can leverage LabTech's location caches to mitigate bandwidth concerns, and most updates can be delivered without user interruption. I have EDFs carved into Client and Location tabs that make exceptions for our daytime patching scripts for clients paranoid about either functionality or bandwidth. But for the most part, I can run a Hotfix Inventory and/or a patching script against most of my clients several times a day and nobody ever says "boo". And you might as well do it, because you know that at 5pm or even earlier, that laptop is going back in the bag, that kiosk machine with the Wi-Fi card isn't going to respond to WOL, and you won't be able to update a darn thing.

 

One exception to this rule, however, includes the Malicious Software Removal Tool which, if you install this sucker during the day, will immediately run its scan. Depending on available resources, this might produce unwanted (read: "slow and sketchy") behavior on an endpoint. YMMV, the usual disclaimers apply, don't say BGags didn't warn you.

 

 

VII. Don't Rest On Your Laurels.

 

Okay, you did all these things, you've got your patching percentage up pretty good, you've got scripts that work and a schedule that seems to work flawlessly. Good, congratulations, you should be proud of the work you did.

 

Please get used to the idea that this is all going to unravel in six months.

 

Microsoft is known for pulling the rug out from under your updates and changing their behavior with little warning. Remember October 2016 when most updates went cumulative and you had to check for a veritable nest of KB articles installed on any given endpoint to see if it was protected against WannaCry? Did you actually want to cry? Patching will never be a "set it and forget it" game, which is why Plugins are good: You don't have to do the work.

 

 

Ok, that's all I got.

Edited by Guest
  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Let me get back from Automation Nation first. There's a few different scripts. I'll share the Windows Update Repair script, my WUA validator script, and probably update the original post to include the current minimum WUA version numbers that I'm using as reference.

Share this post


Link to post
Share on other sites

Thanks for the scripts szkoda. The one that deletes Software Distribution works like a charm.

Any word on the health check scripts? Those would be awesome to run nightly. The rest of the info has been amazingly helpful so far. Thanks for posting.

Share this post


Link to post
Share on other sites

BGags, thanks for that opening WUA query. Among other things, I'm using it with a scheduled script to populate an EDF for each Windows agent so I can also check the WUA on a system at a glance.

Share this post


Link to post
Share on other sites

I'm trying to figure out the best way to run the powershell script to enable the patching of other Microsoft products. I know the basic scripting concept, but I'm trying to figure out how to validate whether that's already been done on a machine. Any suggestions on that specifically? Or will it not hurt to run that powershell script on top of itself over and over again?

Share this post


Link to post
Share on other sites

I'm a personal fan of the nightly health check. I wrote an offline script that runs through every eligible agent, looking for patterns of failures. Some of the patterns include:

 

1. More than three patch install failures in a row with no success

2. More than three hotfix inventory failures in a row with no success

3. Excessive patching (if patches are detected as missing and re-delivered multiple times per week, something's up)

4. Empty patching history

 

When my health check script finds a problematic pattern, it makes a couple decisions. First, it checks to see if a Windows Update Repair has already been run for that endpoint by looking for an indicative alert. If that alert isn't already present, it sets an EDF that a WU Repair is needed. That EDF does two things. First, it exempts that endpoint from further attempts at patching through various checks, searches, and other conditionals. Second, it adds the endpoint to a group that fires off the Windows Update Repair script on a fairly assertive schedule. After the WU Repair itself gets run, it sets an Informational Alert on that agent indicating a successful run of the repair, and then the script clears its EDF.

 

BGags,

 

Would you consider posting that Nightly Health Check script? I think a lot of us would find it useful.

 

I too have a script much like yours to re-initialize Windows Update, but linking it to a regular check like the above would be a really cool thing. Note: I'm using the updated Patch Manager, it seems to work decently now (though I agree, the first releases were a disaster). I'm hovering regularly between 95-96%, which should increase next month when they add some changes to take care of superseded patches and how they affect compliance.

Share this post


Link to post
Share on other sites

One final post here before letting other people do some talking (yes, I know, put down the coffee cup LoneWolf), I'm going to cross-post it here because it seems like a far better place to start down a new path and leave the old patch thread:

 

As we all know that Windows 10 patching is hit-or-miss in the current LabTech configuration (largely due to Microsoft restructuring), has anyone created scripting that does any of the following for better control?

 

-Use the WUSHOWHIDE.CAB setup from Microsoft to turn off (and on) Automatic Updates so we can control patch windows?

-Set Internet connections to "Metered" to disable updating (doable via a registry flag, I believe)?

-Set Windows Update policies (esp. on Anniversary and Creators) to defer feature upgrades and to delay immediate patching? (to prevent getting patched with something buggy that might get rolled back by Microsoft in the first 48-72 hours of release)

 

Very interested in what other people are doing (if anything) and to see any scriptwork associated. I think the next battle will be controlling the deployment...AND the non-deployment of Windows 10 patching, what with its "Thou SHALT receive updates whether thou likest or not" approach.

Share this post


Link to post
Share on other sites

A couple things about the health check script:

 

1. I am working on one meant for public consumption. The vast majority of folks who will want to use it are all on the "new" patch manager (I'm not), and so my values for searching for approved hotfixes are going to be different than others, so I'm trying to parameterize that.

 

2. I'm also working through something right now where I'm finding that superseded hotfixes aren't being properly removed from Automate. I've got a number of systems that report anywhere from 1 to 10 updates missing, but when those updates are attempted again, the response is "No Updates to Install" because they're either already installed or they've been superseded by something newer. I'm adding this to my Health Check routines now and after it's more or less foolproof, I'll write it up in this article.

 

3. I'd like you to reflect in point VII above: Don't rest on your laurels. I'm trying to develop a new Health Check script whose parameters are more flexible, because what works for me this month might need to be seriously adjusted next month depending on what comes down the pike.

 

Thanks for your patience and support.

Share this post


Link to post
Share on other sites

I'll second the last point in your first post Bgags - I'd got my patching up to around 90% around 6 months ago and now it's all fallen away :-(

 

I'm also seeing the superseeded updates not being removed and also getting lots of 'No updates to install' messages when WU on the workstation AND LT are saying that there are updates to install. I've raised a ticket with LT for the 'no updates' message.

 

I'd be interested to see the scripts you reference above including the minimum WUA versions

Share this post


Link to post
Share on other sites

LT support have provided me with this SQL which improved my patching reports massively. If you are using the new patch manager use "approved=2"

CREATE TEMPORARY TABLE TempHotfix SELECT * FROM hotfix;
DELETE FROM hotfix WHERE installed=0 AND approved=1 AND Last_date < (SELECT MAX(last_date) FROM Temphotfix WHERE computerid=hotfix.computerid);
DROP TEMPORARY TABLE TempHotfix;

I'm told this is removing superseeded patches. I assume this needs to be regularly run. Whilst this would be useful to have built-into LT itself, I may just create a script to do it daily.

Greg

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...