Jump to content

BGags

Members
  • Content Count

    85
  • Joined

  • Last visited

Community Reputation

2 Neutral

My Information

  • Location
    Easthampton, MA
  • Agent Count
    4000 - 6000 Agents

Converted

  • INTERESTS
    SQL, scripting, long walks on the beach.
  • OCCUPATION
    Automation Coordinator

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. A couple things about the health check script: 1. I am working on one meant for public consumption. The vast majority of folks who will want to use it are all on the "new" patch manager (I'm not), and so my values for searching for approved hotfixes are going to be different than others, so I'm trying to parameterize that. 2. I'm also working through something right now where I'm finding that superseded hotfixes aren't being properly removed from Automate. I've got a number of systems that report anywhere from 1 to 10 updates missing, but when those updates are attempted again, the response is "No Updates to Install" because they're either already installed or they've been superseded by something newer. I'm adding this to my Health Check routines now and after it's more or less foolproof, I'll write it up in this article. 3. I'd like you to reflect in point VII above: Don't rest on your laurels. I'm trying to develop a new Health Check script whose parameters are more flexible, because what works for me this month might need to be seriously adjusted next month depending on what comes down the pike. Thanks for your patience and support.
  2. What was helpful for me: I also made a dataview for the EDF. I love dataviews!
  3. Let me get back from Automation Nation first. There's a few different scripts. I'll share the Windows Update Repair script, my WUA validator script, and probably update the original post to include the current minimum WUA version numbers that I'm using as reference.
  4. This document is meant as a successor to and replacement for "How I got to 95% patch efficacy in Eight Easy(?) Steps", mainly because about half of it is obsolete, but also because "efficacy" wasn't used correctly in that context and I know better now. Mostly. My hope is to make this more of a living document. I'd like to update this original front post as changes to it are needed rather than force the reader to descend into seven pages of comments to find the most up-to-date solution for any given problem. DISCLAIMER: I'm still running the "Classic" Patch Manager, based on the advice of so many other Geeks who advise that the LT11 Patch Manager is still problematic as of this writing. Either way, this discussion focuses mostly on the functionality of Windows itself relative to how LabTech delivers updates, so a large portion of this document should remain fairly universal. OTHER DISCLAIMER: I am not responsible for what happens when you follow my advice. Nope. Finally, these are mainly descriptions of what I've found to be useful coupled with a discussion about how Windows Updates actually work. There isn't much meat here on Windows 10, mainly because right now it patches pretty good, and the problem of delivering whole operating system upgrades is slightly beyond the scope of this discussion (for now). Let's dive in. I. Master the Windows Update Agent Version. Put simply, the Windows Update Agent is the collection of code that uses Microsoft Update to check the patch status of the OS versus available updates from Microsoft, as well as drive update scheduling and UI behavior. Between the time when the Win7 and Win8/8.1 kernels were released and now, the WUA has changed tremendously to the point where the original WUA code shipped with the OS will no longer effectively patch a system in a reasonable amount of time if you try to use it today. You may find that you get no useful data, a failed hotfix inventory, a pegged CPU, and/or other unpleasant behavior. This means it's imperative that you track the WUA of your endpoints and make sure they're up-to-date, especially when you onboard new clients that come from questionable patching practices. The most common way to do this is simply to set up an EDF for each computer and have a script populate it. Automate 11's Patch Manager has this facility built-in. Once you have that information, you can take action on bringing the Windows Update Agent up-to-date for the various operating systems in question. The WUA version of any Windows system after XP/2003 can be found by checking the Product Version of the wuaueng.dll file in System32. This can be accomplished with a one-line PowerShell command: (Get-Item 'C:\Windows\system32\wuaueng.dll').VersionInfo.ProductVersion This gets you a collection of WUAs that correlate to an operating system. Now what? There are a million ways to maintain your WUA versions. I have it on good authority that Cubert's Patch Remedy plugin is pretty sweet. That said, here's how I do it: Since the actual object of the game is to get your targets to the point where they're reliably checking for and delivering updates, I've found that it's not necessarily important to get any given endpoint up to the very latest WUA to get them to patch. What I do is to figure out the minimum WUA version needed for each OS, get my target to that point, and then let the built-in patching mechanisms take over and figure out the rest for itself. To do this, I've set up multiple Properties (Dashboard -> Config -> Configurations -> Properties) to track the minimum WUA required for reliable update delivery, so I have a single place to update this value for any script that uses it (did you know that Properties in Automate are extensible? Holy cow!). Then, I have a WUA Validator script that runs before any patching-related action which checks the endpoint's WUA version against the relative hard-set minimum property, and sets a variable to PASS / FAIL on whether or not a patching agent may go forward. Any script that wants to patch calls the Validator, checks the variable, and if it gets a FAIL, it doesn't perform the patching command. I've turned off my built-in Hotfix Inventory schedule for desktops and laptops. I throw scripts at them instead. And those scripts run the WUA Validator first. This has been alarmingly successful for me. So how the heck to you validate a damn version number? It's not like you can use a math compare, since versions are strings with multiple dots in them. If you're at 7.6.7601.19161 (horrid), and you wanna be at 7.6.7601.23453 (phew!), you have to basically do an alpha sort with numbers. Here's a neat SQL trick I'll drop for you: SELECT * FROM (SELECT '@WUA_Minimum@' AS WUA UNION SELECT '@WUA_Current@' AS WUA) AS `WUAs` ORDER BY WUA LIMIT 1 And here it is in a script step: What's going on here? You're basically making a tiny table with one column and two rows, sorting it, and then taking the first row. This will always give you the "lower" of the two versions we're comparing, which we can then check against our minimum version. If our WUA is in compliance, the endpoint's current WUA should be at or above minimum, so it sorts above the minimum, and our variable check then succeeds and we issue a PASS. If not, screw you, endpoint. Your FAIL means I'm sending you to the WUA Remediation Groups for a 12 Step script. Of course, all these fine updates require a restart, so you'll want to factor that into your scheduling. Make sure you re-populate your EDFs after you restart your targets, as the WUA version won't update until after the restart. A NOTE ABOUT BANDWIDTH With the July 2016 WUA updates for the above operating systems, Microsoft fundamentally changed the way the WUA checks for updates from Microsoft to make them a lot faster. The WUA used to make lots and lots of queries to Windows Update over and over again, encrypting and decrypting everything. This took a long time, and certain versions of the WUA (screw you forever, KB3083324) could even peg your CPU while doing it. Now, the most recent WUA versions download the entire OS update manifest from Microsoft in one big gulp and do the comparisons locally. This is faster and more efficient, but if you've triggered all your Hotfix Inventories during the day for all systems at once in that one client location that's still plugging along with its stupid bonded T1, they'll probably notice a bit of a bandwidth hit. I also try to do WOL wherever possible, do the majority of my hotfix inventories in the evenings, and do my hotfix inventories for laptops additionally during the day around lunchtime because you know those suckers are going straight back into users' bags at the end of the day. II. Master Your Update Scheduling. The thing about the Windows Update process is that it can literally only do one thing at a time. If you're trying to deliver an update but then you suddenly shoot a Hotfix Inventory command at your agent, you'll break the process. If you're doing a Hotfix Inventory but then also do an Update Config, you'll break the inventory because, believe it or not, the Update Config command also deliberately stops and restarts the Windows Update service. Ugh. What the hell, Automate. Why would you do that? This tells us that we have to be careful how we schedule things that are Windows Update-related. Make sure your Resend Hotfix Inventory templates are scheduled away from your Update Config templates. Make sure any update delivery is scheduled away from both of the above. Make sure the Windows Update service isn't being restarted by some other process or software (I'm looking at YOU, LogMeIn). Have you ever looked at your patch command history to find a patch that failed with error code 0x8024001E? That's the code you get when the Windows Update service gets stopped while it's attempting to perform a task. There are other causes of that error, but the biggest one (assuming your WUA is up-to-date) is a scheduling conflict. You've also got to look at your templates and schedules. Did you know that any inventory command run by a schedule doesn't actually show up in your Commands history? The inventories run, but they're run by the agents directly without being queued and issued from the database agent first. Raise your hand if you found that out the hard way (hi). III. Tracking and Fixing Windows Update Problems Stuff breaks. With Windows Update, it might be a temporary thing (interrupted download, cosmic rays, the usual), or it might be something that requires a little extra abuse. Ideally, you want an automatic solution that 1. Fixes stuff automatically where automatic fixing is possible 2. Recognizes intermittent failures and allows endpoints to "try again" 3. Informs the admin when something's wrong that can't be automatically fixed. Because every version of Windows since Vista still uses the Side-by-Side mechanisms (that's what WinSxS stands for, btw) for installing and linking updates, these steps conveniently apply to every supported version of Windows as of this writing. As with so many things with Automate, there are a million ways to do it. I'm a personal fan of the nightly health check. I wrote an offline script that runs through every eligible agent, looking for patterns of failures. Some of the patterns include: 1. More than three patch install failures in a row with no success 2. More than three hotfix inventory failures in a row with no success 3. Excessive patching (if patches are detected as missing and re-delivered multiple times per week, something's up) 4. Empty patching history When my health check script finds a problematic pattern, it makes a couple decisions. First, it checks to see if a Windows Update Repair has already been run for that endpoint by looking for an indicative alert. If that alert isn't already present, it sets an EDF that a WU Repair is needed. That EDF does two things. First, it exempts that endpoint from further attempts at patching through various checks, searches, and other conditionals. Second, it adds the endpoint to a group that fires off the Windows Update Repair script on a fairly assertive schedule. After the WU Repair itself gets run, it sets an Informational Alert on that agent indicating a successful run of the repair, and then the script clears its EDF. The Alert alters the behavior of the nightly health check because if it finds Windows Update problems and sees that a Repair has already been run, then throw a ticket because clearly that crap isn't fixing itself. What's in the Windows Update Repair script, you ask? Oh, the usual: 1. Stop the Windows Update service and nuke the subfolder contents of %windir%\SoftwareDistribution\ 2. Stop the Cryptographic Services service and nuke %windir%\System32\catroot2\ 3. Reset the winhttp proxy (can't hurt, right?) 4. Check for and remove any WSUS policies applied (various registry entries) If you feel like getting really clever, you could even try fixing or alerting on the type of patching error that gets spit out in your patching history text by correlating it with the type of patching error you get. Or, you can just look at this handy reference document and see what applies to you: A List of Windows Update error Codes IV. Make Sure You're Patching More Than Just The Operating System. This is a classic from the previous article, but it still applies: You know when you go into the Control Panel and you get that bit on the screen that says "You receive updates: ...For Windows and other products from Microsoft Update"? Did you know you can set that programmatically? It's ugly, but here's a line of PowerShell that you can throw into a script to do this for any supported Microsoft OS: $ServiceManager = New-Object -ComObject "Microsoft.Update.ServiceManager"; $ServiceManager.ClientApplicationID = "My App"; $ServiceManager.AddService2( "7971f918-a847-4430-9279-4a52d1efe18d",7,"") Hey, don't blame me, I didn't write the OS. V. Clear The "Pushed" Update Status ("Classic" Patch Manager only) Another classic from the previous article! The "Pushed" status for a patch means that LabTech has tried twice to install the patch and it didn't succeed, so it's not going to try to install it anymore. That's literally what it means, straight from the developer's mouth. Are you kidding? LabTech! Don't give up! You can do it! That's the behavior. And for my environment, it doesn't work. It's not effective, it gets in the way, and it lowers my patch efficacy numbers. You can't actually "get rid of" this functionality, but you can certainly reset the Pushed flags. I do it directly with a SQL query every night with a stored procedure. Here's the SQL: UPDATE hotfix SET pushed='0' WHERE installed=0 AND pushed=1 AND approved=1 Before you do this, if you want to see how many patches you're missing due to the Pushed flag, here's the SQL to get that count: SELECT COUNT(hotfix.`HotFixID`) FROM hotfix WHERE installed=0 AND pushed=1 AND approved=1 VI. Patch Early, Patch Often. Once you have your Windows Update agents tamed and responsive, there's usually no reason not to deliver updates during the day. You can leverage LabTech's location caches to mitigate bandwidth concerns, and most updates can be delivered without user interruption. I have EDFs carved into Client and Location tabs that make exceptions for our daytime patching scripts for clients paranoid about either functionality or bandwidth. But for the most part, I can run a Hotfix Inventory and/or a patching script against most of my clients several times a day and nobody ever says "boo". And you might as well do it, because you know that at 5pm or even earlier, that laptop is going back in the bag, that kiosk machine with the Wi-Fi card isn't going to respond to WOL, and you won't be able to update a darn thing. One exception to this rule, however, includes the Malicious Software Removal Tool which, if you install this sucker during the day, will immediately run its scan. Depending on available resources, this might produce unwanted (read: "slow and sketchy") behavior on an endpoint. YMMV, the usual disclaimers apply, don't say BGags didn't warn you. VII. Don't Rest On Your Laurels. Okay, you did all these things, you've got your patching percentage up pretty good, you've got scripts that work and a schedule that seems to work flawlessly. Good, congratulations, you should be proud of the work you did. Please get used to the idea that this is all going to unravel in six months. Microsoft is known for pulling the rug out from under your updates and changing their behavior with little warning. Remember October 2016 when most updates went cumulative and you had to check for a veritable nest of KB articles installed on any given endpoint to see if it was protected against WannaCry? Did you actually want to cry? Patching will never be a "set it and forget it" game, which is why Plugins are good: You don't have to do the work. Ok, that's all I got.
  5. One of the problems is that IE 9 and below report as a piece of software visible in Programs and Features, but IE 10 and 11 are distributed as OS Updates. With IE, you can go with the file version of iexplore.exe, but there's also a registry key you could hunt down. For file versions in general, here's a snippet of PowerShell code that usually works: Write-Output (Get-Childitem "").VersionInfo.ProductVersion Go check out the Script Exchange section. I'm pretty sure someone there has posted an IE Version detection script that posts to an EDF. If not, I have one, and will post it if you don't find anything.
  6. Certificate Authorities tend to accumulate a craplog of logfiles that need to be occasionally cleaned out, only to reaccumulate over time. Little-known feature is that you can put the CA into a circular logging mode where they don't accumulate anymore. All SBS servers (eww) come equipped with a running CA service. You can set a search to look for servers with a running certsvc and fire this script at it once a month. You can set a size threshold for cleanup in a Global I've set. Fire it off once a month and never worry about CA log accumulation again. Clear-AD-Cert-Logs.zip
  7. ...okay, upon further review, I apparently am behind on my own Server 2012 / R2 WUA. What the hell. See? You shouldn't listen to me.
  8. Hey, everyone. A funny thing happened when reviewing the h_patching data for 2012 / R2 servers the other day. I noticed that the WUA version reported in my EDF did NOT match the WUA version reported by the actual Windows update command. The ProductVersion of wuaueng.dll is 7.9.9600.17489, but the reported version in the history is 7.9.9600.18235. Well, CRAP. What's perhaps more alarming is that I did a search for "determine windows update agent version powershell server 2012", and I got back an article on SpiceWorks (wait five seconds for that pop-up!) that also referenced using wuaueng.dll as an authoritative WUA version indicator. I wonder if someone else came across that DLL and figured it was a good indicator, or where the heck that could have come from otherwise. I hope it wasn't from me! I don't know which .DLL or .EXE is actually authoritative as of this writing, but wudriver.dll seems likely. Wuauclt.exe is another possibility. Wuapp.exe also shares the same version in Server 2012 / R2, but this is not a universal indicator, as wuapp.exe has been replaced in Windows 10 by an App. If you really wanna cheat, you can use this POSH command pretty reliably for now: Get-ChildItem("C:\Windows\System32\wu*.*") | foreach{Write-Output $_.VersionInfo.ProductVersion} | Sort-Object $_.VersionInfo.ProductVersion | Select -last 1 See what you get for listening to me? Man, you just can't win some days. Especially with Windows Update, it seems.
  9. BGags

    WUA Version

    This topic has been covered at length already. Check out this thread for more information than you really ever wanted to know about the Windows Update Agent. viewtopic.php?f=7&t=2123
  10. This is the sound of me up-voting this request.
  11. BGags

    GPO for LT agent install

    This may be getting slightly off-topic, but I'm surprised that you, Tim, haven't had luck with the probe push. I use it all the time; with LT 10.5 after Patch 4, it works really well and pretty quickly. The only thing that you need to keep in mind is that if you absolutely have to put the Probe on a DC, the LT agent on that DC has to be running as a service account with Domain Admin rights, because the SYSTEM account on a DC doesn't have enough rights.
  12. BGags

    Trying to install .exe file through labtech

    I got this by searching "Filezilla Silent Install" https://wiki.filezilla-project.org/Silent_Setup You want the /s switch, but you probably have to specify the /user=all option in conjunction with it.
  13. You want your script to run. You schedule the script with Wake Offline Agent. You know you've got a location with offline desktops that will respond to a WOL. They've even got a half dozen laptops in docking stations that will also WOL (those weirdos), but they've also got That Marketing Guy's laptop and the VP's Surface that never leaves his desk drawer. You don't want your script to be sitting that those agents' script queues forever. What to do? First, check the script's "Offline" checkbox so that its functions run even if the agent is determined to be offline. Then, use the script's initial If condition (usually set to True) to do a check: SELECT computers.computerid FROM computers WHERE computers.computerid=%computerid% AND CAST(computers.`LastContact` AS DATETIME) > ( DATE_SUB(NOW(), INTERVAL 2 MINUTE)) Set this to check for "Equals" "%computerid%". If the test comes back true, your computer is awake, so run the script. If not, the script fails to the main Else section where you can throw a Script Log Message, raise an Alert, spew a ticket, or whatever. Then bail. The fun part in the query is the CAST() function, which converts the LastContact text field to a datetime subtype that you can then use for date math. If you're already using your initial condition for something else, write a Function script also flagged as Offline, have it set a variable based on the results (@IsOnline@=1), then parse the variable in the parent script and proceed as desired. Happy scripting.
  14. My MSP has been addicted to the App-Care drug for way too long (I know, I know). We're moving to Chocolatey. I'm demo-ing Cubert's freakishly awesome plugin in my sandbox right now. The main problem I'm facing is a fundamental change in methodology in how each product does their thing with third-party apps. App-Care would seek out and maintain software that was installed by the user and client, bringing it up to the most recent version (in theory). So, if a 20-seat client had four installs of Firefox, it would update those four installs of Firefox. Chocolatey seems to want you to assign applications to all of LabTech or per-client as a whole, then make exceptions for places you don't want the applications to go. Unless I'm missing something huge (which does happen, if you know me!), there doesn't seem to be an out-of-box way to simply have Chocolatey maintain pre-installed applications, since Chocolatey requires its own framework to be able to manage the state of the installed applications. For us, going from a "We'll maintain your apps for you" to "We'll standardize your apps across all your computers for you" is a change that few of my clients would welcome. My questions are thus: 1. Can Chocolatey, either the product alone or its use through Cubert's plugin, be massaged into maintaining existing software deployments without automatically installing everything anew? 2. What do people who use the plugin do in these cases? Do you simply tell your clients that you're going to install and maintain List X of software in their environment? If so, was this always the case for your MSP, or did your policies change when you adopted Chocolatey? 3. Am I making this too complicated? (you won't hurt my feeilngs if you say yes). Thanks, everyone.
  15. Good call on the transfer clients. Add FileZilla (both client and server). While we're at it, how 'bout Notepad++ and/or Paint.NET?
×