Jump to content
jasnall

Hyper-V host automated patching

Recommended Posts

Hello, I am looking to see if anyone has a good solution for Hyper-V Host patching. We were told in the past that there was not a great way to automatic this through LT so I wanted to reach out again to see what other were doing.

Looking to see if people are automatically patching and rebooting Hyper-V hosts

Want to make sure there is proper shutdown and power on of VM's

Thanks

Share this post


Link to post
Share on other sites

We have done this with labtech for several years with no issues. We keep the vms set so they suspend when the host shuts down. We just patch the hosts right along with the vms. It works great.

Share this post


Link to post
Share on other sites
Hello, I am looking to see if anyone has a good solution for Hyper-V Host patching.

...

Are you referring to patching Clusters, or standalone hosts?

 

We have no issues with LabTech patching our Hyper-V servers, with the following disclaimer:

You must make sure that your Hyper-V host is configured to perform a regular shutdown without crashing the guests. Guest VM's should be configured to shutdown or save when the host shuts down.

For Guests to recover, they must be configured to power up automatically, or based on previous state.

You should patch Hyper-V on a different day from your guests.

 

The reason for the last item is because if a guest VM is itself patching, it could be in the middle of a critical operation during its own startup or shutdown and ignore the ordered shutdown from the host, causing the host to eventually just power the VM down. Unexpected shutdown is not a good thing for the OS, but it is really not good when you are on step 2 of 3 applying updates....

 

I cannot speak to Cluster Patching for Hyper-V.

Share this post


Link to post
Share on other sites
Hello, I am looking to see if anyone has a good solution for Hyper-V Host patching.

...

Are you referring to patching Clusters, or standalone hosts?

 

We have no issues with LabTech patching our Hyper-V servers, with the following disclaimer:

You must make sure that your Hyper-V host is configured to perform a regular shutdown without crashing the guests. Guest VM's should be configured to shutdown or save when the host shuts down.

For Guests to recover, they must be configured to power up automatically, or based on previous state.

You should patch Hyper-V on a different day from your guests.

 

The reason for the last item is because if a guest VM is itself patching, it could be in the middle of a critical operation during its own startup or shutdown and ignore the ordered shutdown from the host, causing the host to eventually just power the VM down. Unexpected shutdown is not a good thing for the OS, but it is really not good when you are on step 2 of 3 applying updates....

 

I cannot speak to Cluster Patching for Hyper-V.

 

I agree with patching on different day than guests. how do you get the guests in maintenance mode?

Share this post


Link to post
Share on other sites
I agree with patching on different day than guests. how do you get the guests in maintenance mode?

 

Maintenance Mode? Bah.. Maintenance Mode is for people that triage their own tickets...

 

(But we have maintenance as a daily window, so the servers are all under the same maintenance window even on their non-patching days...)

Share this post


Link to post
Share on other sites
I agree with patching on different day than guests. how do you get the guests in maintenance mode?

 

Maintenance Mode? Bah.. Maintenance Mode is for people that triage their own tickets...

 

(But we have maintenance as a daily window, so the servers are all under the same maintenance window even on their non-patching days...)

 

Ah ok. We have some customers that are 24/7 so cant do that globally. I suppose I can start using those maintenance windows at the location level....

Share this post


Link to post
Share on other sites

Not to resurrect this unduly, but I am running into this specific challenge with some VMs monitored 24/7. For ones with standard monitoring (i.e. they don't trigger alerts overnight) it's not a big deal to patch Hyper-V hosts so long as the guest VM configs are set to Save. That way even if the host reboots for patching, the guests will save whatever state they were in and just resume when the host is back up. However, for 24/7 systems, even if we have maintenance mode set to trigger automatically (a great feature) in the new Patch Manager, that will only apply to whichever system is rebooting. Thus if the guests are done patching but the host is rebooting, I will start to get alerts waking up techs when the heartbeat fails on the guests.

Ideally I'd trigger a script to set maintenance mode on the location and apply it to a reboot policy for these 24/7 Hyper-V hosts, but I don't see a script function to do that. You can set maintenance mode on an individual computer, but does anyone know how to script that for a location? That's the missing piece of this puzzle for me.

In the meantime, it's manually triggering patch deployment and setting location maintenance to silence alerting for these 24/7 Hyper-V hosts...

Share this post


Link to post
Share on other sites

wonder if anyone ever solved this hyper-v host maintenance mode puzzle? Just about to set maintenance mode manually at each location but feels like must be a neater way!

Share this post


Link to post
Share on other sites
On 4/29/2017 at 2:34 PM, tlphipps said:

We have done this with labtech for several years with no issues. We keep the vms set so they suspend when the host shuts down. We just patch the hosts right along with the vms. It works great.

Would you please share the process that you are following?

Share this post


Link to post
Share on other sites
On 4/29/2017 at 8:21 PM, DarrenWhite99 said:

Are you referring to patching Clusters, or standalone hosts?

 

We have no issues with LabTech patching our Hyper-V servers, with the following disclaimer:

You must make sure that your Hyper-V host is configured to perform a regular shutdown without crashing the guests. Guest VM's should be configured to shutdown or save when the host shuts down.

For Guests to recover, they must be configured to power up automatically, or based on previous state.

You should patch Hyper-V on a different day from your guests.

 

The reason for the last item is because if a guest VM is itself patching, it could be in the middle of a critical operation during its own startup or shutdown and ignore the ordered shutdown from the host, causing the host to eventually just power the VM down. Unexpected shutdown is not a good thing for the OS, but it is really not good when you are on step 2 of 3 applying updates....

 

I cannot speak to Cluster Patching for Hyper-V.

Would you please explain these steps?

1. You must make sure that your Hyper-V host is configured to perform a regular shutdown without crashing the guests.

2. Guest VM's should be configured to shutdown or save when the host shuts down.

Thanks

Share this post


Link to post
Share on other sites

We are also in a situation where VM outages due to Host patching causes false positive server offline tickets. I did start working on a pre patch script but could never work out how to finish it. The basic idea was to identify that the server was a host, then list all VMs against it, then set maintenance mode for each of them. If anybody has any suggestions I'm all ears!

Share this post


Link to post
Share on other sites

Just make sure you have maintenance windows defined for your patching period. In our case, our patching window runs from 2-4am. So we have a maintenance window defined during that time for all servers. No alerts then regardless of what machines are up/down since they're all rebooting during that period.

@Dayrak I'm really not sure how else to explain it. We patch hyper-v hosts just like any other server. We don't treat them differently AT ALL.

In the configuration of each Hyper-v VM, we make sure to choose the Automatic Stop Action as 'save.' (see screenshot).

Our experience is this causes ZERO issues. Even if the VM is in the middle of patching when host reboots. The VM just starts right back up and finishes what it was doing with the VM turns back on.

 

2019-03-20 09_59_23-Window.png

Share this post


Link to post
Share on other sites
7 minutes ago, lgs141 said:

@tlphipps How do you configure your maintenance windows? We patch monthly so the built in Maintenance windows don't work for us as that only allows weekly scheduling.

We leave the maint. window set weekly even though patches only really roll out monthly. It's a 2 hour gap in 'monitoring' during an overnight period. We find that acceptable since we still get alerted when the maintenance period is over for any machines still offline (I can't think of a time that has happened in last 10 years)

Share this post


Link to post
Share on other sites

Have to be careful with SAVE auto stop action.  VM's with a ton of RAM can take way longer to save than to just shut down.  We have customers with VM's in the 100's of GB of ram.  Save action also creates a blank file of equal size on the host to accommodate that (similar to hiberfile.sys).  Make sure you have the disk space before you go setting all VM's to save
 

I like @maxgruv idea of setting location into maintenance mode via script in the reboot policy, I'll work on that.  That way we don't have to go by group level maintenance windows which may apply unnecessarily as can only do weekly.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×