Jump to content
jnealy

LTSVC.exe is spiking VM's CPU

Recommended Posts

We have had this ongoing issue were LTSVC.exe will spike our VM's CPU for about 5 minutes starting at 5pm. We have had LT Support look into it and they keep telling us that it's not Labtech. We viewed the LTError and LTErrorOld logs every time. I have sent them several screenshots showing LTSVC.exe at the top of the list. When looking at the logs it shows that it just scanned for hotfixes. Looked at all the schedules, commands and scripts that are part of the VM's and I didn't see anything that would be running around 5pm. Has anyone else had this issue or heard of it that could help us out?

Share this post


Link to post
Share on other sites

I have been having the same problem now for a couple weeks. My funny time is 3:15pm Central time. LTSVC and SVCHost spikes to 100% combined. I opened a ticket last weekend about it when on Saturday, sure enough at 3:15pm Central it spiked with no one else on the server. I only have this happening on one VM so far and I can't find a rhyme or reason to it either. I just know I better be in that server at 3:15pm to kill those two tasks and get the server back to normal.

 

My Support tech had me tweak my server schedule a bit but that hasn't helped. I disabled the perf and sesnsor monitors and that helped, it doesn't intermittently spike like it would on occasion, but this 3:15pm thing is driving me crazy too.

 

I upgraded to LT 10 in early June but I don't think it started from that upgrade.

 

Anyone else have feedback for us? Thanks.

Share this post


Link to post
Share on other sites

I've seen similar issues on random machines physical and virtual. LTSVC & SVCHost both spiking together.

 

Stopping and disabling the Windows Update service has fixed it on the 4 or so workstations I've seen this occur on.

 

Since its been so rare I haven't bothered to figure why.

Share this post


Link to post
Share on other sites

Same here. Restarting the Windows Update service seems to be fixing the issue temporarily. Was considering setting up a monitor to check for the spike and restarting that service, but have not bothered to do it yet.

Share this post


Link to post
Share on other sites

I've seen the same. I know it was coming from WU service and disabling fixed it for me. Haven't had time to look into it any more, though.

Share this post


Link to post
Share on other sites

We had this issue and engaged LabTech support about it.

As it turns out, this is a known issue with Microsoft.

 

The "Hotfix Inventory" command is the culprit. It has to do with the Windows Update API, and a similar issue exists when doing updates via MS System Center.

 

LT support sent me this supportability statement about the issue: https://docs.labtechsoftware.com/knowledgebase/article/9558

They also referred this article on MS TechNet: http://blogs.technet.com/b/configurationmgr/archive/2015/04/15/support-tip-configmgr-2012-update-scan-fails-and-causes-incorrect-compliance-status.aspx

 

I know it specifies memory use and also specifies Win7 32 bit OS, but when I pushed about that, the response was along the lines of "well, MS is working on a patch and when that's out we're hoping it will fix other OS versions as well as Win7 x86".

 

At the moment, I'm sitting in a "wait and see" holding pattern, but I'll perhaps respond to the open ticket I have with LT support for a status update on that patch. I'll let you know what I hear back.

 

I hope this is helpful!

Share this post


Link to post
Share on other sites

That's what they told me too, but I have the newest Windows Update Agent installed and am still observing this issue. Just my observation. I'm going back to them with this.

Share this post


Link to post
Share on other sites

Hey all,

 

I've been dealing with this issue for a few weeks now. I've been dealing with a combination of two issues. There's a bug with how the server schedule processes hotfix scans. Essentially, the hotfix scans are queued up instead of running when they're supposed to. When running a hotfix inventory, this causes svchost.exe usage to spike, rendering the server basically unusable until the process is killed. This also explains why the spikes appear to be happening at somewhat random times.

 

This only appears to be affecting our Server 2008 and R2 installs whether or not they are patch managed by WSUS or Labtech. The KB article linked in the supportability article (https://support.microsoft.com/en-us/kb/3050265) has R2 patches, but these definitely haven't fixed the issue. Microsoft released another KB article (https://support.microsoft.com/en-us/kb/3075851) that helps a bit with scan speed performance, but still doesn't resolve the issue.

 

I've tried multiple resets of Windows Update, reinstalls and updates of the Update Agent. The only thing I've been able to accomplish is getting the server patched up. I can confirm that the workaround to this is disabling the Windows Update service. However, for obvious security reasons, we can't do that forever.

Share this post


Link to post
Share on other sites

Just adding a bit of information to the pool.

 

Agents with more than 2 cores/processors are less affected.

 

Resetting WU components and re-installing the WU Client does not alleviate issues. (https://support.microsoft.com/en-us/kb/971058)

 

These issues are apparent even when the Agent's WU administered by WSUS (commonality is just the Update and Hotfix Inventory).

 

Neither KB3050265, KB3056987 or KB3075851 alleviate symptoms (most recent updates to the WU Client).

 

Will be poking at this later today/evening if anyone has any ideas they'd like to see tested.

Share this post


Link to post
Share on other sites

I opened a ticket with LT support on Friday. They told me to just make hotfix scans a weekly thing and set it up on the weekends. Then they closed the ticket. I'm hoping the somewhat scathing email I've sent illustrating how unacceptable that was gets someone to actually look at this instead of just trying to close a ticket.

Share this post


Link to post
Share on other sites

We are seeing this issue also. We have applied the hotfixes and adjusted the hotfix scans to be weekly on Sundays. And the issue still persists, hopefully LT support is getting enough tickets that they are looking into this issue instead of hoping that adjusting the hotfix inventory scan will fix the problem.

Share this post


Link to post
Share on other sites

Bit of an update - I personally had overlooked the CBS log file entirely and found the below. These are recorded before and after resetting Windows Update components.

 

 

2015-08-16 22:22:49, Info CBS Session: 30464155_4272959738 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4272969746 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4272979754 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4272999770 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273009778 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273019786 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273029794 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273039802 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273049810 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273059818 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273069826 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273079834 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273089842 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273099850 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273109858 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273119866 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

2015-08-16 22:22:49, Info CBS Session: 30464155_4273129874 initialized by client WindowsUpdateAgent.

2015-08-16 22:22:49, Info CBS Failed to internally open package. [hrESULT = 0x800f0805 - CBS_E_INVALID_PACKAGE]

Share this post


Link to post
Share on other sites

Finally someone got a ticket escalated to development. I have tried for over a month with no luck. As soon as you hear something please post it here, I'm very interested to hear what they tell you. Also I don't believe it is a MS issue because as soon as we removed LT from one of our test servers the issue went away. Add LT back to the server after a week of testing with LT off and the issue comes back within a few hours.

Share this post


Link to post
Share on other sites

Confirmed, we have this issue as well. It seems that a Microsoft KB3050265 & 3075851 solved the issue on Windows 7 and 2008 R2.

Did you guys try to install these patches manually on the affected PCs ?

Share this post


Link to post
Share on other sites

There's actually been three now:

KB3050265 - June 2015 - WUA version 7.6.7601.18847

https://support.microsoft.com/en-us/kb/3050265

 

KB3065987 - July 2015 - WUA 7.6.7601.18917

https://support.microsoft.com/en-us/kb/3065987

 

KB3075851 - August 2015 - WUA 7.6.7601.18937

https://support.microsoft.com/en-us/kb/3075851

 

I seem to still be experiencing issues with the August update installed -- only when Labtech is installed -- but I'm still testing.

Share this post


Link to post
Share on other sites

I have workstations that fail with the CBS log entry posted above and installing that first KB seems to work so far. I'm testing more. On the workstations though the updates just fail though, not exactly as described above. I'll continue testing.

Share this post


Link to post
Share on other sites

I've continued to work on this issue. If I'm not mistaken, invoking the Windows Update API using the powershell code below replicates the issue:

######### Begin Code Block ###########

$UpdateCollection = New-Object -ComObject Microsoft.Update.UpdateColl

$Searcher = New-Object -ComObject Microsoft.Update.Searcher

$Session = New-Object -ComObject Microsoft.Update.Session

 

$Result = $Searcher.Search("Type='Software' and IsHidden=0")

######### End Code Block ###########

 

If anyone else wants to test this and check. All I'm doing is searching for a list of updates.

Share this post


Link to post
Share on other sites

I'm new to the forum, so I apologize if this suggestion has already been tried:

 

I seem to have had some luck on PC's with CPU spiking problems, simply by wiping out the software distribution folder and resetting the performance counters. I have custom commands and scripts in LT to accomplish this, since I have to do it so often to fix empty patch windows/etc.

 

net stop wuauserv

del /F /Q /s C:\windows\softwaredistribution\*

net start wuauserv

 

(I don't have this in the same script, thus the separation here)

lodctr /r

 

Might also want to check into DNS issues and dump that as well (ipconfig /flushdns), since I ended up having a whole site full of empty patch windows due to WU communication problems caused by a worthless ISP's DNS server. In OpenDNS and Google I trust, for DNS forwarder configurations on DC's (that and/or root hints). Slightly off topic there, but hopefully worth knowing.

 

Anecdotally it has helped me in many instances, but I can't confirm that this is a solution to the exact issue you're facing down in this thread. I have run into customers scrambling to blame LT agents for causing it, when in reality it's all Windows Update's fault IMHO. Hopefully this helps someone out.

Share this post


Link to post
Share on other sites
Guest

Try:

-Export hotfix tables' data for safe keeping

-Update hotfix set pushed=0 where installed=0 and pushed=1 and approved = 1

-Restart agent command

-Resend hotfix command

-Install all approved patches command

-Reboot

Share this post


Link to post
Share on other sites
it seems you guys have issue with windows update not ltsvc.

 

Correct. Specifically, we're having issues with the Windows Update API. There are people using MS System Center getting the same sort of issue.

I imagine everyone here is just trying to find a workaround for the issue while MS dawdles getting an actual fix.

Share this post


Link to post
Share on other sites

It looks like the majority of the CPU utilization issues (including LTSVC.exe utilization) seems to have been resolved internally by Microsoft; there is also an Update to the Windows Update Client this month that looks to help with RAM utilization (from observations, helps more with releasing memory after scanning for this issue in particular).

 

https://support.microsoft.com/en-gb/kb/3083324

Share this post


Link to post
Share on other sites

Hey all,

 

I also had this & the WU Client update did nothing to help (for us at least it didnt!). It turned out it was my schedules that were the reasons for it (or at least they didn't help the matter!) I'll explain...

 

The way in which this works is that the LT server sends out a signal to the LT agents and requests the MS Patching current status and also to go and find any additional patches that may be applicable for the server/workstation. In order to do this, your machine has to communicate with the MS Update Servers.

 

There is currently a (not so well) “known issue” with that MS process right now, that Microsoft are continuing to work on a fix for. Due to this “hold up”, the CPU spikes (for svchost which contains the windows update service) and in turn, as “ltsvc” is waiting for this process to finish (so that it can report the findings back to the LT server for management) the “ltsvc” process also spikes.

 

So technically, while the issue is not directly “a labtech issue”, LabTech IS kicking off the Update/Hotfix inventory so we have altered the schedule to run daily, outside of business hours, as opposed to every 6 hours (6:55am, 12:55pm, 6:55pm & 12:55am).

 

If you adjust your schedules to accommodate this known issue... you should be "fine", or at least... workaround it for now, until MS sort their $hit out. FWIW: We have not seen any negative impact on patch health scores by setting this schedule to daily!!

 

Here is my current Server Schedule:

BeLsc5h.jpg

 

Hope this helps!! :)

 

MK.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...