Jump to content
bigdog09

Automate - Patch Upgrade Guide [Apply 2020.7 ASAP]

Recommended Posts

Posted (edited)

Patch 7 is out.  https://university.connectwise.com/university/pageview.aspx?short_name=patch-release-notes It says they are delaying security disclosure a week (July 9) to give everyone time to patch.  Also that they won't patch 2020.1 to .6.

The notes say "Complete the system password reset steps after you applied the patch and after your agents have been successfully updated."  However the linked doc says nothing about having to update all existing agents first...

Edit:
I had stopped to read the release notes.  The end of the email says "Both the 2020.7 and 2019.12 patches address issues that prevented agents from receiving the new system password after a reset was performed. Completing the password reset step after the patch is applied & agents have updated ensures agents receive the new information in case they are currently out of sync."
So that probably affects anyone who changed their system password before now, especially in the last week or two.

Edited by SteveYates
system password info

Share this post


Link to post
Share on other sites
1 minute ago, lbegnaud said:

Anyone know if 2019.12 will get a patch as well?

The CW email says they are patching 2019.12 just not 2020.x.

Share this post


Link to post
Share on other sites
4 hours ago, SteveYates said:

Patch 7 is out.  https://university.connectwise.com/university/pageview.aspx?short_name=patch-release-notes It says they are delaying security disclosure a week (July 9) to give everyone time to patch.  Also that they won't patch 2020.1 to .6.

The notes say "Complete the system password reset steps after you applied the patch and after your agents have been successfully updated."  However the linked doc says nothing about having to update all existing agents first...

Edit:
I had stopped to read the release notes.  The end of the email says "Both the 2020.7 and 2019.12 patches address issues that prevented agents from receiving the new system password after a reset was performed. Completing the password reset step after the patch is applied & agents have updated ensures agents receive the new information in case they are currently out of sync."
So that probably affects anyone who changed their system password before now, especially in the last week or two.

Does it mean we have to change the password twice again?

Share this post


Link to post
Share on other sites
1 minute ago, eversurf said:

Does it mean we have to change the password twice again?

I guess, except this time they fixed a bug which prevent agents from picking up the password change, so you're supposed to rotate passwords after rolling out the new agent.

Share this post


Link to post
Share on other sites
Posted (edited)

I jumped in, to get it out of the way.  They didn’t specifically say but I changed our template to upgrade the agent immediately, since they implied that would happen and said no commands would process if it didn’t.

Good news is the patch went in fine.  Bad news, about 10% of our agents dropped offline.  I found on all of them stopping and starting ltservice service via CW control's remote command line got it to upgrade.  Some logged connection errors in lterrors.txt which could be our upload being swamped.  However they didn’t reconnect in a half hour.  And, more than one logged a stream of errors like this:

LTService  v200.201      - 7/2/2020 11:47:45 PM  - Proc Cmds7 ToInteger2 - Conversion from string "randomletters" to type 'Integer' is not valid.:::

many times per second.  All that is before changing the system password.

For anyone not aware, Dashboard/Management/Outdated is a great way to see old agent versions.

Our one XP computer (ugh) needed the service restart but is connecting.

Edited by SteveYates
  • Thanks 1

Share this post


Link to post
Share on other sites
13 hours ago, SteveYates said:

I jumped in, to get it out of the way.  They didn’t specifically say but I changed our template to upgrade the agent immediately, since they implied that would happen and said no commands would process if it didn’t.

Good news is the patch went in fine.  Bad news, about 10% of our agents dropped offline.  I found on all of them stopping and starting ltservice service via CW control's remote command line got it to upgrade.  Some logged connection errors in lterrors.txt which could be our upload being swamped.  However they didn’t reconnect in a half hour.  And, more than one logged a stream of errors like this:

LTService  v200.201      - 7/2/2020 11:47:45 PM  - Proc Cmds7 ToInteger2 - Conversion from string "randomletters" to type 'Integer' is not valid.:::

many times per second.  All that is before changing the system password.

For anyone not aware, Dashboard/Management/Outdated is a great way to see old agent versions.

Our one XP computer (ugh) needed the service restart but is connecting.

I went ahead and did it also, and I had the same issue. I had a lot more than 10% drop out (maybe more like 30%) and had to restart the agents manually. 

It was definitely not as seemless as most recent patches have been, so be prepared.

Share this post


Link to post
Share on other sites
Posted (edited)

Thanks for the information guys.

Front Page has been updated with the latest past as well as a Pre and Post patch guide to ensure that your agents will check in after upgrading. As always please post any issues you're experiences so that we can all stay informed.

Thanks!
bigdog.gif

Edited by bigdog09
  • Thanks 1

Share this post


Link to post
Share on other sites

@rittwage and @SteveYates, approximately how many agents do you have?  CW mentions that the issue is exacerbated by large numbers of agents, but they don't mention what number qualifies as "large agent environments".

Share this post


Link to post
Share on other sites
3 minutes ago, Greg-ATG said:

@rittwage and @SteveYates, approximately how many agents do you have?  CW mentions that the issue is exacerbated by large numbers of agents, but they don't mention what number qualifies as "large agent environments".

Not really something I would disclose, but "many thousands".  I did the upgrade before Connectwise had the "pre-install" guide unfortunately. I recovered from it, but it was painful for a couple hours.

 

  • Thanks 1

Share this post


Link to post
Share on other sites

Can anyone explain the reasoning behind the system password reset that CW is instructing in the release notes? Their exact wording is:

Quote

Complete the system password reset steps after you applied the patch and after your agents have been successfully updated. This ensures agents receive the new information in the case that they are out-of-sync.

This makes no sense to me. If the reset supposedly fixes the problem of agents not getting updated, but you are supposed to do it after they have been successfully updated, then it seems a bit pointless. What does "out-of-sync" mean anyways?

Share this post


Link to post
Share on other sites
Posted (edited)

We don't have a high number of agents currently.  I suspect it has more to do with the bandwidth available but that's just a guess.  I'd just expect it to happen.  Per CW's partner notification message Friday the agents should self-recover after 24h.

Tech support replied with what looked like a canned email suggesting terminating all three LT processes and starting the services.  They did not address my "Conversion from string "randomletters" to type 'Integer' is not valid" error logged several times per second on the endpoints.

re: changing system password, read

but in general it sounds like if someone got hold of your installer they could use the system password to "join" the system and communicate with the server at a more privileged level.  Changing the password twice rotates out the "previous" password which was still valid for installs.  Edit: this is also why they are blocking the direct download of the installer.

re: change after the patch, they didn't explain that well at all but per the release notes "Important: The 2020.7 and the 2019.12 patches address issues that prevent agents from receiving the new system password after a reset was performed. Complete the system password reset steps after you applied the patch and after your agents have been successfully updated. This ensures agents receive the new information in the case that they are out-of-sync." So the advice last month to change passwords apparently didn't work in some cases because of this bug??

Edited by SteveYates
  • Like 1

Share this post


Link to post
Share on other sites

Thanks for all the info everyone.  Going to pull the trigger on this tonight or possibly wait until the weekend due to the potential for agents to not check in by SOB tomorrow morning.  Question though, I'm sure I should know but where would I change the system password?  Also, I don't have any knowledge of what the current password is as I did not build the server but inherited it as the admin now.

Share this post


Link to post
Share on other sites
3 minutes ago, ryno252 said:

Thanks for all the info everyone.  Going to pull the trigger on this tonight or possibly wait until the weekend due to the potential for agents to not check in by SOB tomorrow morning.  Question though, I'm sure I should know but where would I change the system password?  Also, I don't have any knowledge of what the current password is as I did not build the server but inherited it as the admin now.

See the release notes https://university.connectwise.com/university/pageview.aspx?short_name=patch-release-notes which links to https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/070/260/050.

You don't need to know the password it's just used for agent signups.  Basically it's internal, but it turns out if someone got it they could do things like create user accounts...hence the flurry of security notices last month.

Share this post


Link to post
Share on other sites
4 minutes ago, SteveYates said:

See the release notes https://university.connectwise.com/university/pageview.aspx?short_name=patch-release-notes which links to https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/070/260/050.

You don't need to know the password it's just used for agent signups.  Basically it's internal, but it turns out if someone got it they could do things like create user accounts...hence the flurry of security notices last month.

Thanks Steve.  That's what I needed.

Share this post


Link to post
Share on other sites

I just completed the upgrade and did not have any issues. Upgraded to 2020.7, waited for the agents to update, rebooted, then reset the system password twice. 

The agent command history shows the server password change.

image.png.83d027937571d618c5d2c1bfd3e6984b.png

Share this post


Link to post
Share on other sites

I've seen the same " Proc Cmds7 ToInteger2 - Conversion from string" error messages in LTErrors logs that @SteveYates and others have mentioned, only I've seen them as part of a set of re-occurring, nearly identical error message entries.

I suspected it could be related to the upgrade to Patch 7, but wasn't sure when I contacted Support earlier today about it. I provided a set of logs from a problematic LT Agent, and executed the requested commands to kill the Processes and restart the Services. The issue(s) just started back up again along with the Services.

The LT Agents, themselves, are non-functional, but we are still getting a Heartbeat from them so the servers aren't triggering any "offline" alerts. We just cannot properly manage them. :/

The "Server Password" aspect of this makes sense as far as the LT Agents not being able to speak to the LT Server, itself, but why are those computers still able to send in Heartbeats?

Also, some of the other messages in those re-occurring blocks... well... I think they indicate another issue present which isn't strictly the fault of the Patch, but is being exposed by it.

Share this post


Link to post
Share on other sites
Posted (edited)
38 minutes ago, KI_EricS said:

The issue(s) just started back up again along with the Services.

Restarting the ltservice worked for all but one of our PCs, and that one kept showing the errors so I uninstalled/reinstalled the agent.  That can be done from Control:

start "" powershell -command "& { (New-Object Net.WebClient).DownloadFile('https://someserver/secretpathtoyour/agent_install.msi', 'c:\windows\temp\agent_install.msi') }"
start "" powershell -command "& { (New-Object Net.WebClient).DownloadFile('https://yourLTserver/Labtech/Deployment.aspx?ID=-2', 'c:\windows\temp\agent_uninstall.exe') }"
dir c:\windows\temp\agent* (to ensure they finished downloading)
cmd /c start "" c:\windows\temp\agent_uninstall.exe (and wait a couple minutes until dir c:\windows\ltsvc\*.exe finds nothing)
cmd /c start "" msiexec /i c:\windows\temp\agent_install.msi /qn /quiet /norestart

Or uninstall via Control and install via probe, GPO, etc.

Edit: we have MAC signup enabled so when we uninstall/reinstall it "installs into" the existing computer, not a new one.

Edited by SteveYates
  • Thanks 1

Share this post


Link to post
Share on other sites

I have found that you still cannot download the generic agent installer, only the custom agents for clients even in 2020.7. ConnectWise support advised that deployment.aspx is still disabled and devs are working on it. Anyone else seeing the same?

Share this post


Link to post
Share on other sites
30 minutes ago, activtech said:

I have found that you still cannot download the generic agent installer, only the custom agents for clients even in 2020.7. ConnectWise support advised that deployment.aspx is still disabled and devs are working on it. Anyone else seeing the same?

I had reported that after the 2020.6 hotfixes and was told it was a known issue.  Reading between the lines, I don't think that's getting "fixed" anytime soon.  The default installer includes the server password so new agents can sign up.  Bad guys can also sign up with the server password and might be able to do bad things.  Ergo, they blocked the installer downloads so bad guys can't get the server password.  Unless they can come up with a different way to link agents to a server that doesn't allow bad guys access, I don't see how they can make it public again.

One can create installers for the default company/location (we named ours _New Computers) which is basically the same thing.  However creating the .exe requires picking an admin username/password so that doesn't work on most PCs.  One can create an MSI without an admin password though.

I didn't explain it, but that's why my example above used the MSI and had the "secretpathtoyour/agent_install.msi" URL if you choose to put an installer someplace downloadable.

Share this post


Link to post
Share on other sites
Posted (edited)
14 hours ago, SteveYates said:

One can create installers for the default company/location (we named ours _New Computers) which is basically the same thing.  However creating the .exe requires picking an admin username/password so that doesn't work on most PCs

Well as I'm sure you know, the .exe installers used to work - they've managed to screw up yet another previously-working thing. Manually running the resulting .exe installer returns a "Unable to retrieve installation credentials due to an unknown installer request. Please create a new installer." error message (after I created an installer with a 'fake' admin account since it requires that you specify one).

You can bypass this bug however and get the .exe agent installers working again by calling them with /s ... ie, Agent_Install.exe /s

...which is something fun to have to talk someone through on the phone.

...and now, if you'll forgive me, everything below is just me screaming into the void, because that's all any of us can do since Connectwise certainly doesn't care about their screw-ups.

I'm about sick of the absolutely abysmal coding/qualitycontrol disasters with Labtech. How the hell does garbage like the above get out the door of Connectwise? It's one disaster after another with these people... new updates will nuke half your agents about half the time (that SSL disaster from a year or so ago, and now all this disaster with 2020.7 checkin issues), and then some absolute moron decides "Hey! Let's just ASSUME that every person using our product makes separate custom installers and is absolutely super duper comfy with the idea of HARD CODING a DOMAIN ADMIN password into something that will be locally distributed to all the PCs, because that's good security practice. Let's totally code our product to rely on THAT rather than that the installer will simply be run using an account that already has local admin privileges, since that's all we need temporarily, since our product will run as SYSTEM account anyway. Heck, let's even design it so that even though it could test if it had the rights it needed, let's instead just have it bomb out without even looking if the hard-coded credentials don't work." UGHHT@#H$BN@#!K$BN@K#RNK#@R

I'd be running in the other direction and never look back, but from everything I've read, it sounds like all the other RMM solutions out there are also garbage in various other ways... Makes me think maybe I should just code my own basic RMM. I know for a damn fact I could do a better job than these morons. I get that there's a lot involved and one person would be hard pressed to do the whole thing themselves, yes, but the #1 thing I care about it just that I can 1.) deploy an agent and 2.) not lose contact with agents and 3.) I can run command line commands and transfer files ... they can't help but keep screwing up 1 or 2 over and over and over again while they put dev effort instead into other stuff that is a million times less important. I exclusively do everything else via my own scripts I call, mostly composed by powershell scriptlets. For all other needs (patching etc) I could just use other focused options.

....okay, sorry-not-sorry about that... I just had to vent! :)

Oh, and incidentally, I've had terrible luck with msiexec /i c:\windows\temp\agent_install.msi /qn /quiet /norestart - not sure why, but it seems to rarely work. When I manually run it not in quiet mode it will usually work, but then that shows the system password hash on the screen, which I don't exactly love, and you also can't do that from the command line....

Edited by Graham
  • Like 1

Share this post


Link to post
Share on other sites
15 hours ago, activtech said:

I have found that you still cannot download the generic agent installer, only the custom agents for clients even in 2020.7. ConnectWise support advised that deployment.aspx is still disabled and devs are working on it. Anyone else seeing the same?

I received the following response from a support chat yesterday about this.

"Ahh. Sorry about that we have a know issue where partner are not able to download the default agent.
However, Our dev team is working on this and they will provide fix in up coming patch 20.8x."

Share this post


Link to post
Share on other sites
35 minutes ago, Graham said:

Oh, and incidentally, I've had terrible luck with msiexec /i c:\windows\temp\agent_install.msi /qn /quiet /norestart - not sure why, but it seems to rarely work. When I manually run it not in quiet mode it will usually work, but then that shows the system password hash on the screen, which I don't exactly love, and you also can't do that from the command line....

My experience is that when the "/quiet" installation fails, you'll find that the "Server Address" value in registry (HKLM\SOFTWARE\LabTech\Service) doesn't have the correct data.  I've built in a registry check in my own script that then runs the uninstaller and reinstalls the MSI if this registry check fails.  The 2nd or 3rd attempt generally works.

Share this post


Link to post
Share on other sites
1 hour ago, Graham said:

Oh, and incidentally, I've had terrible luck with msiexec /i c:\windows\temp\agent_install.msi /qn /quiet /norestart - not sure why, but it seems to rarely work. When I manually run it not in quiet mode it will usually work, but then that shows the system password hash on the screen, which I don't exactly love, and you also can't do that from the command line....

/qn is the same as /quiet /norestart I removed those and it works for me.

Share this post


Link to post
Share on other sites
7 hours ago, Greg-ATG said:

My experience is that when the "/quiet" installation fails, you'll find that the "Server Address" value in registry (HKLM\SOFTWARE\LabTech\Service) doesn't have the correct data.  I've built in a registry check in my own script that then runs the uninstaller and reinstalls the MSI if this registry check fails.  The 2nd or 3rd attempt generally works.

 

8 hours ago, Graham said:

Oh, and incidentally, I've had terrible luck with msiexec /i c:\windows\temp\agent_install.msi /qn /quiet /norestart - not sure why, but it seems to rarely work. When I manually run it not in quiet mode it will usually work, but then that shows the system password hash on the screen, which I don't exactly love, and you also can't do that from the command line....

You can (could? I assume this still works) pass the server information to the MSI

/quiet /norestart SERVERADDRESS=https://yourserver.com SERVERPASS={yourpass} LOCATION={id}

Found that those were needed when I was deploying via Intune the first time; didn't matter if it was a custom client specific MSI, if I didn't pass the extra params it just installed with no server information.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...