Jump to content
MSPGeek Admin Team

URGENT - Automate update needed by March 9th 2019

Recommended Posts

What exactly is the problem?
One of the DLLs critical to encryption in the affected versions has a bug that means on the 9th of March it will stop working, subsequently taking all services that rely on it with it (Control Center/Agent/Automate Server).

How can I tell if I am affected?
Step 1 - Check your server version
In your Automate Control Center, go into Help and go into About. You want to pay attention to the Current Control Center, Automation Server, Web Server and Remote Agent versions. The following versions are affected:

Automate 12 Patch 11 - 12.0.474 (AKA 12.0.11.474)
Automate 12 Patch 12 - 12.0.496 (AKA 12.0.12.496)
Automate 2019 Patch 1 - 19.0.37 (AKA 19.0.1.37)

If you are on Automate 12 Patch 10 or below you remain unaffected and need to do nothing else.

Step 2 - Check your Automate Agent versions
This is the version of Automate that gets installed on your Windows endpoints. Mac and Linux agents are not affected. Even if you install the latest Automate patch, unless you make sure the agents are updated too you will still lose agents.

In your Automate Control Center, go into Automate then Dataviews. Go in the Status folder and open Agent Status. Pay attention to page size and total in the top right. We recommend upping the page size to view all agents. You can see the current agent installed version in the Agent Version column. You can use the dropdown above the Agent Version column header to see what unique agent versions you have.

Any Windows agent in this Dataview on version 120.474, 120.496 or 190.37 by March the 9th will be permanently lost and will require a reinstall

For those of you who prefer an SQL method (please note this query won't be accurate AFTER patching unless you do the full patch)

SELECT `name` AS ComputerName,clientname,locationname,computers.computerid,serviceversion,lastcontact,IF((LastContact > DATE_ADD(NOW(), INTERVAL - 15 MINUTE)),"ONLINE","OFFLINE") AS AgentStatus FROM computers INNER JOIN v_xr_computers ON computers.computerid = v_xr_computers.computerid WHERE serviceversion > 120.451 AND serviceversion < 190.58

How do I fix this?

ADDITIONAL FIXES AVAILABLE PLEASE SEE BELOW.
Step 1 - Update or Patch your Automate Server

Install a full upgrade to Automate 19 Patch 2 - (Please see your product dashboard on ConnectWise University) (this is the option the majority of the MSPGeek Admin team chose)
Install a patch, without upgrading your version if you're on Automate 19 Patch 1 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 12 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 11 - (We will post this link when we get it)

Step 2 - Update your agents
You can force an update on an agent basis by going into Commands > LabTech > Force remote to Update. We highly recommend though Darren White's excellent internal monitor available here https://www.mspgeek.com/files/file/45-cwa-agent-version-update-monitor/ . To import this, go to Systems > General > Import > SQL and that's all you need to do. The agent update process is cleverly embedded within the monitor, and it will do it in a staged way depending on how big your environment is. 

I am on Hosted/Cloud Automate, does this still affect me?
Absolutely - we suggest that you contact support and ask for your instance to be updated ASAP.

What do I do next?
Continue to monitor your agent versions and ensure that affected versions gradually decrease. Get your clients to turn on machines that haven't been on for a few weeks. 

Where can I get further help/discuss this further?
We have created a channel called #dday in Slack for discussion specifically relating to this.

ConnectWise has provided a utility to use ConnectWise Control to update your Automate Remote Agents. 

Using Control to Update your Automate Remote Agents

This should be used with Automate Server Versions, 12.0.11.475, 12.0.12.497, 19.0.1.38, 19.0.2.58

 

When run, the executable prompts the user for a ConnectWise Control server URL, Username and Password, as well as the option to create a session group on the Automate server.
image.png.5d6cc729ac6826c760dddf97e11de830.png
Once entered, the user clicks the "Deploy Hotfix" button. This will:

  • Install (and enable) a Signed ConnectWise Control extension onto the ConnectWise Control server.

    • The extension is embedded in the exe and streamed to the Control Server directly. It does NOT use the extension browser/store.

    • The embedded extension is signed and approved by Control, so it should work with all Control licenses.

    • The extension includes the the hotfix executable. The remote clients will download the hotfix from their respective Control servers.

    • It also includes a .bat file that will be run against all Windows machines which handles hotfix need identification as well as downloading and executing the hotfix.

  • Call the "HotfixCWAOnAccessSessions" endpoint in the extension. This endpoint:

    • Goes through every ACCESS session in Control (the ones used by Automate for remote access to agents)

    • For Windows machines it will queue a Control Command that forces the "cmd" interpreter to run for up to 5 minutes to complete the tasks in the included .bat file.

    • This bat file: 

      • Checks the agent version based on the "HKLM\Software\LabTech\Service\Version" key

      • If found and it matches one of the known good versions, will report that version and end.

      • If the version is not found it will check for the affected file in the "%windir%\ltsvc" folder and if found, continue to validate the file. If the file is not found, it will report that Automate is not installed.

      • If a different version of the Agent is identified (or the affected file is found as mentioned) it will check the file version using wmic. If the build of the version is below 16 it will begin download and execution of the patch file.

      • It will fail through from using powershell to using a vbs file to attempt to download the executable from the Control server.

      • Once the file is downloaded, it will run it and then check every second or so for the affected file version. Once it has been found to change it will report the installed Automate version with the affected file version in square brackets following it.

      • If the file does not update within 30 seconds, it will report that the update failed.

      • Cleans up the added files used to download/execute the hotfix. (Note - they are written to the %windir%\ltsvc folder)

    • For non-windows machines, it updates CustomProperty7 with a message that the machine is not a windows system.

  • There is also an event handler in the extension that will check for the response from the command sent to check/run the hotfix that:

    • "User Deletes" the original command sent to the remote agent to perform the check/execution.

      • Note - this simply removes it from the UI - it is still available via Reporting.

    • Sets CustomProperty7 of the session to the last line returned in the response (which is a status, such as a version number or a failure message).

  • At this point (after the extension is installed, and the hotfix deployment started) it will notify the user that deployment has started.

  • If the "Create Session Group" button is checked it will then

    • Get existing session groups on the Control server.

    • Check for any with the name "Out of Date Automate"

      • And also check that it matches the one intended to be setup.

        • If the same, will report success.

        • If different (but same name) will report it cannot create the session group, but that hotfix has started.

    • If no match, add a new session group.

Note that this queues a command, which will run the next time an offline machine connects to the Control server
image.png.bfa5404f4fd7bf13740b8b767052f47b.png

Note that when the extension is installed, a Control Administrator can also call the Command against all Access sessions from Control by navigating to Admin > Extensions > "ConnectWise Automate: Agent Recovery" > Options > Run On All Machines

image.png.bd4f824dd4f34e2603f71e2ac90e5e16.png

 

To run the tool on specific machines (including non-Access sessions), a Control Administrator can also select one or more sessions on the Host page and use the "Apply Automate Agent Hotfix (Mar 2019)" command using the right-click menu, "••• More" menu, or details panel when more than one session is selected.

 image.png.5596fd5bf726235a45659e6f5cf548a8.png

Cleanup and Remove Extension:

  • This link in the executable is to be used once the hotfix has been deployed to all affected agents on a system, and the extension is to be removed.

  • When clicked, will prompt the user to confirm they want to perform the action.

  • If confirmed, it will :

    • Call the "CleanupHotfixExtension" endpoint in the installed extension, which empties CustomProperty7 on all ACCESS sessions.

    • If Create Session Group is checked

      • Get session groups from the Control server

      • Check for "Out of Date Automate" group

        • If it exists, and it matches the expectation, removes it.

        • If it exists, but is not the same, does not remove it.

    • Calls Uninstall Extension for the installed extension.

 

As a note, the extension can not be uninstalled from the Admin > Extensions page without first being disabled on the Control Server. If uninstall is selected while the extension is active, it will notify the user that it must first be disabled.

 See the attached EXE file for download HotfixAutomateVisControl.exe

HotfixAutomateViaControl.exe

  • Thanks 3

Share this post


Link to post
Share on other sites

Hi there, I work for ConnectWise. We have been working to solve an issue discovered by the LabTech Geek/MSP Geek Community Administrators, for the past week. We take this issue seriously and have been working to build and test solutions for all impacted versions.

Automate partners on the following versions are affected :

• Automate 2019.1 [19.0.1]

• Automate version 12 patch 12 [12.0.12]

• Automate version 12 patch 11 [12.0.11]

An update must be applied to your Automate server that addresses how we handle the transition to our new code signing certificate. Failing to address this issue this month will result in the failure of agent and control center communication.

Next Steps:

- All Automate cloud partners will be upgraded to 2019.2 next week to resolve this issue.

- On premise Automate administrators running impacted versions will be receiving emails today with options to address this issue.

If you have additional questions about this issue, we have created a forum to answer and share related questions here. For further critical updates, please visit here and select the Critical Updates tab.

 

Share this post


Link to post
Share on other sites

Update status:

 

  • We have over 800 partners on the full 2019.2 patch since it was released to general availability last Wednesday
  • We will be upgrading all cloud partner instances this week as part of their monthly maintenance window
  • We have seen good adoption of hotfixes across all impacted version since their release last Friday
  • There are no current support trends related to the 2019.2 release or the hotfixes

 

If you have not received an email and are running an impacted version listed above, we do not have a valid Automate Administrator email address for your instance.  We recommend that you apply one of the resolutions option above for your server as soon as possible.  We are tracking all impacted partner instances and have plans in place to contact and push out patches to those that remain unpatched.  Our goal is to have all partners servers resolved by the Friday the 22nd.

 

We will continue to use this forum to update you on progress related to this issue.

 

Sincerely,

ConnectWise Automate Product Management

Share this post


Link to post
Share on other sites

We, as a team, have noticed over the last couple of days that there has been some pretty heavy criticism of ConnectWise across forums, Slack, and social media outlets like Reddit in relation to the problems with the Automate binaries. A lot of the criticism has been relatively unfounded. The problems found within the product are not issues that we believe could have been picked up by any standard QA or testing measures - bearing in mind that our team member only stumbled on to this by complete accident.

In regards to the criticism surrounding ConnectWise's notification to partners, it's much easier for the leaders in a community like ours to notify our members as we are not bound by the processes, procedures and multiple teams required to get a fix like this out successfully at a corporate level. Though there are clear areas for improvement in QA and testing processes, every member of the MSPGeek team was impressed at the speed of response and subsequent delivery of a fix by the Automate team.

Our community was founded and based upon the idea of mutual assistance, open sharing, good communication and, for the past 6 years, has provided a trusted platform for users to support each other while helping ConnectWise make the product better. Let's continue to help ConnectWise by providing constructive criticism while helping them, and each other, through this bump in the road; it's the MSPGeek way.

  • Like 7
  • Thanks 4

Share this post


Link to post
Share on other sites

Update Status:

• More than 4,100 partners have completed the full 2019.2 patch since it was released earlier this month

• All cloud partner servers have been updated to 2019.2

• We have seen good adoption of hotfixes across all impacted versions since they were released on 2/8/19

• Currently there are no negative support trends related to the 2019.2 release or the hotfixes

• New messaging will be sent out today to Automate administrators and primary contacts for partners that are still impacted by this issue 

We will continue to message impacted partners until all partners have completed the patch. We are tracking all impacted partner instances and have plans in place to push out hotfixes to those that remain unpatched if required. Our goal is to have all partners’ servers resolved by Friday, 2/22/2019.

We will continue to use this forum to update you on progress related to this issue.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.

×
×
  • Create New...