What exactly is the problem?
One of the DLLs critical to encryption in the affected versions has a bug that means on the 9th of March it will stop working, subsequently taking all services that rely on it with it (Control Center/Agent/Automate Server).
How can I tell if I am affected? Step 1 - Check your server version
In your Automate Control Center, go into Help and go into About. You want to pay attention to the Current Control Center, Automation Server, Web Server and Remote Agent versions. The following versions are affected:
Automate 12 Patch 11 - 12.0.474 (AKA 220.127.116.114)
Automate 12 Patch 12 - 12.0.496 (AKA 18.104.22.1686)
Automate 2019 Patch 1 - 19.0.37 (AKA 22.214.171.124)
If you are on Automate 12 Patch 10 or below you remain unaffected and need to do nothing else.
Step 2 - Check your Automate Agent versions This is the version of Automate that gets installed on your Windows endpoints. Mac and Linux agents are not affected. Even if you install the latest Automate patch, unless you make sure the agents are updated too you will still lose agents.
In your Automate Control Center, go into Automate then Dataviews. Go in the Status folder and open Agent Status. Pay attention to page size and total in the top right. We recommend upping the page size to view all agents. You can see the current agent installed version in the Agent Version column. You can use the dropdown above the Agent Version column header to see what unique agent versions you have.
Any Windows agent in this Dataview on version 120.474, 120.496 or 190.37 by March the 9th will be permanently lost and will require a reinstall
For those of you who prefer an SQL method (please note this query won't be accurate AFTER patching unless you do the full patch)
SELECT `name` AS ComputerName,clientname,locationname,computers.computerid,serviceversion,lastcontact,IF((LastContact > DATE_ADD(NOW(), INTERVAL - 15 MINUTE)),"ONLINE","OFFLINE") AS AgentStatus FROM computers INNER JOIN v_xr_computers ON computers.computerid = v_xr_computers.computerid WHERE serviceversion > 120.451 AND serviceversion < 190.58
How do I fix this?
ADDITIONAL FIXES AVAILABLE PLEASE SEE BELOW. Step 1 - Update or Patch your Automate Server
Install a full upgrade to Automate 19 Patch 2 - (Please see your product dashboard on ConnectWise University) (this is the option the majority of the MSPGeek Admin team chose)
Install a patch, without upgrading your version if you're on Automate 19 Patch 1 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 12 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 11 - (We will post this link when we get it)
Step 2 - Update your agents
You can force an update on an agent basis by going into Commands > LabTech > Force remote to Update. We highly recommend though Darren White's excellent internal monitor available here https://www.mspgeek.com/files/file/45-cwa-agent-version-update-monitor/ . To import this, go to Systems > General > Import > SQL and that's all you need to do. The agent update process is cleverly embedded within the monitor, and it will do it in a staged way depending on how big your environment is.
I am on Hosted/Cloud Automate, does this still affect me?
Absolutely - we suggest that you contact support and ask for your instance to be updated ASAP.
What do I do next?
Continue to monitor your agent versions and ensure that affected versions gradually decrease. Get your clients to turn on machines that haven't been on for a few weeks.
Where can I get further help/discuss this further?
We have created a channel called #dday in Slack for discussion specifically relating to this.
ConnectWise has provided a utility to use ConnectWise Control to update your Automate Remote Agents.
Using Control to Update your Automate Remote Agents
This should be used with Automate Server Versions, 126.96.36.1995, 188.8.131.527, 184.108.40.206, 220.127.116.11
When run, the executable prompts the user for a ConnectWise Control server URL, Username and Password, as well as the option to create a session group on the Automate server.
Once entered, the user clicks the "Deploy Hotfix" button. This will:
Install (and enable) a Signed ConnectWise Control extension onto the ConnectWise Control server.
The extension is embedded in the exe and streamed to the Control Server directly. It does NOT use the extension browser/store.
The embedded extension is signed and approved by Control, so it should work with all Control licenses.
The extension includes the the hotfix executable. The remote clients will download the hotfix from their respective Control servers.
It also includes a .bat file that will be run against all Windows machines which handles hotfix need identification as well as downloading and executing the hotfix.
Call the "HotfixCWAOnAccessSessions" endpoint in the extension. This endpoint:
Goes through every ACCESS session in Control (the ones used by Automate for remote access to agents)
For Windows machines it will queue a Control Command that forces the "cmd" interpreter to run for up to 5 minutes to complete the tasks in the included .bat file.
This bat file:
Checks the agent version based on the "HKLM\Software\LabTech\Service\Version" key
If found and it matches one of the known good versions, will report that version and end.
If the version is not found it will check for the affected file in the "%windir%\ltsvc" folder and if found, continue to validate the file. If the file is not found, it will report that Automate is not installed.
If a different version of the Agent is identified (or the affected file is found as mentioned) it will check the file version using wmic. If the build of the version is below 16 it will begin download and execution of the patch file.
It will fail through from using powershell to using a vbs file to attempt to download the executable from the Control server.
Once the file is downloaded, it will run it and then check every second or so for the affected file version. Once it has been found to change it will report the installed Automate version with the affected file version in square brackets following it.
If the file does not update within 30 seconds, it will report that the update failed.
Cleans up the added files used to download/execute the hotfix. (Note - they are written to the %windir%\ltsvc folder)
For non-windows machines, it updates CustomProperty7 with a message that the machine is not a windows system.
There is also an event handler in the extension that will check for the response from the command sent to check/run the hotfix that:
"User Deletes" the original command sent to the remote agent to perform the check/execution.
Note - this simply removes it from the UI - it is still available via Reporting.
Sets CustomProperty7 of the session to the last line returned in the response (which is a status, such as a version number or a failure message).
At this point (after the extension is installed, and the hotfix deployment started) it will notify the user that deployment has started.
If the "Create Session Group" button is checked it will then
Get existing session groups on the Control server.
Check for any with the name "Out of Date Automate"
And also check that it matches the one intended to be setup.
If the same, will report success.
If different (but same name) will report it cannot create the session group, but that hotfix has started.
If no match, add a new session group.
Note that this queues a command, which will run the next time an offline machine connects to the Control server
Note that when the extension is installed, a Control Administrator can also call the Command against all Access sessions from Control by navigating to Admin > Extensions > "ConnectWise Automate: Agent Recovery" > Options > Run On All Machines
To run the tool on specific machines (including non-Access sessions), a Control Administrator can also select one or more sessions on the Host page and use the "Apply Automate Agent Hotfix (Mar 2019)" command using the right-click menu, "••• More" menu, or details panel when more than one session is selected.
Cleanup and Remove Extension:
This link in the executable is to be used once the hotfix has been deployed to all affected agents on a system, and the extension is to be removed.
When clicked, will prompt the user to confirm they want to perform the action.
If confirmed, it will :
Call the "CleanupHotfixExtension" endpoint in the installed extension, which empties CustomProperty7 on all ACCESS sessions.
If Create Session Group is checked
Get session groups from the Control server
Check for "Out of Date Automate" group
If it exists, and it matches the expectation, removes it.
If it exists, but is not the same, does not remove it.
Calls Uninstall Extension for the installed extension.
As a note, the extension can not be uninstalled from the Admin > Extensions page without first being disabled on the Control Server. If uninstall is selected while the extension is active, it will notify the user that it must first be disabled.
See the attached EXE file for download HotfixAutomateVisControl.exe