Jump to content
MSPGeek Admin Team

URGENT - Automate update needed by March 9th 2019

Recommended Posts

What exactly is the problem?
One of the DLLs critical to encryption in the affected versions has a bug that means on the 9th of March it will stop working, subsequently taking all services that rely on it with it (Control Center/Agent/Automate Server).

How can I tell if I am affected?
Step 1 - Check your server version
In your Automate Control Center, go into Help and go into About. You want to pay attention to the Current Control Center, Automation Server, Web Server and Remote Agent versions. The following versions are affected:

Automate 12 Patch 11 - 12.0.474 (AKA 12.0.11.474)
Automate 12 Patch 12 - 12.0.496 (AKA 12.0.12.496)
Automate 2019 Patch 1 - 19.0.37 (AKA 19.0.1.37)

If you are on Automate 12 Patch 10 or below you remain unaffected and need to do nothing else.

Step 2 - Check your Automate Agent versions
This is the version of Automate that gets installed on your Windows endpoints. Mac and Linux agents are not affected. Even if you install the latest Automate patch, unless you make sure the agents are updated too you will still lose agents.

In your Automate Control Center, go into Automate then Dataviews. Go in the Status folder and open Agent Status. Pay attention to page size and total in the top right. We recommend upping the page size to view all agents. You can see the current agent installed version in the Agent Version column. You can use the dropdown above the Agent Version column header to see what unique agent versions you have.

Any Windows agent in this Dataview on version 120.474, 120.496 or 190.37 by March the 9th will be permanently lost and will require a reinstall

For those of you who prefer an SQL method (please note this query won't be accurate AFTER patching unless you do the full patch)

SELECT `name` AS ComputerName,clientname,locationname,computers.computerid,serviceversion,lastcontact,IF((LastContact > DATE_ADD(NOW(), INTERVAL - 15 MINUTE)),"ONLINE","OFFLINE") AS AgentStatus FROM computers INNER JOIN v_xr_computers ON computers.computerid = v_xr_computers.computerid WHERE serviceversion > 120.451 AND serviceversion < 190.58

How do I fix this?
Step 1 - Update or Patch your Automate Server

Install a full upgrade to Automate 19 Patch 2 - (Please see your product dashboard on ConnectWise University) (this is the option the majority of the MSPGeek Admin team chose)
Install a patch, without upgrading your version if you're on Automate 19 Patch 1 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 12 - (We will post this link when we get it)
Install a patch, without upgrading your version if you're on Automate 12 Patch 11 - (We will post this link when we get it)

Step 2 - Update your agents
You can force an update on an agent basis by going into Commands > LabTech > Force remote to Update. We highly recommend though Darren White's excellent internal monitor available here https://www.mspgeek.com/files/file/45-cwa-agent-version-update-monitor/ . To import this, go to Systems > General > Import > SQL and that's all you need to do. The agent update process is cleverly embedded within the monitor, and it will do it in a staged way depending on how big your environment is. 

I am on Hosted/Cloud Automate, does this still affect me?
Absolutely - we suggest that you contact support and ask for your instance to be updated ASAP.

What do I do next?
Continue to monitor your agent versions and ensure that affected versions gradually decrease. Get your clients to turn on machines that haven't been on for a few weeks. 

Where can I get further help/discuss this further?
We have created a channel called #dday in Slack for discussion specifically relating to this.

  • Thanks 2

Share this post


Link to post
Share on other sites

Hi there, I work for ConnectWise. We have been working to solve an issue discovered by the LabTech Geek/MSP Geek Community Administrators, for the past week. We take this issue seriously and have been working to build and test solutions for all impacted versions.

Automate partners on the following versions are affected :

• Automate 2019.1 [19.0.1]

• Automate version 12 patch 12 [12.0.12]

• Automate version 12 patch 11 [12.0.11]

An update must be applied to your Automate server that addresses how we handle the transition to our new code signing certificate. Failing to address this issue this month will result in the failure of agent and control center communication.

Next Steps:

- All Automate cloud partners will be upgraded to 2019.2 next week to resolve this issue.

- On premise Automate administrators running impacted versions will be receiving emails today with options to address this issue.

If you have additional questions about this issue, we have created a forum to answer and share related questions here. For further critical updates, please visit here and select the Critical Updates tab.

 

Share this post


Link to post
Share on other sites

Update status:

 

  • We have over 800 partners on the full 2019.2 patch since it was released to general availability last Wednesday
  • We will be upgrading all cloud partner instances this week as part of their monthly maintenance window
  • We have seen good adoption of hotfixes across all impacted version since their release last Friday
  • There are no current support trends related to the 2019.2 release or the hotfixes

 

If you have not received an email and are running an impacted version listed above, we do not have a valid Automate Administrator email address for your instance.  We recommend that you apply one of the resolutions option above for your server as soon as possible.  We are tracking all impacted partner instances and have plans in place to contact and push out patches to those that remain unpatched.  Our goal is to have all partners servers resolved by the Friday the 22nd.

 

We will continue to use this forum to update you on progress related to this issue.

 

Sincerely,

ConnectWise Automate Product Management

Share this post


Link to post
Share on other sites

We, as a team, have noticed over the last couple of days that there has been some pretty heavy criticism of ConnectWise across forums, Slack, and social media outlets like Reddit in relation to the problems with the Automate binaries. A lot of the criticism has been relatively unfounded. The problems found within the product are not issues that we believe could have been picked up by any standard QA or testing measures - bearing in mind that our team member only stumbled on to this by complete accident.

In regards to the criticism surrounding ConnectWise's notification to partners, it's much easier for the leaders in a community like ours to notify our members as we are not bound by the processes, procedures and multiple teams required to get a fix like this out successfully at a corporate level. Though there are clear areas for improvement in QA and testing processes, every member of the MSPGeek team was impressed at the speed of response and subsequent delivery of a fix by the Automate team.

Our community was founded and based upon the idea of mutual assistance, open sharing, good communication and, for the past 6 years, has provided a trusted platform for users to support each other while helping ConnectWise make the product better. Let's continue to help ConnectWise by providing constructive criticism while helping them, and each other, through this bump in the road; it's the MSPGeek way.

  • Like 5
  • Thanks 3

Share this post


Link to post
Share on other sites

Update Status:

• More than 4,100 partners have completed the full 2019.2 patch since it was released earlier this month

• All cloud partner servers have been updated to 2019.2

• We have seen good adoption of hotfixes across all impacted versions since they were released on 2/8/19

• Currently there are no negative support trends related to the 2019.2 release or the hotfixes

• New messaging will be sent out today to Automate administrators and primary contacts for partners that are still impacted by this issue 

We will continue to message impacted partners until all partners have completed the patch. We are tracking all impacted partner instances and have plans in place to push out hotfixes to those that remain unpatched if required. Our goal is to have all partners’ servers resolved by Friday, 2/22/2019.

We will continue to use this forum to update you on progress related to this issue.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.

×