Jump to content

Server Passwords, the myth, the truth, the....I got nothing


Recommended Posts

Recently Connectwise released a statement recommending Server Passwords for Agent Signup on their Automate be rotated two times. People obviously have concerns about this. This post will strive to answer all questions and concerns regarding this notice.

Q: What happens when I change this password?
A: To answer this you have to understand how Agent Communication works, the agent during signup uses the Server Password to create what's called an Agent Password. This Agent Password is unique to all agents that are communicating with the server, and is what is actually used for communicating. After signup the Server Password is never used again (although still stored in registry) unless a situation arises where the Agent Password is broken forcing the agent to perform a re-signup

Q: What is this nonsense about rotating twice, do I need to? How long do I wait before rotating twice?
A: The server password was designed much like the AD Computer Password for a Domain. When you rotate the password, all agents online are immediately updated with the new password however the old password is moved into the Config table under "OldSystemPassword". Agents with the old password are still allowed to signup therefore to be safe you want to change it twice to wipe out the previous possibly compromised password completely. Alternatively you can change the OldSystemPassword in SQL without rotating it twice, but really it does the same thing. Therefore you DO NOT need to wait before updating the password the second time, as agent communications work independently of the server password and will therefore update the latest password without having the "current" old password. Please note that agents that are offline at the time will still work when they come back assuming the agent password is correct and matches what's in the database for that agent. 

Q: What does this mean to re-install the Probe?
A: The probe agent needs to be completely reinstalled. Yes the entire agent. This is only if you use the probe for agent deployment. If you don't or don't care about it then don't worry about. Please note running the Redo-LTService commandlet from the LTPosh Module does count as reinstalling the agent. Make sure you remove it from being a probe first.

Q: WHAT LTPOSH WAIT HOW IS THAT SECURE?
A: Just like any other agent installer, don't leave it lying around.You can use the -ServerPassword along with any of the install commands to install the Agent. Additionally for those people who like to feel extra warm under additional layers you can safely rotate the OldSystemPassword value in a script via SQL and then pass that into the -ServerPassword parameter. As explained above, the value in OldSystemPassword will still work for Agent Signup. That way you can safely rotate this signup password without breaking probes or legitimate agent re-signups all the time.

Q: What should I make my new server password?
A: The SQL Table is limited to 16 Characters max. You can go longer but its unlikely to actually keep anything beyond the first 16 Characters. Almost all characters are supported. While I haven't tested any, the special characters that I know have worked are listed below.

` ' " \ [ ] { }

 

Please speak to Darren regarding testing alternative uses with LTPosh that will be merged into the main branch once testing is complete. This will involve using a temporary token instead of either current or old System Passwords. You can see more https://mspgeek.slack.com/archives/C1YPT4QT1/p1592842394495500?thread_ts=1592840762.450400&cid=C1YPT4QT1 assuming the Slack Overlords are nice to you.

  • Like 1
  • Thanks 1
Link to post
Share on other sites
  • MGreen pinned this topic
On 6/22/2020 at 11:42 AM, MGreen said:

After signup the Server Password is never used again (although still stored in registry)

Do you mean the agent password is stored in the registry?  The value of HKLM\SOFTWARE\LabTech\Service\ServerPassword appears to be unique when comparing several different agents.

On 6/22/2020 at 11:42 AM, MGreen said:

you can change the OldSystemPassword in SQL

Where is OldSystemPassword located in the SQL database?

Am I the only one that wants answers to questions before blindly changing the system password?

  1. How would the system password get compromised in the first place?
  2. What are the consequences/risks of having the system password compromised?  From the explanation above, it would appear that it would simply allow the attacker to install an Automate agent.
  3. What is the risk of having a compromised agent?  It would seem to me that agents should be inherently untrusted and shouldn't be able to compromise the Automate server or other agents
  4. Is changing the system password just a temporary fix that needs to be repeated regularly in the future or is it just a one-time fix to be completed after applying the recent hotfixes?

The communications from ConnectWise have been light on details and I'm not seeing a lot of chatter about this on the MSPGeek or CW University forums.

Link to post
Share on other sites
On 6/22/2020 at 10:42 AM, MGreen said:

Recently Connectwise released a statement recommending Server Passwords for Agent Signup on their Automate be rotated two times. People obviously have concerns about this. This post will strive to answer all questions and concerns regarding this notice.

 

Source citation please? And, what is the risk of someone getting their hands on the server password from an agent installer lying around?

Edited by BlueToast
Link to post
Share on other sites
On 6/23/2020 at 10:52 PM, Greg-ATG said:

Do you mean the agent password is stored in the registry?  The value of HKLM\SOFTWARE\LabTech\Service\ServerPassword appears to be unique when comparing several different agents.

Where is OldSystemPassword located in the SQL database?

Am I the only one that wants answers to questions before blindly changing the system password?

  1. How would the system password get compromised in the first place?
  2. What are the consequences/risks of having the system password compromised?  From the explanation above, it would appear that it would simply allow the attacker to install an Automate agent.
  3. What is the risk of having a compromised agent?  It would seem to me that agents should be inherently untrusted and shouldn't be able to compromise the Automate server or other agents
  4. Is changing the system password just a temporary fix that needs to be repeated regularly in the future or is it just a one-time fix to be completed after applying the recent hotfixes?

The communications from ConnectWise have been light on details and I'm not seeing a lot of chatter about this on the MSPGeek or CW University forums.

The OldSystemPassword is in the config table alongside the SystemPassword. If you don't recognize what I"m talking about then you shouldn't be messing with it and should stick with the official Connectwise Recommended methods (see https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/070/260/050)

The specifics of the risks involved are not discussed for obvious reasons. Suffice it to say it is a security risk and the recommendation from Connectwise via an email to the partners who had suspected invalid compromised agents.

If you didn't receive this email then you can make the decision yourself to either play on the safe side and change it anyways or assume you're okay and don't change it.

I assure you there is a lot of chatter about the available public details regarding this on our Slack team. Feel free to join there and ask the questions you have if you still need further clarification.

Please note that Darren has released the details of his Token generating script that will enable Temporary deployment passwords with LTPosh.

@BlueToast the server password is stored in plain text in the MSI

Link to post
Share on other sites
23 hours ago, plarue said:

What about group policy agent deployment. Will the MSIs that are sitting out there still work? Do they need to be replaced?

Any installers previously used after the password change will no longer work. So yes, you will need to re download and update your GPO with new msi

  • Thanks 1
Link to post
Share on other sites
On 6/25/2020 at 6:23 PM, MGreen said:

The specifics of the risks involved are not discussed for obvious reasons.

Security through obscurity is not considered an appropriate approach to security.  Even without going into details, it should be possible for you to explain in general terms what the security risks of having a compromised agent are.  If there are in fact risks of having a compromised agent, then that would imply that Automate is an inherently insecure platform that partners deserve to know about (which is why this is an appropriate discussion to have here for other partners to see rather than privately within Slack).

  • Like 1
Link to post
Share on other sites
On 6/23/2020 at 7:52 PM, Greg-ATG said:
  • How would the system password get compromised in the first place?
  • What are the consequences/risks of having the system password compromised?  From the explanation above, it would appear that it would simply allow the attacker to install an Automate agent.
  • What is the risk of having a compromised agent?  It would seem to me that agents should be inherently untrusted and shouldn't be able to compromise the Automate server or other agents
  • Is changing the system password just a temporary fix that needs to be repeated regularly in the future or is it just a one-time fix to be completed after applying the recent hotfixes?

First, some basics of vulnerability classes and severity of risk.
Any connection to your server from a host other than the server is a remote connection. A request from the server is local.
Any request to retrieve or to submit information may include some token identifying the session. A session without any identifying token/secret is unauthenticated. A session with a valid secret is authenticated.
Different URLs/endpoints/services/etc. on the server accept different types of requests, different types of authentication, and may impose different restrictions based on the identity of the session. These are based on privileges or permissions.

The classes of vulnerabilities from greatest risk to least are ones that can be exploited by an attacker that is:
Remote, Unauthenticated
Remote, Authenticated, Unprivileged
Remote, Authenticated, Privileged
Local, Unauthenticated
Local, Authenticated, Unprivileged
Local, Authenticated, Privileged

Agents are authenticated by their unique ID plus a unique (per-agent) password. A "compromised" agent is an agent whose unique password is known by a third party.  Having this password means that requests to the server are AUTHENTICATED Remote connections.  Agents are low privilege, only able to update or request information associated with themselves.

The recently discovered vulnerabilities were exploitable by a remote authenticated attacker, authenticating as an agent (using the agent's password).
To obtain an agent password an attacker must register as an agent, which requires the server password.  The ability for an unauthenticated remote user to obtain the server password is what made the exploit be considered to be vulnerable to an unauthenticated attacker. This is why the first response was denying access to the server password so that the risk was limited to only existing agents. Then the patches resolved the discovered vulnerabilities based on having an agent password.

So, case closed? No more worries? Well.... The patches resolved these KNOWN vulnerabilities. These vulnerabilities were not "inherent", as in the security was not based on the agent being trusted to "behave". But programming mistakes still allowed a misbehaving agent to trick the server. The nature of a complex product is that there may be other undiscovered flaws.

If a user's password was compromised, wouldn't you require the password to be reset?  Removing nonfunctional agents is the same thing, a precaution to insure that if that agent's password has been compromised we are "resetting" it so that it can no longer be used. I hope this clears up what the general risks are to having a compromised agent password.

Directly answering your 4 questions:

  • The system password is included in every installer bundle, so anyone able to obtain an installer package has access to the server password.  Until you change it, anyone that may have collected it in the past can make use of it in the future.
  • The risks of an attacker having the system password are simply that they can move from unauthenticated attacks to authenticated attacks, as they are able to establish a password and authenticate as an unprivileged agent user.
  • The risks of a compromised agent password are simply that if other vulnerabilities exist that can be exploited by an unprivileged agent, an attacker already has valid credentials.
  • As long as the server password permits an agent to register (probably not changing unless the entire process is changed) it's reasonable to consider periodic rotation to become considered a best practice to increase security. (My own opinion)

The bottom line: Regardless of whether or not you consider Automate to be an "inherently" insecure platform as long as a compromised agent is present, a server is inherently more vulnerable to a potential future attack.

  • Thanks 4
Link to post
Share on other sites

 

On 6/24/2020 at 7:38 PM, BlueToast said:

Source citation please

There is one now...

https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/070/260/050

"product team recommends the system password reset cycle is performed any time agent installers are suspected to have been attained by unintended parties"

Well that explains why they broke the direct agent download URL.

 

https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Supportability_Statements/Automate_Security_Release_July#On-Premi
Patch 7 is out. Note the part about "The 2020.7 and the 2019.12 patches address issues that prevent agents from receiving the new system password after a reset was performed." So hopefully no one ran into that problem last month...

  • Thanks 1
Link to post
Share on other sites
  • 4 weeks later...

Hey all,

my understanding of the below is to disable probe on the device and then simply re-enable it again post 2020.7 patch.  I'm finding that there is an existing config already in place for the probe once I do this but I'm assuming that's all it takes to redo the agent deployment method with the new system password in place.  Any help is appreciated as I have quite a few sites to do one at a time unless there's a better way.

Q: What does this mean to re-install the Probe?
A: The probe agent needs to be completely reinstalled. Yes the entire agent. This is only if you use the probe for agent deployment. If you don't or don't care about it then don't worry about. Please note running the Redo-LTService commandlet from the LTPosh Module does count as reinstalling the agent. Make sure you remove it from being a probe first.

Link to post
Share on other sites
  • 2 months later...

We have very few Macs, and use CWA primarily to have CWC installed for remote control when requested.  Found out today the Mac agents haven't checked in since July.

/usr/local/ltechagent/agent.log every 10 seconds:

<info> 10/02/2020 11:25:07 (agent.c:517) Version: 200.002
<info> 10/02/2020 11:25:07 (agent.c:485) Loading agent state.
<info> 10/02/2020 11:25:07 (agent.c:623) Signed in with id 2408.
<error> 10/02/2020 11:25:07 (agent.c:635) Failed to unencrypt LabTech agent password.

Fix - delete the state file:
sudo rm -f /usr/local/ltechagent/state

CW told me to restart the agent but I didn't have to on the one I was able to run this on so far.

Link to post
Share on other sites
On 10/3/2020 at 9:03 AM, SteveYates said:

We have very few Macs, and use CWA primarily to have CWC installed for remote control when requested.  Found out today the Mac agents haven't checked in since July.

/usr/local/ltechagent/agent.log every 10 seconds:

<info> 10/02/2020 11:25:07 (agent.c:517) Version: 200.002
<info> 10/02/2020 11:25:07 (agent.c:485) Loading agent state.
<info> 10/02/2020 11:25:07 (agent.c:623) Signed in with id 2408.
<error> 10/02/2020 11:25:07 (agent.c:635) Failed to unencrypt LabTech agent password.

Fix - delete the state file:
sudo rm -f /usr/local/ltechagent/state

CW told me to restart the agent but I didn't have to on the one I was able to run this on so far.

Thank you....  this fixed my mac clients also.  No need to restart the service, they just check in fine once the state file is removed.

Link to post
Share on other sites
On 10/2/2020 at 6:03 PM, SteveYates said:

Failed to unencrypt LabTech agent password.

My understanding from the initial discussion was this was due to the server password change in July.  All four Macs were working and stopped again this week with the same error.  My ticket was marked/renamed:

KNOWN ISSUE: Ticket #13960362 regarding "Posix agents randomly log "Failed to unencrypt LabTech agent password."(13659726)".

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...