Jump to content
GeekOfTheSouth

Automate Security Issue- Patch 11 and 12

Recommended Posts

Found an issue and have raised it with CW multiple times... Feel like I am getting blown off. Patch 11 and 12 have a directory traversal bug that is pretty serious... Can any of you reproduce?

Basically this url pattern: http://LTServer.yourdomain.com:12413/..../..../..../..../..../..../..../..../..../windows/win.ini

Allows you to enumerate and download any file on the automate server... ANY file. With no authentication.

  • Thanks 2

Share this post


Link to post
Share on other sites

I can't reproduce it either. My IIS log file shows the following:

2019-01-04 10:32:05 192.168.168.18 GET /..../..../..../..../..../..../..../..../..../windows/win.ini - 443 - 192.168.123.123 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/71.0.3578.98+Safari/537.36 - 404 0 2 15

and I'm on patch 12

As a test I tried creating a test.txt file in the root of the disk and then tried generating a path by successively adding multiple '..../'s without managing to pick up the file in the root.

Is it possible you've modified something in the standard IIS config?

Share this post


Link to post
Share on other sites

Looks like that is over https? It has to be over http and port 12413. I've confirmed it on two different LT servers and CW confirmed last night, finally.

Share this post


Link to post
Share on other sites

hmm, interesting that port is open and listening on the server but I get 404 if I try and open the url. What on earth is that listening on 12413.

In any case, I've repeated the test above, dropping a test.txt in the root of the C disk and then trying 

http://LTServer.yourdomain.com:12413/..../..../test.txt
http://LTServer.yourdomain.com:12413/..../..../..../test.txt
through to 
http://LTServer.yourdomain.com:12413/..../..../..../..../..../..../..../..../..../..../..../..../test.txt

all of which are returning 404

I am only running Automate on my server. Could it be another component?

Share this post


Link to post
Share on other sites
54 minutes ago, imurphy said:

hmm, interesting that port is open and listening on the server but I get 404 if I try and open the url. What on earth is that listening on 12413.

In any case, I've repeated the test above, dropping a test.txt in the root of the C disk and then trying 


http://LTServer.yourdomain.com:12413/..../..../test.txt
http://LTServer.yourdomain.com:12413/..../..../..../test.txt
through to 
http://LTServer.yourdomain.com:12413/..../..../..../..../..../..../..../..../..../..../..../..../test.txt

all of which are returning 404

I am only running Automate on my server. Could it be another component?

I do have screenconnect running on that same server...

Share this post


Link to post
Share on other sites

Meh.. I thought this would be harder to track down given the brainpower in this thread....

start http://localhost:12413/FINDME.txt

Oh Process Monitor... Show me file system accesses containing "FINDME"

KeepCalmAndRunProcMon-Port12413.png.758ce5c11f8ce260bac908ac80810adc.png

And what is LabTech\FileService.exe? It's the new Connectwise Automate File Service (CWAFileService).

Once I knew where it was targeting, I didn't need to guess how to ask for a file:

I was able to access http://localhost:12413/..../Windows/LTsvc/LTErrors.txt and retrieve the file.

So, it does permit directory traversal, although that is basically what it is for. My understanding is that it is used for split server environments to allow the servers to replicate/retrieve files between themselves. As far as I know, it shouldn't be used/accessed by anything external. If someone has that port publicly accessible, then that would definitely not be good.

This part bears repeating.

IF ANYONE HAS THIS PORT PUBLICLY ACCESSIBLE IT IS NOT GOOD!

I really hope that I don't need to explain myself.

  • Like 1
  • Thanks 2

Share this post


Link to post
Share on other sites

I believe the new Connectwise Automate File Service (CWAFileService) was added as of v12 patch 10.

This is a fun one, especially if it is on hosted servers. I still am not able to reproduce this on any of the servers I manage, hosted or otherwise, but none of them are fresh post-patch 10 installs, so maybe that's why. Either way, if it is on hosted servers, I wouldn't want to be on the CWA Cloud team this week...

  • Thanks 1

Share this post


Link to post
Share on other sites
Posted (edited)

Was it Patch 10? That’s probably right, but I had Patch 8 in my head... (I know P8 was when they added support for Duo back in after not supporting it in Patch 6).. But for some reason I was thinking the file service went back that far also. P8 or P10, it’s still a recent addition..

Because it is an unusual port that would generally need to be specifically opened to the public, it SHOULDNT be publicly available. I have seen nothing suggesting it’s accessible for any hosted partners. But a new admin could have read that page and opened it publicly by mistake. And many on-prem environments probably do have it accessible to the internal network (which includes many machines that should NOT have access).

Update - It was definitely Patch 10, per https://university.connectwise.com/university/pageview.aspx?short_name=patch-release-notes

Edited by DarrenWhite99
Confirming Patch 10 release

Share this post


Link to post
Share on other sites

Weird, the example darren posted above doesn't work for me either.

with 

 http://localhost:12413/..../Windows/LTsvc/LTErrors.txt 

I get a 404, even when opening it locally from the console of the server.

Share this post


Link to post
Share on other sites
1 hour ago, imurphy said:

Weird, the example darren posted above doesn't work for me either.

with 


 http://localhost:12413/..../Windows/LTsvc/LTErrors.txt 

I get a 404, even when opening it locally from the console of the server.

 

So a few things that seem to be a requirement:

FileService must be started

Automate 12 Patch 11 or above (possible patch 10)

Must connect over HTTP (12413)

Port must be accessible through the firewall (Windows or 3rd Party)

 

You can read more on directory traversal here:

https://null-byte.wonderhowto.com/how-to/perform-directory-traversal-extract-sensitive-information-0185558/

 

 

Share this post


Link to post
Share on other sites

My Ticket was moved to a board that CW says is not visible to their partners, so I won't be able to update much. The ticket is still not assigned, according to them.

 

I don't really like that, but whatever.

Share this post


Link to post
Share on other sites
1 hour ago, imurphy said:

Weird, the example darren posted above doesn't work for me either. 


 http://localhost:12413/..../Windows/LTsvc/LTErrors.txt 

I get a 404, even when opening it locally from the console of the server.

That worked for me because LTShare was pointing to C:\LTShare. I had trouble getting it to load from powershell, I had to use http://hostname:12413//..../ (Without the extra "/" it seems that all the "..../..../" sequences were stripped)  So it worked for me from Chrome, but it might fail from Edge/IE/Firefox depending on how it wants to present the GET request. If you are getting 404, you ARE getting to the service, and if you are not coming from a trusted IP THAT IS NOT GOOD.

Share this post


Link to post
Share on other sites

I've created a user than only has access to LTShare and am running fileservice as that... Not sure what all that will break, but better than my internal network being able to hit that service.

While doing so, I was running procmon on the server and that service really does some strange things. It looked like it was running through all my scheduled report files over and over again, then it would start enumerating other folders in the windows directory... Very strange.

Share this post


Link to post
Share on other sites

Some points:

  • Good thought on the service account user. From my understanding of the stated/known purpose of the service that shouldn't break it.
  • Putting LTShare on a separate volume is another good mitigation.
  • Through observation I can say that the service monitors for file changes (It seems to catch file create actions, notably it MISSES when I modify a file). I don't specifically know what process might trigger it to scan folders.
  • Why is the server in the same layer2 segment as your LAN? Putting it in a separate subnet/vlan would permit you to apply firewall rules at the L3 boundary.
  • The port is only intended for access by IIS. Even if the server is on your LAN subnet, if you only have one server (not split) then you can use Windows Firewall to permit access from the localhost and block access from all other IPs. (And even if you are split, you can permit access from only the Automate server front ends and block everything else)


 

  • Like 1

Share this post


Link to post
Share on other sites
2 minutes ago, DarrenWhite99 said:

Some points:

  • Good thought on the service account user. From my understanding of the stated/known purpose of the service that shouldn't break it.
  • Putting LTShare on a separate volume is another good mitigation.
  • Through observation I can say that the service monitors for file changes (It seems to catch file create actions, notably it MISSES when I modify a file). I don't specifically know what process might trigger it to scan folders.
  • Why is the server in the same layer2 segment as your LAN? Putting it in a separate subnet/vlan would permit you to apply firewall rules at the L3 boundary.
  • The port is only intended for access by IIS. Even if the server is on your LAN subnet, if you only have one server (not split) then you can use Windows Firewall to permit access from the localhost and block access from all other IPs. (And even if you are split, you can permit access from only the Automate server front ends and block everything else)


 

It is on a DMZ vlan, and I definitely could lock it down at the ASA... But I would rather just stop the service from answering. There are other servers on that vlan that I don't fully control security on.

Same with windows firewall... Ours is maintained by GPO, and I don't have exclusive access.

Curious, how does putting the LTShare on another volume stop the service from doing this?

 

Share this post


Link to post
Share on other sites

@GeekOfTheSouth , I apologize that your report was not properly escalated.  We were able to track your original ticket down and it appears that it was closed waiting on response.  The ticket you opened on Thursday has been escalated to T3 support.  We are going to pull that directly into development.   We take security reports seriously, and I want to make sure they are escalated to development to be handled as soon as possible.

 

On the issue that you are reporting:

 

  1. Our documentation is in error.  The file reporting service is purely meant for communication on a single server via localhost or in a split web server installation between the web servers hosted in a DMZ and the Automation Server.  In both cases the IIS worker processes communicate via this port to download/upload files from the Automation server.  At no time should port 12413/TCP be opened to any other systems just as we recommended that MySQL 3306/TCP is also closed. 
  2. We have requested that documentation be changed be to avoid other partners configuring their servers in this way.
  3. We have verified with our cloud team that instances maintained or created by them have 12413/TCP firewalled.
  4. We have verified that implementation documentation does not list 12413/TCP as a required open port for Automate servers.
  5. Our architecture team has reviewed the traversal behavior and we are working to address that issue in an upcoming patch.
  6. We are also going to separately assess the file service communication to increase security between Web Servers in a DMZ environment and the Automate server for further enhancements.

 

If anyone has open access to 12413/TCP configured to their Automate server, we recommend that it be closed as soon as possible.  We are assessing our options internally to identify partners that may have their servers with this port open so we can reach out to them directly. 

 

While we failed to get the original ticket escalated due to issues reproducing the problem, Thursday’s ticket was moving through the proper escalation path. Development relies on the reproduction steps generated by T3 as an important requirement to quickly analyze and solve issues.  If you feel you are not getting a response please touch base with your Account or Support Manager and they will directly escalate issues to product so we can look into the ticket to get the reproduction steps we need.  We also ask that before publically disclosing potential vulnerabilities that you consider the impact on the Automate community of a zero day disclosure.

 

 

  • Like 2
  • Thanks 3

Share this post


Link to post
Share on other sites
39 minutes ago, GeekOfTheSouth said:

Curious, how does putting the LTShare on another volume stop the service from doing this?

It is a mitigation to limit the potential for unauthorized data access. You were able to escape LTShare and access other files, but only on the same drive. If the drive only has LTShare data, then only LTShare files can be accessed. It's not great, but it would limit any potential exposure of Windows system files, MySQL Data, etc.

Share this post


Link to post
Share on other sites
13 minutes ago, srproductmanager said:

@GeekOfTheSouth , I apologize that your report was not properly escalated.  We were able to track your original ticket down and it appears that it was closed waiting on response.  The ticket you opened on Thursday has been escalated to T3 support.  We are going to pull that directly into development.   We take security reports seriously, and I want to make sure they are escalated to development to be handled as soon as possible.

 

On the issue that you are reporting:

 

  1. Our documentation is in error.  The file reporting service is purely meant for communication on a single server via localhost or in a split web server installation between the web servers hosted in a DMZ and the Automation Server.  In both cases the IIS worker processes communicate via this port to download/upload files from the Automation server.  At no time should port 12413/TCP be opened to any other systems just as we recommended that MySQL 3306/TCP is also closed. 
  2. We have requested that documentation be changed be to avoid other partners configuring their servers in this way.
  3. We have verified with our cloud team that instances maintained or created by them have 12413/TCP firewalled.
  4. We have verified that implementation documentation does not list 12413/TCP as a required open port for Automate servers.
  5. Our architecture team has reviewed the traversal behavior and we are working to address that issue in an upcoming patch.
  6. We are also going to separately assess the file service communication to increase security between Web Servers in a DMZ environment and the Automate server for further enhancements.

 

If anyone has open access to 12413/TCP configured to their Automate server, we recommend that it be closed as soon as possible.  We are assessing our options internally to identify partners that may have their servers with this port open so we can reach out to them directly. 

 

While we failed to get the original ticket escalated due to issues reproducing the problem, Thursday’s ticket was moving through the proper escalation path. Development relies on the reproduction steps generated by T3 as an important requirement to quickly analyze and solve issues.  If you feel you are not getting a response please touch base with your Account or Support Manager and they will directly escalate issues to product so we can look into the ticket to get the reproduction steps we need.  We also ask that before publically disclosing potential vulnerabilities that you consider the impact on the Automate community of a zero day disclosure. 

 

 

Thank you... Sorry about posting publicly, but I was a little afraid it was going to go down the same path and the developer wouldn't be able to reproduce. Will my ticket still be updated when a resolution is in place?

Share this post


Link to post
Share on other sites
16 minutes ago, srproductmanager said:

We also ask that before publically disclosing potential vulnerabilities that you consider the impact on the Automate community of a zero day disclosure.

 

 

I completely agree, which is precisely why there should be a proper structure in place for reporting security vulnerabilities. Losing vulnerabilities because a support ticket got closed because a partner didn't respond is serious amateur hour stuff. This is also the second time I know of it has happened (one of my privately reported ones got lost in the same way, mostly because the initial support engineer could not comprehend what I was trying to raise).

I implore ConnectWise to put a proper procedure in place for reporting security vulnerabilities allowing for responsible disclosure. In the mean time at least train the existing staff to escalate anything like this immediately to the appropriate resource.

  • Like 7

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×