Jump to content

RMM Security Best Practices


Recommended Posts

Following the MSPs who were impacted by this https://www.reddit.com/r/msp/comments/ani14t/local_msp_got_hacked_and_all_clients_cryptolocked/ a number MSPGeekers had an impromptu call to discuss security in general and what best practices we all followed to ensure our systems are as secured as possible.

This prompted an idea from @MetaMSP that we have a place where best practices can be defined - things that we can put in place to make our RMMs as secure as possible. I will update this with a list of generally agreed upon methods based on the discussion.

How can I apply better security?

1) Enable Multi-Factor Authentication. This is a functionality that already exists within Automate in the form of plugins, and for the effort to implement it gives a massive boost to security. As an MSP every single account you have should have 2FA on it

2) Do not publish your Automate URL publicly - anywhere. If you are linking to your Automate site, or even your Control site from anywhere on your website - remove it and ensure to the best of your ability it is removed from search engine indexes. Attackers can find servers like this on Google using very simple methods and you will be one of the first they attempt to attack.

3) Review all plugins/extensions installed and disable, remove the ones you no longer use. Having an added benefit of speeding your system up, each of these adds a small risk profile as you are relying on third party code being secure running in the background. Removing plugins you no longer use or need reduces the surface area of attack.

4) Review ports you have open and close ports that are not needed. You will find the ConnectWise documentation here on what ports should be open. https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/020/010/020 . Don't just assume this is right - check. Ports like 3306 (MySQL DB Port) and 12413 (File Redirector Service) should absolutely not be opened up externally.

5) Keep your Automate up to date. ConnectWise are constantly fixing security issues that are reported to them. You may think you are safe on that "old" version of Automate/LabTech, but in reality you are sitting on an out-of-date piece of software that is ripe for someone attacking

6) DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears). <-- Courtesy of @MetaMSP

7) DO ensure that robots.txt is properly set on your Automate server. If you can Google your Automate server's hostname and get a result, this is BROKEN and should be fixed ASAP. <-- Courtesy of @MetaMSP

8 ) Firewall Blocking. I personally block every country other than the UK and the USA from my Automate Server on our external firewall. This greatly reduces your chance of being attacked out of places like China, Korea, Russia etc.

9) Frequently review the following at your MSP

  • Check that the username and passwords set are secure, better yet randomise them all and use a password manager
  • Treat vendors/services that don't support or allow 2FA with extreme prejudice. I will happily drop vendors/services that don't support this. If you 100% still need to keep them, setup a periodic review and pressure them secure their systems because you can almost guarantee if they are not doing customer logins properly that there will be other issues
  • Setup a periodic review to audit users that are active on all systems. PSA, Office365, RMM, Documentation Systems (ITGlue/IT Boost)
  • Audit 3rd Party Access, Consultants and Vendor access to your systems <-- Thanks @SteveIT

10) DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears). <-- Courtesy of @MetaMSP

 

 

  • Like 4
  • Thanks 2
Link to post
Share on other sites

A few suggestions from a self-education and awareness aspect:

  • Have a look at https://www.sans.org/reading-room/whitepapers/compliance/compliance-primer-professionals-33538 - it's nearly 10 years old but is still a good overview of still-applicable regulations and standards.
  • Better yet, register at https://www.sans.org [free] and get access to behind-the-authwall whitepapers similar (and often more recent) than the above.
  • Head over to https://www.us-cert.gov/ncas/alerts and sign up to get regular emails or subscribe to the RSS feed to stay current with impactful vulnerabilities.
  • Layer, layer, layer. No mitigation measure is 100% effective, and to quote @DarrenWhite99:
    Quote

    To be exploited you only need to leave one door open. Being secure is a matter of trying to close every possible door, AND having a plan for WHEN you get robbed. 
    It’s like outrunning a pack of wolves.  Your only chance is to not be the slowest runner in the group.

    That said, however, if we work together as a community to harden the platforms and tools we use as a group, we might just make Automate less of a target overall.

  • Like 4
  • Thanks 1
Link to post
Share on other sites

Going to try and inject a bit more activity in this thread, although what I am doing is merely documenting what others have suggested so far:

  • DO ensure that robots.txt is properly set on your Automate server. If you can Google your automate server's hostname and get a result, this is BROKEN and should be fixed ASAP.
  • DON'T publish your Automate server's FQDN. This includes linking to it from your company's website. That's like a flashing signpost that says "THIS WAY TO HACK ME!"
  • DON'T allow customers downstream access to your Automate server. This exponentially increases your attack surface area as every customer must then be as secure as you must.
  • DO enable multifactor authentication / MFA / 2FA on All The Things. Seriously. Single factor (username/password) authentication is not sufficient in 2019.
  • DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears).
  • DO disable AND REMOVE and unused or unneeded plugins from your Automate environment. Each is another potential attack vector that you are eliminating by removing.
  • DON'T expose port 3306 to your external network interface / the Internet. This is MySQL's default port and is yet another attack vector that you can easily eliminate.

More to follow as it is suggested.

  • Like 2
Link to post
Share on other sites

For us, it's causing us to do a review of...

  • Any 3rd party consulting or services we used that had access into our systems
  • API keys used by our current integrations
  • Any users that don't have MFA enabled
  • Any connections to our NOC systems from foreign/poor reputation IPs

I'd definitely be interested in what others out there are doing in light of this. Would love to have a checklist of sorts coming out of this where I could come back to our team with, showing we are doing our due dilligence.

  • Like 2
Link to post
Share on other sites

 

11 hours ago, Andrew P said:

Does anyone have their Automate server behind a WAF (Web Application Firewall)?? I haven't found really any documentation of Automate behind one.... Would like some insight if possible due to the increasing threats and attacks.

Well, I mean you and I were going to test it out. 

 

When I was asking in slack if others tried it, no one had any input so I think we're on new ground with it. 

Edited by kelliott
Link to post
Share on other sites
On 2/9/2019 at 6:27 AM, kelliott said:

 

Well, I mean you and I were going to test it out. 

 

When I was asking in slack if others tried it, no one had any input so I think we're on new ground with it. 

Oh I am ALL for testing and being new territory, but you would think SOMEone has had to try it somewhere....

Link to post
Share on other sites

Most people should be using the default.

User-agent: *
Disallow: /


I'm sure several people already understand what robots.txt is and does, but I'll elaborate a bit for those that might be out of the loop.

When web robots crawl your site the standard is to first look for <domain>/robots.txt to see if there are any rules that it needs to abide by. This is an industry standard that's been around since the 90s. Some websites use it to prevent certain sections of their website from being indexed online and therefore not searchable in a search engine. In Automate's case we don't want anything indexed, so you'll want to make sure that your robots.txt matches the above text. Granted there are exceptions to every rule, so it's possible that someone out there has a good reason to have a more customized robots script, but I can't think of any reason. This is obviously an oversimplification, so I'll link a more detailed overview below.

For more information:
http://www.robotstxt.org/robotstxt.html

 

Edited by bigdog09
  • Like 1
  • Thanks 2
Link to post
Share on other sites

Other things I am thinking about but do not have answers for yet

  • IIS sites - are any of these considered legacy and not save to remain active? image.png.bed2c49eb169037f7cdcc2265ffc5ccd.png
  • Automate plugins - is there a sanity check we can do or any plugins that we should consider not safe to maintain? I am not aware of any that expose anything publicly in the same way this Kaseya/Manage integration did, but feel it's worth discussing.
Edited by SteveIT
Link to post
Share on other sites

A few more suggestions... simple stuff but maybe worth mentioning:

  • Schedule a security audit by a third party.
  • This is kind of a sub-point to an existing one, but audit and limit administrative accounts. Just because everyone in the room is an "admin", doesn't mean they all have to be your admin. Those that do, need admin accounts apart from their normal ones.
  • Have your security vendors review your policies as new features/recommendations don't always get implemented after release.
Link to post
Share on other sites

Here are a couple of SQL queries to help make sure that your users are set up correctly in Automate.

Make sure that all of your user accounts are being audited for everything they're doing. The logs don't take up much space, so it's worth it to audit everything.

-- Auditing not set to Everything
SELECT `users`.`Name` 
FROM `users` 
WHERE `AuditLevel` != 5 AND `users`.`Name` != 'root'; 

 

 

You should limit your SuperAdmins. Like @danrdj said, just because they have the admin title it doesn't mean that they need full access. Run this below to find your current SuperAdmins and make sure there isn't someone on that list that doesn't need to be there.

-- Users have SuperAdmin
SELECT `users`.`UserID`, `users`.`Name`, IF((BIT_OR(`userclasses`.`Permissions`) & (1 << 0) = (1 << 0)), 'True', 'False') AS 'SuperAdmin', COUNT(`userclasses`.`name`) AS 'Class Count'
FROM `users` 
LEFT JOIN `userclasses` ON FIND_IN_SET(`userclasses`.ClassID, `users`.`ClientID`) 
WHERE  `users`.`Name` != 'root'
GROUP BY `users`.`UserID`;

 

  • Thanks 1
Link to post
Share on other sites
  • 3 weeks later...

Couple of questions

1. I know it's already been asked in here, but are any of you using a WAF or some other way of securing your remote admin capabilities? Everything comes through on the same port, so we're looking into a way of separating out agent traffic while securing remote admin traffic.

2. Are any of you monitoring the user audit logs closely?  We're wanting a monitoring service to comb through those logs and alert us for items we care about.

Edited by bigdog09
Link to post
Share on other sites
  • 1 month later...

We used to block external access to the "Old" Web control center through IIS filtering policies, but now we need to block the new one + the API itself and possibly other components.

As far as I know there is zero documentation on locking down those accesses. Probably due to a focus on Automate Cloud which, by definition, is open to the Internet.

Shouldn't we be able to differentiate agent check-in from privileged API/console access? How about restricting API accounts to specific IP ranges, since they can't be protected by MFA?

It feels like such a security-critical system should provider better control and segmentation. MFA is nice (and necessary) but it isn't sufficient in itself when plugins and integrations are an essential functionality.

EDIT : Here's what I was able to find out so far regarding the IIS sites/folders :

  • aspnet_client : no idea, not much there - access seems disabled
  • automate : new web control center UI
  • crystalreportviewers12 : I'm assuming its related to crystal report (!) - access seems disabled
  • cwa : Connectwise Automate API
  • dashboard-pod : some work in progress? seems like mostly frontend assets
  • LabTech : this seems to be the main point of communication for agent check-in, deployments, etc.
  • WCC2 : Legacy Web Control Center
 
I blocked external access to everything except the LabTech "folder", I guess we'll see 😃
I'm expecting this to break external integrations that communicate through the API (ex: IT Glue), but then I can allow their source IPs assuming they public them.
Edited by exosource
Additional info
Link to post
Share on other sites

I whitelisted internal ranges / localhost, denied everything else. Had to play a bit with the IIS IP filters they didn't behave as expected (deny/allows hierarchy and inheritance). Make sure to confirm it is behaving as expected from internal/external IPs.

Other than that working like a charm, have yet to test API integrations which are likely to need some whitelisting. Anything that uses a Labtech/Automate account for integration will likely come through the API path. Outbound integrations (i.e. Labtech pluging authenticating to external service) should be fine.

  • Thanks 1
Link to post
Share on other sites
  • 8 months later...

Hey Guys, Can anyone tell me if automate can export automate login events via a syslog tool. We are using a SIEM tool to monitor our security and have found automate and connectwise control really lack these features.

Connectwise control:  Does have a syslog extension however it only exports session logs. It should also include logins to the web interface.

Automate: Can't see any native functionality to export login attempts. Please educate me if i am incorrect

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...