Jump to content
Gavsto

RMM Security Best Practices

Recommended Posts

Following the MSPs who were impacted by this https://www.reddit.com/r/msp/comments/ani14t/local_msp_got_hacked_and_all_clients_cryptolocked/ a number MSPGeekers had an impromptu call to discuss security in general and what best practices we all followed to ensure our systems are as secured as possible.

This prompted an idea from @MetaMSP that we have a place where best practices can be defined - things that we can put in place to make our RMMs as secure as possible. I will update this with a list of generally agreed upon methods based on the discussion.

How can I apply better security?

1) Enable Multi-Factor Authentication. This is a functionality that already exists within Automate in the form of plugins, and for the effort to implement it gives a massive boost to security. As an MSP every single account you have should have 2FA on it

2) Do not publish your Automate URL publicly - anywhere. If you are linking to your Automate site, or even your Control site from anywhere on your website - remove it and ensure to the best of your ability it is removed from search engine indexes. Attackers can find servers like this on Google using very simple methods and you will be one of the first they attempt to attack.

3) Review all plugins/extensions installed and disable, remove the ones you no longer use. Having an added benefit of speeding your system up, each of these adds a small risk profile as you are relying on third party code being secure running in the background. Removing plugins you no longer use or need reduces the surface area of attack.

4) Review ports you have open and close ports that are not needed. You will find the ConnectWise documentation here on what ports should be open. https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/020/010/020 . Don't just assume this is right - check. Ports like 3306 (MySQL DB Port) and 12413 (File Redirector Service) should absolutely not be opened up externally.

5) Keep your Automate up to date. ConnectWise are constantly fixing security issues that are reported to them. You may think you are safe on that "old" version of Automate/LabTech, but in reality you are sitting on an out-of-date piece of software that is ripe for someone attacking

6) DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears). <-- Courtesy of @MetaMSP

7) DO ensure that robots.txt is properly set on your Automate server. If you can Google your Automate server's hostname and get a result, this is BROKEN and should be fixed ASAP. <-- Courtesy of @MetaMSP

8 ) Firewall Blocking. I personally block every country other than the UK and the USA from my Automate Server on our external firewall. This greatly reduces your chance of being attacked out of places like China, Korea, Russia etc.

9) Frequently review the following at your MSP

  • Check that the username and passwords set are secure, better yet randomise them all and use a password manager
  • Treat vendors/services that don't support or allow 2FA with extreme prejudice. I will happily drop vendors/services that don't support this. If you 100% still need to keep them, setup a periodic review and pressure them secure their systems because you can almost guarantee if they are not doing customer logins properly that there will be other issues
  • Setup a periodic review to audit users that are active on all systems. PSA, Office365, RMM, Documentation Systems (ITGlue/IT Boost)
  • Audit 3rd Party Access, Consultants and Vendor access to your systems <-- Thanks @SteveIT

10) DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears). <-- Courtesy of @MetaMSP

 

 

  • Like 3
  • Thanks 1

Share this post


Link to post
Share on other sites

A few suggestions from a self-education and awareness aspect:

  • Have a look at https://www.sans.org/reading-room/whitepapers/compliance/compliance-primer-professionals-33538 - it's nearly 10 years old but is still a good overview of still-applicable regulations and standards.
  • Better yet, register at https://www.sans.org [free] and get access to behind-the-authwall whitepapers similar (and often more recent) than the above.
  • Head over to https://www.us-cert.gov/ncas/alerts and sign up to get regular emails or subscribe to the RSS feed to stay current with impactful vulnerabilities.
  • Layer, layer, layer. No mitigation measure is 100% effective, and to quote @DarrenWhite99:
    Quote

    To be exploited you only need to leave one door open. Being secure is a matter of trying to close every possible door, AND having a plan for WHEN you get robbed. 
    It’s like outrunning a pack of wolves.  Your only chance is to not be the slowest runner in the group.

    That said, however, if we work together as a community to harden the platforms and tools we use as a group, we might just make Automate less of a target overall.

  • Like 3
  • Thanks 1

Share this post


Link to post
Share on other sites

Going to try and inject a bit more activity in this thread, although what I am doing is merely documenting what others have suggested so far:

  • DO ensure that robots.txt is properly set on your Automate server. If you can Google your automate server's hostname and get a result, this is BROKEN and should be fixed ASAP.
  • DON'T publish your Automate server's FQDN. This includes linking to it from your company's website. That's like a flashing signpost that says "THIS WAY TO HACK ME!"
  • DON'T allow customers downstream access to your Automate server. This exponentially increases your attack surface area as every customer must then be as secure as you must.
  • DO enable multifactor authentication / MFA / 2FA on All The Things. Seriously. Single factor (username/password) authentication is not sufficient in 2019.
  • DON'T share credentials except in cases of absolute necessity (one login is available ONLY and you can't afford a single point of failure if that one person that knows it disappears).
  • DO disable AND REMOVE and unused or unneeded plugins from your Automate environment. Each is another potential attack vector that you are eliminating by removing.
  • DON'T expose port 3306 to your external network interface / the Internet. This is MySQL's default port and is yet another attack vector that you can easily eliminate.

More to follow as it is suggested.

  • Like 2

Share this post


Link to post
Share on other sites

For us, it's causing us to do a review of...

  • Any 3rd party consulting or services we used that had access into our systems
  • API keys used by our current integrations
  • Any users that don't have MFA enabled
  • Any connections to our NOC systems from foreign/poor reputation IPs

I'd definitely be interested in what others out there are doing in light of this. Would love to have a checklist of sorts coming out of this where I could come back to our team with, showing we are doing our due dilligence.

  • Like 2

Share this post


Link to post
Share on other sites

Periodic checks in the entire ConnectWise suite of the following is a good idea:

  • Plugins or extensions that are no longer needed, or could be replaced with a simple script
  • User accounts left active
  • API keys
  • Service accounts
  • That the CW products are up-to-date

Share this post


Link to post
Share on other sites

Does anyone have their Automate server behind a WAF (Web Application Firewall)?? I haven't found really any documentation of Automate behind one.... Would like some insight if possible due to the increasing threats and attacks.

Share this post


Link to post
Share on other sites

 

11 hours ago, Andrew P said:

Does anyone have their Automate server behind a WAF (Web Application Firewall)?? I haven't found really any documentation of Automate behind one.... Would like some insight if possible due to the increasing threats and attacks.

Well, I mean you and I were going to test it out. 

 

When I was asking in slack if others tried it, no one had any input so I think we're on new ground with it. 

Edited by kelliott

Share this post


Link to post
Share on other sites
On 2/9/2019 at 6:27 AM, kelliott said:

 

Well, I mean you and I were going to test it out. 

 

When I was asking in slack if others tried it, no one had any input so I think we're on new ground with it. 

Oh I am ALL for testing and being new territory, but you would think SOMEone has had to try it somewhere....

Share this post


Link to post
Share on other sites

Most people should be using the default.

User-agent: *
Disallow: /


I'm sure several people already understand what robots.txt is and does, but I'll elaborate a bit for those that might be out of the loop.

When web robots crawl your site the standard is to first look for <domain>/robots.txt to see if there are any rules that it needs to abide by. This is an industry standard that's been around since the 90s. Some websites use it to prevent certain sections of their website from being indexed online and therefore not searchable in a search engine. In Automate's case we don't want anything indexed, so you'll want to make sure that your robots.txt matches the above text. Granted there are exceptions to every rule, so it's possible that someone out there has a good reason to have a more customized robots script, but I can't think of any reason. This is obviously an oversimplification, so I'll link a more detailed overview below.

For more information:
http://www.robotstxt.org/robotstxt.html

 

Edited by bigdog09
  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Other things I am thinking about but do not have answers for yet

  • IIS sites - are any of these considered legacy and not save to remain active? image.png.bed2c49eb169037f7cdcc2265ffc5ccd.png
  • Automate plugins - is there a sanity check we can do or any plugins that we should consider not safe to maintain? I am not aware of any that expose anything publicly in the same way this Kaseya/Manage integration did, but feel it's worth discussing.
Edited by SteveIT

Share this post


Link to post
Share on other sites

A few more suggestions... simple stuff but maybe worth mentioning:

  • Schedule a security audit by a third party.
  • This is kind of a sub-point to an existing one, but audit and limit administrative accounts. Just because everyone in the room is an "admin", doesn't mean they all have to be your admin. Those that do, need admin accounts apart from their normal ones.
  • Have your security vendors review your policies as new features/recommendations don't always get implemented after release.

Share this post


Link to post
Share on other sites

Here are a couple of SQL queries to help make sure that your users are set up correctly in Automate.

Make sure that all of your user accounts are being audited for everything they're doing. The logs don't take up much space, so it's worth it to audit everything.

-- Auditing not set to Everything
SELECT `users`.`Name` 
FROM `users` 
WHERE `AuditLevel` != 5 AND `users`.`Name` != 'root'; 

 

 

You should limit your SuperAdmins. Like @danrdj said, just because they have the admin title it doesn't mean that they need full access. Run this below to find your current SuperAdmins and make sure there isn't someone on that list that doesn't need to be there.

-- Users have SuperAdmin
SELECT `users`.`UserID`, `users`.`Name`, IF((BIT_OR(`userclasses`.`Permissions`) & (1 << 0) = (1 << 0)), 'True', 'False') AS 'SuperAdmin', COUNT(`userclasses`.`name`) AS 'Class Count'
FROM `users` 
LEFT JOIN `userclasses` ON FIND_IN_SET(`userclasses`.ClassID, `users`.`ClientID`) 
WHERE  `users`.`Name` != 'root'
GROUP BY `users`.`UserID`;

 

Share this post


Link to post
Share on other sites
3 hours ago, orbit said:

Where should robots.txt live - I assume we had it (on latest version) but can't see it.

It should be in the wwwroot folder, so most likely c:\inetpub\wwwroot\robots.txt

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×