Jump to content
sstanley

Quick and Dirty guide to having your server delisted by Google

Recommended Posts

Who am I and why should you take my advice? I go by Scott [Striven] on the slack. I work for a fairly large security-focused MSP.  I've been an Automation Engineer for a couple years now, and have experience that dates back decades in IT and Security, and I have a history of creating automation to combat malware and facilitate server and workstation repairs. With that out of the way..

Did you know your automate server by default is crawled by Google's search algorithm? This means someone can find your automate server via the appropriate google search. I don't believe it's a coincidence that our elastic cluster found zero hits from the IP that was scanning everyone's automate servers. I believe it's due to this configuration change, so it's basically something that I feel I need to share with the community at large.

We discovered this a few years back, but it's extra relevant with the advent of the recent attacks on Automate servers, so I thought I'd share with you the steps we took to mitigate this problem. A quick shoutout to Alexis at https://www.rushordertees.com who provided us with the guidance to do this, since she basically handles SEO optimization over there, and this is definitely an SEO trick. 

 

Note the customHeaders in our web.config for our Labtech front end. 

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <urlCompression doStaticCompression="false" doDynamicCompression="false" />
        <httpProtocol allowKeepAlive="true">
            <customHeaders>
                <add name="X-Robots-Tag" value="noindex" />
            </customHeaders>
        </httpProtocol>
        <httpErrors errorMode="Custom" />
    </system.webServer>
</configuration>

The X-Robots-Tag is a much more efficient way of tagging a site as not to be indexed than the robots.txt, and when using this method, robots.txt should be removed, if it exists.

More information on the technical details of X-Robots-Tag can be found here: https://developers.google.com/search/reference/robots_meta_tag#xrobotstag

That's it! Quick and easy.

 

  • Like 1
  • Thanks 2

Share this post


Link to post
Share on other sites
Posted (edited)

@tlphipps If you'd like to test it you can use the Chrome developer tools; go to the network tab and reload a page in the webUI. Then click an item and look at the headers section:

Screenshot_2020-07-09_21-23-07.png

Edited by Automation Theory
  • Thanks 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...