Jump to content
wearearewe

Maintenance Mode and Internal Monitors

Recommended Posts

We continually get alerts such as:

The first Critical Blacklist Event found: Log: System, ID: 1055, Source: Microsoft-Windows-GroupPolicy, TimeStamp: 2017-02-28 02:53:55

Critical Blacklist Event that was found: Log: DNS Server, ID: 4013, Source: Microsoft-Windows-DNS-Server-Service, TimeStamp: 2017-02-28 02:5

 

This alert and others like it are logged within the Maintenance Mode window. Since Internal Monitors run on data stored in the LT DB after the fact they have no knowledge of Maintenance Mode.

 

Does anyone know how to ignore Internal Monitor Alerts that occur during a Maintenance Mode window?

Share this post


Link to post
Share on other sites

It's been almost a year since I posted this and it appears I am alone with this issue. I am still curious as to what others do about all the Critical Blacklist Event tickets created by this monitor.

  1. Do you have a work around?
  2. Do you bulk close them without review?
  3. Do you bulk close them with review?
  4. Do you ignore them
  5. Have you turned off the monitor?

Share this post


Link to post
Share on other sites

Pretty much, I believe people are just living with it.  If you wanted to define a fixed window, you could modify the SQL to include something like this in the Additional Conditions: 

AND NOT TIME(eventlogs.timegen) BETWEEN '02:00:00' AND '04:00:00'

That should EXCLUDE any event recorded on the agent between 2:00 AM and 4:00 AM. But this would not necessarily reflect when Maintenance Modes were in effect, and would be the same time span for all clients. Beyond that, there isn't much else that can be done.

Share this post


Link to post
Share on other sites

Darren,

Thanks for the reply. What about SQL that pulls the time from the maintenance window? I'm not familiar enough with the data dictionary to figure that one out but I do plan to do some digging now that you planted the seed.

Share this post


Link to post
Share on other sites

Old thread, I know, but...

I've just been looking into this as we got fed up with our "Reboot" script triggering the LT - Offline Servers internal monitor.

We often schedule servers to reboot out of hours using a script

 

The bit of information above from wearearewe (about Internal Monitors not being aware of Maintenance Mode) explained why putting the "Set maintenance mode" function in our reboot script made no difference...

 

However, to get around this, I've added the following to the end of the Additional Condition section for the Offline Servers monitor:

 AND computers.computerid NOT IN (SELECT computerid FROM maintenancemode)

 

After some (admittedly, very limited) testing, this seems to work well...

I'm aware that it doesn't account for "Scripts Only" maintenance mode but, for the time being, that's not a big issue for us and I'm sure the code could be expanded to deal with this too, without much effort

Share this post


Link to post
Share on other sites
20 hours ago, rowanREN said:

The bit of information above from wearearewe (about Internal Monitors not being aware of Maintenance Mode) explained why putting the "Set maintenance mode" function in our reboot script made no difference...

To be clear: NO MONITOR is "aware" of Maintenance Mode. That is not the design/purpose for Maintenance Mode. If your reboot script sets Maintenance Mode, the offline monitor can still return the computer as a result. But the alert will not be acted upon. If your monitor is triggering and opening a ticket or running a script, either your are not setting maintenance mode correctly or your reboot script is clearing maintenance mode too early (before the agent is checking in again).

Share this post


Link to post
Share on other sites

Thanks for the clarification; I understand that Maintenance Mode suppresses the alerts, rather than telling the monitor to ignore certain agents.

However, the underlying issue I referenced in our system is still valid. The Offline Servers monitor is an Internal Monitor and the alerts come through whether the respective agent is in Maintenance Mode or not. 
I have tested this by setting Maintenance Mode manually, via a script (Set Maintenance Mode function), via a script (SQL Execute), and by manually adding a row to the maintenancemode table in the database directly. None of these options cause the alerts to be suppressed. 

wearearewe's point about Internal Monitors not "having any knowledge of" (I now understand this to be, more accurately, "having any regard for") Maintenance Mode, has lead me to a solution that works for us, whereby we tell the monitor itself to ignore any machines which have Maintenance Mode set.

The net effect is exactly what we're after:

- machines which go offline because we told them to reboot do not result in us being alerted that they've gone offline, unless they stay offline after the Maintenance Mode window expires
- machines which go offline outside of this scenario will alert us as normal

Edited by rowanREN

Share this post


Link to post
Share on other sites

This seems strange because we use maintenance mode all the time to suppress offline alerts, etc.  I say this not to dispute what you are seeing, but so that anyone else reading this doesn't come to the conclusion that they need to modify their monitors to make maintenance mode work properly. If maintenance mode isn't working, there must be something else wrong since that's literally it's entire reason for existing. 😁 Monitors that are tripped DURING maintenance should not trigger an alert. (Also, ALWAYS use the script function to set maintenance mode, don't use SQL.  Maintenance Mode design has changed over time, the script function will do the right thing, custom SQL may not.)

The earlier issue mentioned about Event Log monitors and Maintenance Mode is different because event logs generated during maintenance might be uploaded an hour later, and then checked by a monitor several hours later (if the monitor only runs every 4 or 12 hours, etc.). By that time maintenance mode has often been cleared, and the fact that the event logs were generated during that window isn't something the monitors are capable of knowing.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...