Jump to content
DarrenWhite99

Monitoring disk space - Adding Disk Percentage Monitors to agents

Recommended Posts

The monitors are separate for each drive, but we've experienced issues with the daily agent maintenance script resetting any overrides we've made. We also have a lot of issues with those monitors created by agent maintenance reporting wildly false positive alerts/tickets. We'll get a ticket for a server that says C: has 1 MB of free space available but upon looking at what is getting reported in LT we find the server has 100's of GB free (and we know it didn't temporarily dip down below that).

 

I don't know about the false positives, but you should be able to make basic edits to the created monitors. Looking at the script that adds/maintains them, "Agent Monitor Creation - Disk", it goes through these steps:

1. Retrieve list of Drive IDs (not Drive Letters) on the agent. (Have you ever seen an agent where the USB drive was swapped, and they show multiple "F:" drives and they are all red/offline except for one? Each one is a different drive id)

2. Check to see if a monitor has been created for that DriveID. If the C: drive was re-detected for some reason, it will get a new drive ID and you will see the drive listed twice. It will also end up creating a new monitor for this drive's letter.

3. Check if the monitor ID remembered exists, if it doesn't then re-add it.

4. Check that the monitor's "What" action is "3". 3 corresponds to "Greater Than", so if you tried to change the monitor to be normal when "Greater Than/Equal" 10240MB instead of 1024MB, it will re-create the monitor. If you leave the comparison to "Greater Than" it should not re-create the monitor.

 

We addressed the uselessness of the built in monitors by using Drive percentages instead of MB Free. I copied the "Agent Monitor Creation - Disk" script and tweaked it to create a monitor that for free space, and the percentage is automatically calculated. My script does the following:

1. Retrieve list of Drive IDs on the agent.

2. Check to see if a monitor has been created for that DriveID. If it has I move on. If you deleted this monitor, then you can manually re-create it.

3. Take the current free percentage, cut it in half, then round down to the next lowest multiple of 5. So a drive with 75% free would calculate to 35%, a drive with 22% free would calculate to 10%. It has an extra check to prevent setting to anything lower than 10%, in case of drives that are already over 90% full.

4. Create the monitor using the percent threshold calculated above.

When the monitor is tripped, we can just change the percentage to whatever we want, but we cannot just close the ticket, we need to return it to a normal state via adjustment to the monitor or by making enough free drive space available. That way the ticket gets auto-closed, and the monitor resets. Otherwise we wont get another warning until the drive gets under 1GB (by the default Agent Disk Monitors created by the LabTech script).

 

I also run a second script that looks for invalid monitors. Otherwise we ended up with monitors for drives that were removed or duplicate monitors for the same drive letter. I clean off the built in disk free and my custom disk percentage monitors, and any automatically created monitors that were duplicated. I run each of these at night in our Daily Maintenance scripts. They are offline scripts, and they run through pretty quickly... I have attached both scripts here.

 

Hope they can help!

LTAgentMonitorDiskPercentFree-Creation and Cleanup Scripts.zip

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites

I split the above post into it's own topic, since it has attached files.

Someone asked me about specifying a percent threshold for specific clients, instead of using a calculated threshold as I have it configured. We accomplished this by adding these three lines:  

29,  SET:  @HighMaintenanceClients@ = 1,5,22
30,  IF  %clientid%  Not In  @HighMaintenanceClients@  THEN  Jump to line 32 
31,    SET:  @DrivePercent@ = 5

(Add the above after the existing line 28)

Of course, if you are set on using the same threshold for every client, always, you could just disable line 27 (Get SQL Value) and change line 28 to set @DrivePercent@ to any number you want.

Share this post


Link to post
Share on other sites

Hey mate, thanks so much for these, helped push me in the direction I needed. One question though, do you know the magics required to make these remote monitors data collectors?

Share this post


Link to post
Share on other sites
On 10/23/2018 at 4:49 AM, Jacobsa said:

Hey mate, thanks so much for these, helped push me in the direction I needed. One question though, do you know the magics required to make these remote monitors data collectors?

Baaah!  Lazy!!  Geez... SQLSpy has the answer you seek.

Enabling data collection for a disk space check where the Remote Monitor ID is "INSERTMONITORID" and the computer ID is "INSERTCOMPUTERID" triggers the following query:

Insert Into DataCollectors (AgentID,ComputerID,ValueName,UnitName,YValueID,`MaxValue`) Values(INSERTMONITORID,INSERTCOMPUTERID,'','',0,'') On Duplicate Key Update ValueName=Values(ValueName),UnitName=Values(UnitName),`MaxValue`=Values(`MaxValue`),YValueID=Values(YValueID);

So there is nothing in the agents table itself, you just need a row in the DataCollectors table. You should be able to adjust the script to call the Data Collector query as well when it adds a disk monitor. (You will need to collect the newly created monitor ID first, but IIRC the script is already doing that for you...)

  • Thanks 1

Share this post


Link to post
Share on other sites

Thx Darren for this great Script and your help fixing it for regional issues. We had the issue with the Decimal symbol at Variable @sqlPercent@ Line 27  (because of German Region it's semicolon instead of dot)

For Fixing it just round the Percent Declaration at Line 19:

Replace  `((d.free/d.size)*100) AS percent` with `ROUND(((d.free/d.size)*100),0) AS percent`

 

 

 

Share this post


Link to post
Share on other sites

Hi Darren,

Would you happen to know the SQL how to convert the result type from String to Numeric?
It changes the DataIn in the agents table to something unreadable (PMI:MwB8ADAALgA1AHwARgBhAGwAcwBlAHwAMQAwAHwAMAB8AC0AfAAwAA==)

Share this post


Link to post
Share on other sites
On 4/25/2019 at 12:35 AM, Wupsje said:

Would you happen to know the SQL how to convert the result type from String to Numeric?
It changes the DataIn in the agents table to something unreadable (PMI:MwB8ADAALgA1AHwARgBhAGwAcwBlAHwAMQAwAHwAMAB8AC0AfAAwAA==)

In SQL, generally if it looks like a number, you can treat it like a number. But what you are showing above is an advanced/internationalized value, not a number. The string is Base64, which can be decoded easily in SQL. It should give you a group of values, separated by : or |. If you defined multiple internationalizations, the encoded part can even contain other encoded sections. 

Share this post


Link to post
Share on other sites
On 5/1/2019 at 12:24 PM, DarrenWhite99 said:

In SQL, generally if it looks like a number, you can treat it like a number. But what you are showing above is an advanced/internationalized value, not a number. The string is Base64, which can be decoded easily in SQL. It should give you a group of values, separated by : or |. If you defined multiple internationalizations, the encoded part can even contain other encoded sections. 

If i use SELECT TO_BASE64 in mysql i'm getting different results though. It seems LT somehow uses extra characters.
When i decode this BASE64 string (MwB8ADAALgA1AHwARgBhAGwAcwBlAHwAMQAwAHwAMAB8AC0AfAAwAA==) i get : 3|0.5|False|10|0|-|0
But when i encode that again with base64 i get this back : M3wwLjV8RmFsc2V8MTB8MHwtfDA=

What am i missing that LT is doing?

Share this post


Link to post
Share on other sites
On 5/2/2019 at 7:21 AM, Wupsje said:

If i use SELECT TO_BASE64 in mysql i'm getting different results though. It seems LT somehow uses extra characters.
When i decode this BASE64 string (MwB8ADAALgA1AHwARgBhAGwAcwBlAHwAMQAwAHwAMAB8AC0AfAAwAA==) i get : 3|0.5|False|10|0|-|0
But when i encode that again with base64 i get this back : M3wwLjV8RmFsc2V8MTB8MHwtfDA=

What am i missing that LT is doing?

ASCII vs. Unicode.  Here, this may help. It's what I have scratched together in my notes.


Fields such as 'PMI:MwB8AC0AMAAuADAAMgB8AFQAcgB1AGUAfAA4ADAAfAAwAHwALQB8ADMA' are Base64 Unicode.

Sample decoded values:

Criteria: State Based - Unchecked the "Expression" checkbox. Trending factor=No Trending. Normal <=15, warning between 5 and 80, error >70
With Language Override: en-GB: Normal - 20, Warning - 10 and 85, Error - 75
Comparor=16,DataIn='PMI:3|-1|False|13%7C15%7C8%7C5 and 80%7C3%7C70|0|ZQBuAC0ARwBCAHwAMQAzACUANwBDADIAMAAlADcAQwA4ACUANwBDADEAMAAgAGEAbgBkACAAOAA1ACUANwBDADMAJQA3AEMANwA1AA==|3'
Criteria: State Based - Unchecked the "Expression" checkbox. Trending factor=Use Template Multiplier. Normal <=15, warning between 5 and 80, error >70
Comparor=16,DataIn='PMI:3|2|False|13%7C15%7C8%7C5 and 80%7C3%7C70|0|ZQBuAC0ARwBCAHwAMQAzACUANwBDADIAMAAlADcAQwA4ACUANwBDADEAMAAgAGEAbgBkACAAOAA1ACUANwBDADMAJQA3AEMANwA1AA==|3'
Criteria: State Based - Unchecked the "Expression" checkbox. Trending factor .25. Normal <=15, warning between 5 and 80, error >70
With Language Override: en-GB: Normal - 20, Warning - 10 and 85, Error - 75
Comparor=16,DataIn='PMI:3|0.25|False|13%7C15%7C8%7C5 and 80%7C3%7C70|0|ZQBuAC0ARwBCAHwAMQAzACUANwBDADIAMAAlADcAQwA4ACUANwBDADEAMAAgAGEAbgBkACAAOAA1ACUANwBDADMAJQA3AEMANwA1AA==|3'
Criteria: State Based - Trending factor .25. Normal <=15, warning between 5 and 80, error >70
With Language Override: en-GB: Normal - 20, Warning - 10 and 85, Error - 75
Comparor=16,DataIn='PMI:3|0.25|True|13%7C15%7C8%7C5 and 80%7C3%7C70|0|ZQBuAC0ARwBCAHwAMQAzACUANwBDADIAMAAlADcAQwA4ACUANwBDADEAMAAgAGEAbgBkACAAOAA1ACUANwBDADMAJQA3AEMANwA1AA==|3'
Criteria: State Based - Trending factor .25. Normal <=15, warning between 5 and 80, error >70
Comparor=16,DataIn='PMI:3|0.25|True|13%7C15%7C8%7C5 and 80%7C3%7C70|0|-|3'
Criteria: Threshold Less than 50 - Trending factor .25 - Using DOW Smoothed instead of sample.
Comparor=4,DataIn='PMI:3|0.25|True|50|3|-|3'
Criteria: Threshold Less than 50 - Trending factor .25 - Using Daily Smoothed instead of sample.
Comparor=4,DataIn='PMI:3|0.25|True|50|2|-|3'
Criteria: Threshold Less than 50 - Trending factor .25 - Using Smoothed instead of Sample.
Comparor=4,DataIn='PMI:3|0.25|True|50|1|-|3'
Criteria: Threshold Less than 50 - Trending factor .25 - Changed from "string" to "numeric"
Comparor=4,DataIn='PMI:3|0.25|True|50|0|-|0'
Criteria: Threshold Less than 50 - Trending factor .25
Comparor=4,DataIn='PMI:3|0.25|True|50|0|-|3'
Criteria: Threshold Less than 50 - Trending factor  .75
Comparor=4,DataIn='PMI:3|0.75|True|50|0|-|3'
Criteria: Threshold Less than 80 - Trending factor  .75
Comparor=4,DataIn='PMI:3|0.75|True|80|0|-|3'
Criteria: Threshold Greater than 80 - Trending not specified.
Comparor=3,DataIn='PMI:3|-0.02|True|80|0|-|3'

Format for each parameter: 
UNKNOWN - 3.. Version number?|
TrendingK value - Decimal between 0.01 and 0.99. Special Values: Use Template=2, No Trending=-1|
Use Expressions. Value can be True or False|
Criteria from RESULT field - When State Based is URI Encoded. Format is 'NormalComparisonOp|NormalResult|WarningComparisonOp|WarningResult|ErrorComparisonOp|ErrorResult'
0=Sample,1=Smoothed,2=DailySmoothed,3=DOWSmoothed|
Internationalization Override. '-' if not defined. Otherwise is Base64 Encoded Unicode string. See below|
Result Type for comparison. 0=Numeric, 3=String

Internationalization Format:
Value Stored in the 6th field of the "PMI:" style monitor.
Language Code - 'en-GB'|
Criteria for Result Field. Same format (URI Encoded) string as the 4th field in the PMI monitor.

Example:
Criteria: State Based - Trending factor .25. Normal <=15, warning between 5 and 80, error >70
With Language Override: en-GB: Normal - 20, Warning - 10 and 85, Error - 75
Raw value: ZQBuAC0ARwBCAHwAMQAzACUANwBDADIAMAAlADcAQwA4ACUANwBDADEAMAAgAGEAbgBkACAAOAA1ACUANwBDADMAJQA3AEMANwA1AA==
Decoded Value: en-GB|13%7C20%7C8%7C10 and 85%7C3%7C75

SQL Code to Decode/Encode Advanced Monitors:
SELECT AgentID,GroupID,`Name`,DataIn,CONVERT(CONVERT(CAST(FROM_BASE64(SUBSTRING_INDEX(`DataIn`,':',-1)) AS BINARY) USING utf16le) USING utf8) AS DecodedDataIn,
CONVERT(TO_BASE64(CAST(CONVERT(CONVERT(CONVERT(CAST(BASE64_DECODE(SUBSTRING_INDEX(`DataIn`,':',-1)) AS BINARY) USING utf16le) USING utf8) USING utf16le) AS BINARY)) USING utf8) AS ReEncodedDataIn
FROM groupagents WHERE `DataIn` LIKE 'PMI:%';

Example:
SET @MonitorString='3|0.2|True|85|1|-|3'; SET @EncodedMonitor='PMI:MwB8ADAALgAyAHwAVAByAHUAZQB8ADgANQB8ADEAfAAtAHwAMwA=';
SELECT @MonitorString,CONVERT(CONVERT(CAST(FROM_BASE64(SUBSTRING_INDEX(@EncodedMonitor,':',-1)) AS BINARY) USING utf16le) USING utf8) AS DecodedDataIn;
SELECT @EncodedMonitor,CONVERT(CONCAT('PMI:',TO_BASE64(CAST(CONVERT(@MonitorString USING utf16le) AS BINARY))) USING utf8) AS ReEncodedDataIn;

 

  • Thanks 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×