Jump to content

High Availability of the LabTech Server

Recommended Posts

I am working on a project during my initial Implementation of LabTech to obtain the same level of availability our previous monitoring and management platform has (we are averaging 5 nines at this point with the old system, but it has become to costly to maintain and does not have tools we need)


Below is a 10,000 foot picture of my setup, this does not have many of the finer details, the Database cluster (galera) was already existing along with the BI system.




I am still working through some issues with the initial database load, what we did was a full dump of the Databases (mysqldump --all-databases -uroot -p{Password in the registry] ltdump.sql)

the imported it into the cluster, using mysql proxy we disabled the local MYSQL database and pushed the queries from localhost to the cluster load balancer


It is worth noting that we are using HAProxy on freebsd with carp to handle the Load balancing, we have in the past used GreenSQL as that helps with compliance issues, but for this purpose we are sticking the HAProxy.


I am posting this so those interested might try and hack at it and we can share some info on getting it running

Ultimately my company may go the route of creating a turn key solution for this as we do for other apps for our customers.


Here is a new diagram, this one is a bit simpler and shows what a typical HA Database setup would look like, my setup is very complex due to the nature of what we do.




the above does not cover HA of the lab tech front end itself just the database backend, it also covers for massive scalability of the database



Edited by Guest

Share this post

Link to post
Share on other sites

Lets talk about the high level steps to do this (for a single server)


A few configurations need to be done on your lab-tech server

They are

[list=]Download MySQL Proxy (You will need this for the marketplace and other apps that expect a local MySQL server)

Retrive the MySQL Root password from the registry

Change the Database Host variables in the registry

Stop the Labtech Services

Dump the Local MySQL database


On the DB Host

Install Maria DB 5.x tree from your distributions package manager

Run Mysql_Secure_Install set the root password to the same as the one as you got from the registry on the Labtech host

if you look at the my.ini file in your labtech mysql directory the majority of those can be copy and pasted directly into the my.cnf file (Usualy in /etc on linux or /usr/local/etc on FreeBSD) with the exception of the data path, on my install the config referenced non existent ssl key files do not copy those statements into the linux server as that will result in a fatal error.

Lastly you must add

lower_case_table_names = 1

to the end of the [mysqld] section


Restart the mysql service


import the dump file from the labtech server


Configure the mysql proxy to listen on the labtech host on the loopback interface on port 3306


start the labtech services, you should be good


More to come

Share this post

Link to post
Share on other sites

Very good stuff datagen, I have been thinking of investigating having redundant labtech servers for us as well. I am not familiar with many of the applications you mentioned, are you running the mysql (maria) db on linux or on windows? Ours is on windows, and I could do it easily on MSSQL via availability groups but I don't know how to do this with mysql.


Would your solution help me at all? I don't need geo-redundant right now, just better availability of the database so labtech doesn't have to go down if I need to reboot for patching, but I would like it to scale to geo-redundant with minimal effort in the future preferably but just moving one of the servers offsite.

Share this post

Link to post
Share on other sites

it can be made to do that initaly it is easy onsite with two boxes (or VM's on diffrent hosts and datastores)


Word of caution if your not familiar with clustering out side of the microsoft world it is diffrent, and has its own quirks

Most of the Microsoft cluster came from ideas used in the Unix/Linux world but they implemented totally differently.


If your going to look at a cluster of MySQL server you want to use a Maria based 5.x distribution

For clustering and HA my favorite is Percona XtraDB Cluster, this is open source and you can buy a support contract on it


The family of clustering is the same as Maria using Galera cluster (WSREP, is the API used)

It unlike microsoft clustering is Shared Nothing Multi Master

the only thing the servers need in common is the same root PW and config file


Currently the only clustering for MySQL (and compatible) Databases that will work with LabTech are *Nix Only, this will not run on a windows serve


I think for anyone looking at doing this or similar start by moving your DB to a linux box first then we can add on clustering later,

It is so easy to hang ones self when working with HA systems, no end of problems can come if you do not know what your doing with this


I am happy running it my self, as I have had years of managing linux clusters. but this is not somthing for a first timer yet...


If you want to embark down this road

First thing I would recommend is to Read the manual http://www.percona.com/doc/percona-xtradb-cluster/5.6/, I can help you with the finer points, once you have gone down that road

Step 2, get 2 boxes running with a RHEL based linux (Centos or Oracle) you can do it with debian but I am not a Debian guy and they are just diffrent enough that it can throw you for a loop.


Box Specs

Memory, This is a big player in your system it should be equal to what LT recomends (the OS will use less so you can knock it back by 4gb if you want)

ECC is a MUST, do not try to do this with out ECC Memory

CPU, more the better but the Linux Code is more CPU efficient the the windows code, you can cut the LT recommendation by half

Os Drive 80gb Minimum Mirrored Volume

Data Drive 10k RPM Mirrored same size as LT Recommendation

Logs, and Caches should be on SSD (If your data set is small enough or your wallet big enough you can put the Logs and the Data on SSD)

The log and Cache Drives must be Mirrored (3 way if possible with a spare) (Loss of this data will be catastrophic you will have to restore from a dump, unless another member of the cluster survives)

1gb lan is sufficient for most uses , when you get to the cluster setup, you will need a high quality backend switch (We are talking cisco/juniper no netgear) this switch will only carry data to and from the LT Server and the cluster members

UPS is a must, this should be a high quality Full Online (True Online, Not Line Interactive) Liebert Gxt3 or 4 is best

Dual power supplies are recommended these should go to different UPS units

The Quality of the system you use will effect the quality of service you get, no desktop class machines, stick with an HP DL3xx or Cisco UCS Cxxx (not the entry level Cxx). Stay away from levnovo (I know they have great margins, and they used to be IBM but they are not IBM any more the quaility for this kind of stuff is not there anymore) Dell servers are decent as well I just do not deal in them so I cant quote models


That should get you going

Share this post

Link to post
Share on other sites

Wow thanks for the write up and I will definitely read up on that, sadly it sounds like more than I could jump in to right now as I have very little linux experience so I think I would just get myself in trouble at this point. I will certainly come back to this post after getting a greater knowledge though! (I'm currenly doing some training toward linux+ cert, I know it is probably a dummy course to the pros but I think it should be good to get my feet wet)

Share this post

Link to post
Share on other sites

Linux+ is super basic and they try to be agnostic to a distrbution

this means you will under stand the concepts but not the real day to day stuff


If your going to invest in training and want to go the enterprise linux route go here



RedHat is the 900lb gorilla in the enterprise market, with Suse being a close second.

Debian and its related distro Ubuntu is popular in the education market and with hobbyist market (it is also home of the Open Source Purists)

Share this post

Link to post
Share on other sites

Question to the community, how many people would be interested if some one built a turnkey solution for this?


and How much would it be worth to you?


Also would you prefer such a solution as software + support or the Whole kit hardware included?

Share this post

Link to post
Share on other sites

I would also suggest Centos as a good (and free) RedHat equivalent. It's one of the more solid free Linux distributions that also has an incredible wealth of community support.

Share this post

Link to post
Share on other sites

The preferred platform are RHEL derivatives, that includes Centos

I do not recommend Centos Specifically because they are slow to patch since the hiring of the core developers by red hat in an attempt to kill the project (the Red Hat employees we know HATE Centos and Oracle, to a nearly religious fever)

If you want a free RHEL derivatives

Share this post

Link to post
Share on other sites

this was about a year ago, they have been trending for the last year about 6 - 8 weeks behind RHEL on patches

Oracle is about 2 weeks

Oracle linux is a good base free with lots of enhancements including a custom kernel that is ported from current, so it tends to have a bunch new stuff when it comes to performance and file systems

Also if you buy there support contract (about half as much as Red Hats) you get ksplice which lets you do live kernel updates no reboot needed!

Share this post

Link to post
Share on other sites

They bought Ksplice several years ago (Rhel 5 days) they where a bunch of ex IBMers from the mainframe division

Where such things have been ho hum day to day stuff for years.

They wanted to get this code out to the general community, and the Mainframe Linux Users, but there VC sold to Oracle and that ended up getting the sucked in to Larry's Evil Empire. (I am not a big fan of oracle but i will use there free / cheaper services to better my systems KSplice is one of them)


I even used the kSplice stuff to do a live upgrade from 6 to 7 the machine has not been rebooted since 6 went into production ( uptime this morning 1,572 days)

Share this post

Link to post
Share on other sites

We just bought LabTech today and we need HA. I have read that the 10.5 version should be easier to install in HA. On the LabTech roadmap its scheduled Q3/Q4 this year, but does anyone have exact release date/week?

Share this post

Link to post
Share on other sites

Sorry to revive this thread from the grave, but I wanted to put this warning where people with Google will find it easily - we recently went around the bend with CW:A about this, and they stated the following, directly from their consulting manager JM:


"Clustering is not currently supported".


In addition, the following was stated in another email in the same chain:


"Unfortunately the use of Linux based clustering for MySQL\Database cannot be supported by us as far as a re-import of your data is concerned AND would also not be supported by support moving forward as it is a completely unsupported setup/configuration"


It does not matter if your issue is purely an application/upgrade problem with Automate - you can prove that until you are blue in the face - and you will still be doing a migration before any troubleshooting at all is attempted by them.


TL;DR - Automate is NOT SUPPORTED in any manner of HA configuration, or with ANY involvement of Linux. End of story. They don't have the capability nor the interest to support it. If you have to have support for it for any reason at all, even if you only email them an average of once or twice a year, don't do anything with Linux or any sort of clustering at all. You're just wasting your time.


By way of contrast, I recently had to do a ground up rebuild of our ESET management server. ESET's support doesn't seem to give a toss about what their product is on - they support seemingly every option you could want: clustering, proxying, MSSQL or MySQL, a vast variety of Linux distros plus Windows (although they drew the line at macOS for the server :) ), and they have rather good documentation - thus far, I've nothing but positive experiences with their support. If only Automate got even close to that level in those same regards, I'd surely be happy as a pig in mud.

Share this post

Link to post
Share on other sites

@eburger If you are still after an answer, I asked my Account manager this last week and the reply back was that there is no support at all. No just for the database but as a whole. 

Edited by Axle
saved too soon

Share this post

Link to post
Share on other sites
On 1/23/2019 at 11:46 PM, Axle said:

the reply back was that there is no support at all. No just for the database but as a whole. 

We warned ya lol - but good writeup! Hope everything you have to submit doesn't get closed as environmental though :D

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...