After some VMware host issues that were out of my control, the RHEL VMs that my production and test Casper servers are hosted on were unexpectedly rebooted a couple of times. When I checked the VMs afterwards, everything appeared to be OK. I figured that I had been fortunate, until my Casper test server sent me the nightly “Successful backup of the Casper database” email and my production server didn’t.
Uh oh.
When I checked the directory when my production server stores its backups of the Casper database, there wasn’t a backup from the night before. I immediately launched the JAMF Database Utility application and had it make a backup of the production database. A task which normally would take 10 minutes or so now took 40 minutes.
Not good.
To lighten the load on the database, I went into the JSS and had it manually flush all but the last week’s worth of logs (I normally retain 30 days of logs.)
Once the log flush had completed, I then rebooted the box. On reboot, the JSS initialized and then hung halfway through the JSS startup process.
Really not good.
For the details of how I fixed this, see below the jump.
Casper’s JSS server software comes in two parts, a Tomcat web application and a MySQL database named jamfsoftware. As far as I could tell, the Tomcat part was OK but was having problems communicating with Casper’s jamfsoftware MySQL database.
To ensure that my Casper-managed Macs were not trying to update the database while I was working on fixing it, I stopped the Tomcat web application by running the following command as root:
/etc/rc.d/init.d/jamf.tomcat7 stop
I then restarted MySQL with the following command:
/etc/init.d/mysqld restart
After the MySQL restart, I ran the following mysqlcheck command to repair the database:
mysqlcheck -repair jamfsoftware -u root -p
In this case, mysqlcheck was set to repair all of the tables in the jamfsoftware database.
90 minutes later, the repair process had finished checking and repairing the database. I then ran optimization with mysqlcheck using the following command:
mysqlcheck -o jamfsoftware -u root -p
Since the database tables had just been repaired, optimization finished within seconds. I then rebooted the server. I could have just restarted the Tomcat web application, but I wanted to make sure everything worked on restart.
On restart, the Casper service started up normally. Later that night, I once again received the nightly “Successful backup of the Casper database” email from both my test and production servers.
