Automating AutoPkg runs with autopkg-conductor

About two weeks ago, I noticed I had an SSL error cropping up with one of my AutoPkg recipes:

[Errno socket error] EOF occurred in violation of protocol (_ssl.c:590)

When I investigated what it meant, I wound up at this lengthy issue opened for Python’s requests module. In the end, it seemed to boil down to four issues:

1. I was running AutoPkg on macOS Sierra 10.12.6.
2. The recipe I was running used a processor which called Python’s urllib2 library.
3. Python’s urllib2 library was calling the OS’s installed version of OpenSSL to connect to a server using TLSv1.2 .
4. The version of OpenSSL included with 10.12.6 does not support TLSv1.2 for the urllib2 library.

When I looked into the situation on macOS High Sierra 10.13.5, Apple had addressed the problem by replacing OpenSSL with LibreSSL. Among other improvements, LibreSSL allowed Python’s urllib2 library to be able to connect to servers using TLSv1.2. Problem solved!

Until I ran into another problem.

I had been using AutoPkgr as my way of managing AutoPkg and scheduling AutoPkg runs. However, when I set up AutoPkgr on a 10.13.5 VM and scheduled my AutoPkg nightly run, nothing happened except my CPU spiked to 100% and AutoPkgr locked up with the pinwheel of patience.

OK, maybe it was something with my VM. No problem, set up a new macOS 10.13.5 VM.

Same problem.

Maybe it was because I was trying to run the VM on VMware’s ESXi? Set up a new VM running in VMware Fusion. Same problem.

Maybe AutoPkgr was getting confused by Apple File System? I set up a 10.13.5 VM which used an HFS+ boot volume. Same problem, replicated on both ESXi and Fusion.

No matter what I tried, trying to run recipes using AutoPkgr on macOS 10.13.x resulted in the following:

• The VM’s CPU spiking to 100%
• AutoPkgr locking up with the pinwheel of patience
• My AutoPkg recipes not running

I was able to eliminate AutoPkg itself as being the issue, as running recipes from the command line using AutoPkg worked fine. With that information in mind, I decided to see if I could replicate what I most liked about using AutoPkgr into another form. In the end, my needs boiled down to three:

1. I wanted to be able to run a list of AutoPkg recipes on a scheduled basis. These recipes would be .jss recipes for uploading to a Jamf Pro server.
2. I wanted to be able to post information about those AutoPkg recipes to a Slack channel
3. I wanted all the error messages from an AutoPkg run, but I didn’t care about all the information that came from a successful AutoPkg run.

With that, I decided to draw on some earlier work done by Sean Kaiser, a colleague who had written a script for managing AutoPkg in the pre-AutoPkgr days. For more details, please see below the jump.

Sean’s solution relies on a script and LaunchDaemon running on a Mac, where it runs hourly and is set up to only send him emails if the AutoPkg logs are different from previous runs. The email notifications are a diff against the previous logs, so only the true differences get sent.

For those interested, Sean’s script is available from here:

https://github.com/seankaiser/automation-scripts/tree/master/autopkg

I was more focused on a once-daily run, so I didn’t want to use the diff methodology. After some more research, I found that my colleague Graham Pugh had written pretty much exactly what I needed: An AutoPkg post-processor named Slacker which could be used with an AutoPkg recipe list of .jss recipes to post the results to a Slack channel.

I forked a copy of the Slacker post-processor and (with Graham’s help) made some edits to it to have the output appear exactly the way I wanted it to.

New package message:

No new package message:

Along with the Slacker post-processor, I also found a script for sending multiline output to a Slack channel. This would allow me to send the complete error log from an AutoPkg run to a specified Slack webhook.

Using all of this, I wrote a script named autopkg-conductor which is designed to do the following:

1. Detect a list of AutoPkg recipes at a defined location and verify that the list is readable.
2. If the AutoPkg recipe list is readable and available, run the following actions:

A. Verify that AutoPkg is installed.
B. Update all available AutoPkg repos with the latest recipes.
C. Run the AutoPkg recipes in the list.

The AutoPkg run has all actions logged to ~/Library/Logs, with the logfiles being named autopkg-run-for- followed by the date.

If the optional slack_post_processor and slack_webhook variables are both populated, any AutoPkg .jss recipes should have their output sent to the Slack webhook specified in the slack_webhook variable.

If only the slack_webhook variable is populated, all output from the AutoPkg run is sent to the Slack channel. No filtering is applied, everything is sent.

If neither the slack_post_processor or slack_webhook variables are populated, no information is sent to Slack. All AutoPkg run information will be in the logs stored in ~/Library Logs.

For scheduled runs, I recommend the following:

1. Set up a user account named autopkg to run AutoPkg in.
2. Copy the autopkg-conductor script to /usr/local/bin/autopkg-conductor.sh and set the autopkg-conductor.sh script to be executable.
3. Set up a LaunchDaemon to run /usr/local/bin/autopkg-conductor.sh at a pre-determined time or interval.

For this example, the LaunchDaemon shown below will run /usr/local/bin/autopkg-conductor.sh as the autopkg user once a day at 2:00 AM.

The autopkg-conductor script is available below. It’s also available from GitHub using the following link:

https://github.com/rtrouton/autopkg-conductor