Querying an API for AutoPkg download URLs

As part of working on a new AutoPkg download recipe today, I very quickly got stuck on a problem. The app in question is Poly’s Lens software and its download page uses JavaScript to provide the download URL for the latest version of the Lens software. While this may have benefits for the vendor, this means I can’t scrape the page for the download URL’s address using AutoPkg.

Discussing the issue in the #autopkg channel of the MacAdmins Slack, folks started poking around the Lens app and discovered that it was using the Squirrel framework to provide software update functionality to the app. Assuming that meant that the app would phone home for updates, ahousseini was kind enough to monitor the app’s HTTP and HTTPS traffic using CharlesProxy. Shortly thereafter, he reported seeing Lens send the following API request using curl:

This HTTPS traffic was Lens sending an API request to see if it was running the latest version of the software. The relevant parts from our perspective were the items shown below:

This told us what format we should expect API output to be (in this case, JSON):

This told us the query which was being sent to the API endpoint:

This told us the API endpoint:

Putting this information together, the following curl command gets a response from the API endpoint:

The API response looks similar to this:

Part of the API response’s output includes the download URL for the latest version:

Now that we have this information, how to use it with AutoPkg? For more details, please see below the jump:

You can use AutoPkg’s URLTextSearcher processor for this purpose. The URLTextSearcher processor is designed to use curl to download information from a URL and then perform a regular expression match on the returned information. Because the processor is using curl, one of the processor’s options is to include curl options using the curl_opts input option, to include those options when performing the download request using curl.

The processor also includes a request_headers input option, which allows headers to be included with the download request.

Between being able to send headers and configure curl options, this meant that I had everything needed to query the API. The last part was being able to parse the API endpoint’s response to get the URL which was needed. For that, the re_pattern input option allowed a Python regular expression to parse the output for what was needed.

In an AutoPkg .download recipe, configuring the URLTextSearcher processor like this works to run the desired API call:

When AutoPkg runs the recipe in verbose mode, you should be able to see the URLTextSearcher processor do the following:

1. Run the API call
2. Parse the output
3. Use the designated regular expression to return the desired download URL as output from the processor.

From there, other AutoPkg processors (in this case the URLDownloader processor) can use the output from the URLTextSearcher processor to read the download URL from the output and then use it to download a disk image containing the latest Poly Lens app.

For the full verbose output, please see below:

For reference, I’ve posted AutoPkg recipes for Poly Lens which use the technique I’ve discussed in this post:

https://github.com/autopkg/rtrouton-recipes/tree/master/PolyLens