AutoPkg is a software packaging system for macOS, commonly paired with Munki for distribution. For many organizations, Munki is the default way to coordinate the installation and updating of applications on macOS. At Gusto we use it to provision new machines and update existing ones. We’ve been running AutoPkg on Github Actions since late 2019 and wanted to share how we set up our build pipeline.
AutoPkg refers to the steps required to package a specific utility as “recipes,” which define elements like code signature validation, supplemental installation scripts, and installer download locations. Github Actions is a CI/CD service offering Windows, GNU/Linux, and macOS runners charged per minute of run time. Github Actions is one of the few CI services that offer macOS runners billed by the minute, which is useful for short jobs like AutoPkg recipe runs.
Why Our Previous AutoPkg Environment Was No Fun
Prior to moving our AutoPkg builds to Github Actions, we ran our AutoPkg recipes on a dedicated VM that was hand built and manually updated. It was our house plant; things were mostly fine day to day, but regular maintenance was required and someone needed to watch it when we went on vacation. Any upgrades to the base OS or AutoPkg required careful testing and a rollback plan. In the worst case scenario a disk failure could have resulted in the loss of our entire build process, which thankfully never happened.
Interacting with the box was a bit of a pain too, since we kept access pretty locked down for security reasons. The host machine was located on-premises and required VPN. We couldn't get our desktop support team interested in adding recipes when users requested new apps because we had no review process and everyone was afraid that clicking the wrong button would cause the machine to explode.
The Structure of a Github Actions AutoPkg Run
To make sure we’re rolling out the latest version of applications, we run our AutoPkg build daily. As mentioned earlier, we use Github Actions VMs to save us the headache of maintaining our own build infrastructure.
Our daily Github Actions AutoPkg run does the following:
- checks out the latest version of our AutoPkg recipe overrides
- installs Munki and AutoPkg
- clones all the upstream recipe repos we use to give us an identical working state each time, while allowing our recipe overrides to fail if there are new upstream changes
- runs our fork of Facebook's autopkg_tools.py script, which iterates over a list of recipes, and successful builds are pushed into a separate Git LFS repo
- posts build results to a Slack channel so we can fix any recipe issues with a quick pull request
GitHub Actions: Security and Configuration Considerations
- Use third-party Github Actions wisely. Third-party actions can be silently updated, introducing surprise bugs or malicious code. Since Git and Github have robust protections against SHA1 collisions, pin the SHA1 commit hash for actions instead of including actions by name alone.
- Limit the number of third party actions you use, especially for simple commands like “git push.” Using too many actions will make your jobs harder to debug, slow down your builds, and melt the ice caps faster by wasting electricity. For cross-platform tasks, the Ubuntu runners are cheaper than the macOS runners by an order of magnitude.
- Set timeouts in your Github workflow configuration, lest a zombie script keep the runner alive.
- Speed up your builds by only pulling in the latest commit.
- If you’re using Git LFS, use only the file pointers and not full installers. You can find examples of both in our sample git repository.
In a future release of macOS, only signed package installers will be able to be deployed. I'd like to add our developer cert as a Github Actions Secret to sign internal packages and free software tools built from source.
Since we’re running AutoPkg in ephemeral VMs, we discard cached downloads and end up downloading packages again each day even though we’ve already imported the same version into our internal package repository. We’re planning on submitting an upstream patch to the AutoPkg project to optionally read package download history from a file on-disk or hosted elsewhere, which should significantly speed up AutoPkg runs.
How do I use Gusto’s AutoPkg build process?
- Create an empty Github repo with Actions enabled.
- Clone our Git repo and copy the contents of the autopkg directory to your own private repository.
- Copy workflows/ to .github/ in this repo.
- Create an AutoPkg recipe override (be sure to place it in overrides/).
- Add the override filename to recipe_list.json.
- Add the repo to repo_list.txt.
- Create another empty Github repo with Actions enabled. This will be your Munki repo.
- Copy the name of your Munki git repo to the “Checkout your Munki LFS repo” step in autopkg.yml.
- Add Github Actions secrets for SLACK_WEBHOOK_URL, SLACK_TOKEN, and GITHUB_TOKEN.
We’re eager to see how other organizations are using Github Actions or similar CI systems for running tasks that previously required a dedicated VM, like AutoPkg or Terraform. Hopefully, sharing our process helps make managing software packaging on macOS a little bit easier for you.