Tests, Docs, and Deploys for Clojure

Post by Saul Shanabrook

This semester, I have been working on the Clojush codebase, which is written in Clojure. I want to make it easier for new people to jump in and use it and start contributing. Here I will go through how I added auto generated API docs and auto deploys to Clojars.


To be more specific, every build of the master branch on Travis CI builds the HTML docs, from the codebase, with Codox and commits that to the gh-pages branch, to host them on Github Pages. It also creates a patch release, using lein release, which creates new github commits and a tag, and pushes the release to Clojars.

So I added this project to Travis, then I added private environmental variables to the repo so it is authenticated to push back to Gihub and push the release to Clojars.

Then I created a .travis.yml:

sudo: true  
language: clojure

# https://github.com/technomancy/leiningen/issues/2014#issuecomment-153829977
before_install: yes y | sudo lein upgrade

  - lein test
after_success: ./scripts/after_success.sh 2>&1 | sed "s/$GITHUB_TOKEN/<hidden github token>/"  
  email: false

As you can see all the magic is in ./scripts/after_success.sh. The pipe to sed is to try to prevent exposing your private github credentials. They are added as part of a Git remote, so that it can push back to Github. I like to log all the commands in my bash script, but I found I kept exposing the credentials, even when logging was off, because lein release prints out how it pushes to Git. So I figured it would be easier to just censor that from all of standard err and standard output.

Then I wrote the ./scripts/after_success.sh to build the docs and deploy the release:

#!/usr/bin/env bash
set -o errexit  
set -o nounset  
set -o xtrace  
set -o pipefail

[email protected]_._  

git remote set-url origin https://[email protected]/$TRAVIS_REPO_SLUG.git  
git branch --set-upstream-to origin/master master  
git config user.name "$AUTOMATED_AUTHOR_NAME"  
git config user.email "$AUTOMATED_AUTHOR_EMAIL"  
git config push.default simple  
# dont output all of lein doc, because its overly long because it tries
# to run experiments
lein codox 2>&1 | head -n 100  
./scripts/deploy-docs.sh --verbose
git checkout master  
lein release  

The Github user is used as the author of the commits, so you can choose this to be whatever you want.

The ./scripts/deploy-docs.sh is a slightly modified version of X1011/git-directory-deploy, with added support to automatically make the gh-pages branch if it doesn't exist.

Oh also, in the project.clj I had to add the codox and lein-shell plugins as well as tell lein to use Clojars for releases and changed the :release-tasks to disable code signing on the Git tag and add [skip ci] to the auto created commits (thank you Jean Niklas L'orange
for showing me how to do this last part with lein-shell). Here are the relevant parts of the changed project.clj:

(defproject clojush "2.0.46-SNAPSHOT"
  :plugins [[lein-codox "0.9.1"]
            [lein-shell "0.5.0"]]
  :codox {:source-uri "http://github.com/lspector/Clojush/blob/master/{filepath}#L{line}"
          :namespaces [#"^(?!clojush\.problems)"]
          :output-path "doc"
          :metadata {:doc/format :markdown}}  :repositories [["releases" {:url "https://clojars.org/repo"
                              :username :env
                              :sign-releases false
                              :password :env}]]
  :release-tasks [["vcs" "assert-committed"]
                  ["change" "version" "leiningen.release/bump-version" "release"]
                  ["shell" "git" "commit" "-am" "Version ${:version} [ci skip]"]
                  ["vcs" "tag" "v" "--no-sign"] ; disable signing and add "v" prefix
                  ["change" "version" "leiningen.release/bump-version"]
                  ["shell" "git" "commit" "-am" "Version ${:version} [ci skip]"]
                  ["vcs" "push"]])

Next Steps

This system has been in place for a couple of weeks now and seems to be working pretty well. Currently, it is limited to one concurrent build at a time, because I was worried that if two release try to get made simultaneously then there would be some race conditions. By adding a couple of git pulls before and after the release step, we could reduce this chance, but it is still possible.

Creating a new released patch version after every build is also not ideal for most projects. It wouldn't be hard to scan for some text in the commit message, like [release patch]/[release minor] and only created releases then. Also, maybe creating releases should happen just locally, and not in an automated fashion. What would be nice, would be to decouple releases somewhat from commits. Like some sort of button that any contributor could press to toggle a release. Maybe like a Github bot? You open an issue and say something like @releasebot patch and it replies @lspector do I have your permission to create a patch release? and if he says yes then it goes and does it.