Continuous Benchmarking

PkgJogger provides a quick one-liner for setting up, running, and saving benchmarking results as part of a CI/CD pipeline:

julia -e 'using Pkg; Pkg.add("PkgJogger"); using PkgJogger; PkgJogger.ci()'

Github Actions

Just add uses: awadell1/PkgJogger and you're set! For example, the following will setup julia, run the benchmarks and upload the results for later analysis:

name: PkgJogger
on:
    - push
    - pull_request

jobs:
    benchmark:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - uses: julia-actions/setup-julia@latest
            - uses: awadell1/PkgJogger.jl@latest
            - uses: actions/upload-artifact@v2
              with:
                name: benchmarks
                path: benchmark/trial/*

Isolated Benchmarking Environment

PkgJogger will create a temporary environment with the following:

  1. Activate a temporary Julia environment for benchmarking.
    • If a Julia project file exists in benchmark/, it will be copied to the temporary environment. Manifest files are currently ignored.
    • Otherwise, an empty environment is created.
  2. Add the current project (via Pkg.develop) to the benchmarking environment and resolve dependencies using PRESEVE_NONE.
  3. Add PkgJogger and resolve dependencies using PRESERVE_TIERED.
  4. Strip the LOAD_PATH to the benchmarking environment. The prior LOAD_PATH is restored after benchmarking.

This results in an isolated environment with the following properties:

  • Minimizes PkgJogger's impact on dependency resolution.
  • Packages not explicitly added by Project.toml or benchmark/Project.toml are not available in the benchmarking environment.

Testing Benchmarks

Often benchmarking suites are too large to be included in unit testing, or PkgJogger.ci may be too costly to run with each push/pr/etc. However, regressions are inevitable without continuous testing as changes inadvertently break the benchmark suite. To help with this, PkgJogger provides the @test_benchmarks as a smoke test for possible breakages.

PkgJogger.@test_benchmarks — Macro
@test_benchmarks PkgName

Collects all benchmarks for PkgName, and test that they don't error when ran once.

Example

julia> using PkgJogger, Example

julia> @test_benchmarks Example
│ Test Summary:  | Pass  Total
│ bench_timer.jl |    2      2
[...]

Testing

Each benchmark is wrapped in a @testset, run only once, and marked as passing iff no errors are raised. This provides a fast smoke test for a benchmarking suite, and avoids the usual cost of tunning, warming up and collecting samples accrued when actually benchmarking.

Benchmark Loading

Locating benchmarks for testing is the same as for @jog and can be examined using PkgJogger.locate_benchmarks.

source

Reference

PkgJogger.ci — Function

Sets up an isolated benchmarking environment and then runs the following:

using PkgJogger
using PkgName
jogger = @jog PkgName
result = JogPkgName.benchmark()
filename = JogPkgName.save_benchmarks(result)
@info "Saved benchmarks to $filename"

Where PkgName is the name of the package in the current directory

source