Generated Jogger Modules

At its core, PkgJogger uses meta-programming to generate a Jogger module for running a package's benchmarks. For example, calling @jog on Example gives a jogger named JogExample for running the benchmark suite of Example:

julia> using PkgJogger, Example

julia> @jog Example
JogExample

Similarly, @jog AwesomePkg would create a module named JogAwesomePkg:

PkgJogger.@jog — Macro
@jog PkgName

Creates a module named JogPkgName for running benchmarks for PkgName.

Most edits to benchmark files are correctly tracked by Revise.jl. If they are not, re-run @jog PkgName to fully reload JogPkgName.

Revise must be loaded before calling @jog PkgName in order for edits to be automatically tracked.

Methods

  • suite Return a BenchmarkGroup of the benchmarks for PkgName
  • benchmark Warmup, tune and run the suite
  • run Dispatch to BenchmarkTools.run(suite(), args...; kwargs...)
  • save_benchmarks Save benchmarks for PkgName using an unique filename

Isolated Benchmarks

Each benchmark file, is wrapped in it's own module preventing code loaded in one file from being visible in another (unless explicitly included).

Example

using AwesomePkg, PkgJogger
@jog AwesomePkg
results = JogAwesomePkg.benchmark()
file = JogAwesomePkg.save_benchmarks(results)
source

Benchmark Directory Structure

PkgJogger will recursively search the package's benchmark/ directory for benchmarking files bench_*.jl or directories bench_*/.

For example, the following directory:

.
+-- Project.toml
+-- src/
|   +-- PkgName.j;
|   ...
+-- benchmark
    +-- bench_matrix.jl # Will be included
    ....
    +-- subdir/     # ignored
    +-- bench_ui/   # This gets added
        +-- bench_foo.jl
        ....

Results in a benchmarking suite of:

1-element BenchmarkTools.BenchmarkGroup:
    "bench_matrix.jl" => Suite from "benchmark/bench_matrix.jl"
    ... # Other benchmark/bench_*.jl files
    "bench_ui" => BenchmarkTools.BenchmarkGroup:
        "bench_foo.jl" => Suite from "benchmark/bench_ui/bench_foo.jl"
        ... # Other benchmark/bench_ui/bench_*.jl files

Benchmark Files

PkgJogger expects the following structure for benchmarking files:

# PkgJogger will wrap this file into a module, thus it needs to declare all of
# it's `using` and `import` statements.
using BenchmarkTools
using OtherPkg
using AweseomePkg

# PkgJogger assumes the first `suite` variable is the benchmark suite for this file
suite = BenchmarkGroup()

# This will add a benchmark "foo" to the benchmarking suite with a key of:
# ["bench_filename.jl", "foo"]
suite["foo"] = @benchmarkable ...

# Further nesting within the file's `suite` is possible
s = suite["baz"] = BenchmarkGroup()
s["bar"] = @benchmarkable ... # Key of ["bench_filename.jl", "baz", "bar"]

In the example, we assume the benchmarking file is benchmark/bench_filename.jl. If it was located in a subdirectory benchmark/bench_subdir the resulting suite would have keys of ["bench_subdir", "bench_filename.jl", ...], instead of ["bench_filename.jl", ...]. as shown.

A side effect of this structure is that each benchmarking file is self-contained and independent of other benchmarking files. This means that if you want to run the suite of a single file, you can include the file and run it with: tune!(suite); run(suite)

Jogger Reference

Jogger modules provide helper methods for working with their package's benchmarking suite. For reference, this section documents the methods for @jog Example.

Main.JogExample.save_benchmarks — Function
save_benchmarks(results::BenchmarkGroup)::String

Saves benchmarking results for Example to BENCHMARK_DIR/trial/uuid4().bson.gz, and returns the path to the saved results

Meta Data such as cpu load, time stamp, etc. are collected on save, not during benchmarking. For representative metadata, results should be saved immediately after benchmarking.

Results can be loaded with PkgJogger.load_benchmarks or JogExample.load_benchmarks

Example

Running a benchmark suite and then saving the results

r = JogExample.benchmark()
filename = JogExample.save_benchmarks(r)

Equivalently: JogExample.benchmark(; save = true)

source
Main.JogExample.load_benchmarks — Function
load_benchmarks(id)::Dict

Loads benchmarking results for Example from BENCHMARK_DIR/trial based on id. The following are supported id types:

- `filename::String`: Loads results from `filename`
- `uuid::Union{String, UUID}`: Loads results with the given UUID
- `:latest` loads the latest (By mtime) results from `BENCHMARK_DIR/trial`
- `:oldest` loads the oldest (By mtime) results from `BENCHMARK_DIR/trial`
source
Main.JogExample.judge — Function
judge(new, old; metric=Statistics.median, kwargs...)

Compares benchmarking results from new vs old for regressions/improvements using metric as a basis. Additional kwargs are passed to BenchmarkTools.judge

Identical to PkgJogger.judge, but accepts any identifier supported by JogExample.load_benchmarks

Examples

# Judge the latest results vs. the oldest
JogExample.judge(:latest, :oldest)
[...]
# Judge results by UUID
JogExample.judge("0465dd67-1490-4b8a-b0ea-14478f811b63", "3e1e6960-1d62-4d2f-aee8-968a12177362")
[...]
# Judge using the minimum, instead of the median, time
JogExample.judge("path/to/results.bson.gz", "3df49659-00a1-4741-b259-7f45759b91c7"; metric=minimum)
[...]
source