Generated Jogger Modules

At its core, PkgJogger uses meta-programming to generate a Jogger module for running a package's benchmarks. For example, calling @jog on Example gives a jogger named JogExample for running the benchmark suite of Example:

julia> using PkgJogger, Example

julia> @jog Example
JogExample

Similarly, @jog AwesomePkg would create a module named JogAwesomePkg:

PkgJogger.@jog — Macro

@jog PkgName

Creates a module named JogPkgName for running benchmarks for PkgName.

Most edits to benchmark files are correctly tracked by Revise.jl. If they are not, re-run @jog PkgName to fully reload JogPkgName.

Revise must be loaded before calling @jog PkgName in order for edits to be automatically tracked.

Methods

suite Return a BenchmarkGroup of the benchmarks for PkgName
benchmark Warmup, tune and run the suite
run Dispatch to BenchmarkTools.run(suite(), args...; kwargs...)
save_benchmarks Save benchmarks for PkgName using an unique filename

Isolated Benchmarks

Each benchmark file, is wrapped in it's own module preventing code loaded in one file from being visible in another (unless explicitly included).

Example

using AwesomePkg, PkgJogger
@jog AwesomePkg
results = JogAwesomePkg.benchmark()
file = JogAwesomePkg.save_benchmarks(results)

Compare benchmarking results to the latest saved results

results = JogAwesomePkg.benchmark()
JogAwesomePkg.judge(results, :latest)

source

Benchmark Directory Structure

PkgJogger will recursively search the package's benchmark/ directory for benchmarking files bench_*.jl or directories bench_*/.

For example, the following directory:

.
+-- Project.toml
+-- src/
|   +-- PkgName.j;
|   ...
+-- benchmark
    +-- bench_matrix.jl # Will be included
    ....
    +-- subdir/     # ignored
    +-- bench_ui/   # This gets added
        +-- bench_foo.jl
        ....

Results in a benchmarking suite of:

1-element BenchmarkTools.BenchmarkGroup:
    "bench_matrix.jl" => Suite from "benchmark/bench_matrix.jl"
    ... # Other benchmark/bench_*.jl files
    "bench_ui" => BenchmarkTools.BenchmarkGroup:
        "bench_foo.jl" => Suite from "benchmark/bench_ui/bench_foo.jl"
        ... # Other benchmark/bench_ui/bench_*.jl files

Benchmark Files

PkgJogger expects the following structure for benchmarking files:

# PkgJogger will wrap this file into a module, thus it needs to declare all of
# it's `using` and `import` statements.
using BenchmarkTools
using OtherPkg
using AweseomePkg

# PkgJogger assumes the first `suite` variable is the benchmark suite for this file
suite = BenchmarkGroup()

# This will add a benchmark "foo" to the benchmarking suite with a key of:
# ["bench_filename.jl", "foo"]
suite["foo"] = @benchmarkable ...

# Further nesting within the file's `suite` is possible
s = suite["baz"] = BenchmarkGroup()
s["bar"] = @benchmarkable ... # Key of ["bench_filename.jl", "baz", "bar"]

In the example, we assume the benchmarking file is benchmark/bench_filename.jl. If it was located in a subdirectory benchmark/bench_subdir the resulting suite would have keys of ["bench_subdir", "bench_filename.jl", ...], instead of ["bench_filename.jl", ...]. as shown.

A side effect of this structure is that each benchmarking file is self-contained and independent of other benchmarking files. This means that if you want to run the suite of a single file, you can include the file and run it with: tune!(suite); run(suite)

Filtering Benchmarks

Often it's useful to run only a subset of the full benchmarking suite. For example, to run only the benchmarks within benchmark/bench_filename.jl: JogExample.run("bench_filename").

using PkgJogger, Example
@jog Example
JogExample.run("bench_filename.jl")

!!! compat PkgJogger 0.6.0 Support for filtering via JogExample.suite was added in v0.6.0

Jogger Reference

Jogger modules provide helper methods for working with their package's benchmarking suite. For reference, this section documents the methods for @jog Example.

Main.JogExample.suite — Function

suite()::BenchmarkGroup

The BenchmarkTools suite for Example

source

suite(select...)

Returns the benchmarking suite for Example, optionally filtering based on select.... At it's simplest, JogExample.suite(a, b, ...) is equivalent to JogExample.suite()[a][b]...

Supported Indices

: - Accepts any entry at that level in the tree
r"Regexp" - Accepts any entry matching the regular-expression
key::Any - Accepts any entry with a matching key
@tagged - Filters the suite to only include BenchmarkGroups with a matching tag.

See Indexing into a BenchmarkGroup using @tagged

Warning

An entry in suite must match all indices to be returned. For example, JogExample.suite(:, "bar") would exclude a benchmark at suite["bat"] as the benchmark isn't matched by both : and "bar".

Examples

The suite in bench_foo.jl: JogExample.suite("bench_foo.jl")
Any benchmark matching r"feature" in any bench_*.jl: JogExample.suite(:, r"feature")

source

Main.JogExample.benchmark — Function

benchmark([select...]; verbose = false, save = false, ref = nothing)

Warmup, tune and run the benchmarking suite for Example.

If save = true, will save the results using JogExample.save_benchmarks and display the filename using @info.

To reuse prior tuning results set ref to a BenchmarkGroup or suitable identifier for JogExample.load_benchmarks. See PkgJogger.tune! for more information about re-using tuning results.

Optionally, benchmark a subset of the full suite by providing a set of filters. See PkgJogger.getsuite for more information.

source

Main.JogExample.run — Function

run([select...]; verbose::Bool = false, kwargs)

Run the benchmarking suite for Example. See BenchmarkTools.run for more options

Optionally, run a subset of the full suite by providing a set of filters. See PkgJogger.getsuite for more information.

source

Main.JogExample.profile — Function

profile(select...; profiler=:cpu, verbose=false, ref=nothing, kwargs...)

Profile the benchmarking suite using the given profiler, the benchmark is warmed up, tuned and then ran under the profile.

Like JogExample.benchmark, ref can be used to reuse the results of a prior run during tuning.

Some profilers support additional keyword arguments, see below for details.

Info

At this time, PkgJogger only supports profiling a single benchmark at a time. Automated saving is not supported.

Available Profilers

The following profilers have been implemented, but may not be currently loaded (See Loaded Profilers).

:cpu - loaded by default
:allocs - loaded if Profile.Allocs exists (>=v1.8)
:cuda - loaded if the CUDA and NVTX packages are loaded

Loaded Profilers

The following profilers are currently loaded. Additional profilers are available via package extensions.

profiler=:cpu

Profiles the benchmark using Julia's built-in profiler: Profile.@profile

 profiler=:allocs

Profiles memory allocations using the built-in Profile.Allocs.@profile

Accepts sample_rate as a kwarg to control the rate of recordings. A rate of 1.0 will record everything; 0.0 will record nothing. See Profile.Allocs.@profile for more.

Julia 1.8

The allocation profiler was added in Julia 1.8

profiler=:cuda

Profiles the benchmark using CUDA.@profile.

Warning

This only activates the CUDA profiler, you need to launch the profiler externally. See CUDA Profiling for documentation.

Info

This list was generated on jogger creation (@jog Example), and my not reflect all loaded extensions. See PkgJogger.profile or regenerate the jogger for additional information by running @jog Example again.

source

Main.JogExample.save_benchmarks — Function

save_benchmarks(results::BenchmarkGroup)::String

Saves benchmarking results for Example to BENCHMARK_DIR/trial/UUIDs.uuid4().bson.gz, and returns the path to the saved results

Meta Data such as cpu load, time stamp, etc. are collected on save, not during benchmarking. For representative metadata, results should be saved immediately after benchmarking.

Results can be loaded with PkgJogger.load_benchmarks or JogExample.load_benchmarks

Examples

Running a benchmark suite and then saving the results

r = JogExample.benchmark()
filename = JogExample.save_benchmarks(r)

Equivalently: JogExample.benchmark(; save = true)

source

Main.JogExample.load_benchmarks — Function

load_benchmarks(id)::Dict

Loads benchmarking results for Example from BENCHMARK_DIR/trial based on id. The following are supported id types:

- `filename::String`: Loads results from `filename`
- `uuid::Union{String, UUID}`: Loads results with the given UUID
- `:latest` loads the latest (By mtime) results from `BENCHMARK_DIR/trial`
- `:oldest` loads the oldest (By mtime) results from `BENCHMARK_DIR/trial`

source

Main.JogExample.judge — Function

judge(new, old, [select...]; metric=Statistics.median, kwargs...)

Compares benchmarking results from new vs old for regressions/improvements using metric as a basis. Additional kwargs are passed to BenchmarkTools.judge

Optionally, filter results using select..., see JogExample.suite for details.

Identical to PkgJogger.judge, but accepts any identifier supported by JogExample.load_benchmarks

Examples

# Judge the latest results vs. the oldest
JogExample.judge(:latest, :oldest)
[...]

# Only judge results in `bench_foo.jl`
JogExample.judge(:latest, :oldest, "bench_foo.jl")

# Judge results by UUID
JogExample.judge("cee7fe2e-0756-49af-9f1b-b5e6ca6bd4d3", "7ace2e02-b514-49a7-803a-baa389ae8032")
[...]

# Judge using the minimum, instead of the median, time
JogExample.judge("path/to/results.bson.gz", "6c53b867-7c31-454d-bc26-46e42944e635"; metric=minimum)
[...]

source

Main.JogExample.BENCHMARK_DIR — Constant

BENCHMARK_DIR

Directory of where benchmarking results are saved for Example

source