benchmark_dir(pkg::Module)
benchmark_dir(pkg::PackageSpec)
benchmark_dir(project_path::String)

Returns the absolute path of the benchmarks folder for pkg.

Supported Benchmark Directories:

• PKG_DIR/benchmark

locate_benchmarks(pkg::Module)
locate_benchmarks(path::String, name=String[])

Returns a list of BenchModule for identified benchmark files

judge(new, old; metric=Statistics.median, kwargs...)

Compares benchmarking results from new vs old for regressions/improvements using metric as a basis. Additional kwargs are passed to BenchmarkTools.judge

Effectively a convenience wrapper around load_benchmarks and BenchmarkTools.judge

new and old can be any one of the following: - Filename of benchmarking results saved by PkgJogger - A Dict as returned by PkgJogger.load_benchmarks(filename) - A BenchmarkTools.BenchmarkGroup with benchmarking results

test_benchmarks(s::BenchmarkGroup)

Runs a @testsuite for each benchmark in s once (One evaluation of the benchmark's target) Sub-benchmark groups / benchmarks are recursively wrapped in @testsuites for easy identification.

benchmarks are marked as "passing" if they don't error during evaluation.

tune!(group::BenchmarkGroup, ref::BenchmarkGroup; verbose::Bool=false)

Tunes a BenchmarkGroup, only tunning benchmarks not found in ref, otherwise reuse tuning results from the reference BenchmarkGroup, by copying over all benchmark parameters from ref.

This can reduce benchmarking runtimes significantly by only tuning new benchmarks. But does ignore the following: - Changes to benchmarking parameters (ie. memory_tolerance) between group and ref - Significant changes in performance, such that re-tunning is warranted - Other changes (ie. changing machines), such that re-tunning is warranted

build_module(s::BenchModule)

Construct a module wrapping the BenchmarkGroup defined by s::BenchModule