TL;DR moved code to criterion for benchmarking, wanted to ask if I can make a PR
If I understand correctly it’s been a while since Zola has been benchmarked (not a criticism just an observation). I’ve not done much in terms of benchmarking before, but it is something I want to learn more about so I thought I’d take a look at the Zola benchmarks.
As I understand, the most standard way to do benchmarking in Rust is to use the
criterion crate, which provides both support for statistics and for performance regression testing which is quite nice. Additionally, the current libtest code trips up
rust-analyser (for me at least), so I moved the benchmarking code over to criterion and made a few necessary fixes along the way. I’ve also implemented a
Makefile.toml so you can run the benchmarks with
cargo make full-bench or
cargo make small-bench (later will skip the larger test cases) using
cargo-make. This will set up and clean up the test sites using the
gen.py script. this is to make it easier for people to run the benchmarks for themselves, potentially if they are considering using Zola for themselves.
In terms of the configuration of criterion, I think that could be a lot better, especially the testing timeouts etc. but I thought it would be a good idea to try and get the move to criterion merged first. If people are open to it, I can spend some time tuning the configuration of it.
For the time being, I’ve only implemented the benchmarks. I have run them, but haven’t done any serious examination of the results. If people are interested I’d be willing to do a more extensive writeup on this and the current state of Zola performance.
I’m not sure if and how this would be a good idea, but maybe it’s possible to set up the benchmarking as an integration test for stable/major releases or something? just something to think about.