From 99b5db7522929e9157a5ab20f2bd3c4997ca67ae Mon Sep 17 00:00:00 2001 From: Kamen Mladenov Date: Tue, 11 Feb 2025 17:58:29 +0200 Subject: feat: Add README --- README.md | 171 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 170 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0c24556..ea696a8 100644 --- a/README.md +++ b/README.md @@ -1 +1,170 @@ -# zkVMs-benchmarks \ No newline at end of file +# zkVMs-benchmarks + +This is a repository with setups and programs for zero-knowledge virtual machine benchmarking. +Its ultimate goal is to deliver reproducible and accurate performance metrics accross all zkVMs, so **you** can choose which technology suits your needs! + +Being made with reproducibility in mind, this project also serves as a good framework for running programs accross zkVMs without the complicated and everchanging setups required to do so. + +## Usage + +The backbones of this entire codebase are [Nix](https://nixos.org/) and Linux. +MacOS is not supported! +Windows is supported via WSL. + +**First**, install the Nix package manager, follow their [download instructions](https://nixos.org/download/). +Generally it should be enough to: + +```sh +sh <(curl -L https://nixos.org/nix/install) --daemon +``` + +> [!WARNING] +> It is preferrable to use the nixos.org script, as shown above! +> Certain systems provide Nix with their native package managers, however practice has shown those do not always lead to working setups! + +Now, what follows depends on your usecase. + +### Run/benchmark a "built-in" program + +The `guests` directory provides a variety of programs, all of which are proven to work with their default inputs. +It is advisable to first try and run one of them, so you can make sure Nix is installed and working properly. + +The smallest one is `fibonacci`. +To make *all* zkVMs generate a proof for it just run: + +```sh +nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove +``` + +> [!NOTE] +> Take notice of the space between `--` and `prove`! +> It marks an "end of options", as specified by [the POSIX specification](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02). + +Or to make, for example, [SP1](https://docs.succinct.xyz/docs/sp1/introduction) generate a proof and verify it, you may run: + +```sh +nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- verify +``` + +### Run/benchmark your own program + +1. Clone the [git repository](https://github.com/blocksense-network/zkVMs-benchmarks) + + ```sh + git clone git@github.com:blocksense-network/zkVMs-benchmarks.git + ``` + +2. Navigate to the `guests` directory + + ```sh + cd zkVMs-benchmarks/guests + ``` + +3. Follow the instructions inside the `guests/README.md` file to setup your program + + > [!NOTE] + > Remember to `git add` your project! + +4. Use the `.` path as the `nix run` source. + So, for example, if you want to create a proof for your program with [Jolt](https://jolt.a16zcrypto.com/), you can run: + + ```sh + nix run .#jolt/NAME -- prove + ``` + + Where `NAME` is the name of your program (inside `guests`). + +## Command arguments + +When issuing `nix run`, a number of binaries are built for the relevant zkVM(s) and program, and the "main" binary is ran. +This program has a constant argument schema accross all possibile configurations. + +All arguments after `--` are passed to it. +As a start, you should look at the built-in help message. +Further in this section there are some common configurations of arguments you may want to use. + +```sh +nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- --help +``` + +``` +A CLI tool for running and benchmarking guest programs inside a zkVM environment. +This binary has been built with a single zkVM and guest program in mind. If you +want to run or benchmark your own guest program inside a zkVM, head on over to +https://github.com/blocksense-network/zkVMs-benchmarks + +Usage: host-sp1 [OPTIONS] [PRIVATE_INPUT] [PUBLIC_INPUT] + +Arguments: + What should the zkVM do with the guest + [possible values: execute, prove, verify] + [PRIVATE_INPUT] Path to private input file (in TOML format) + [PUBLIC_INPUT] Path to public input file (in TOML format) + +Options: + -b, --benchmark + Enable benchmark timer and formatted output + -r, --repeat + Benchmark the given action multiple times + -m, --millis + Output timings as milliseconds instead of seconds + -o, --metrics-output + Put the benchmark's formatted output into a file of the given path + -a, --append + Append the benchmark formatted output to the given file, instead of + replacing it + -h, --help + Print help +``` + +### Example: benchmark a single program + +As already mentioned, if you omit a zkVM when issuing `nix run`, all zkVMs will be ran for the given program. +However, when benchmarking, to get a useable output, you need to use `--metrics-output` with `--append`: + +```sh +nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove --benchmark --metrics-output result.csv --append +``` + +### Example: benchmark a single program with custom input and millisecond precision + +Extending on the previous example, we can pass public and private input TOML files as positional arguments, after `prove`: + +```sh +nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove ./private.toml ./public.toml -bamo result.csv +``` + +Other formats and taking inputs from standard input is not supported. + +## Benchmark metrics output format + +When running with the `--benchmark` attribute, additional data is emitted - the metrics output. +This is a very simple CSV content, with two columns: the first a name and the second a value. + +Here is a table with the currently available pairs: + +| Name | Value type | Shown | +| -------- | ---------- | ------ | +| zkvm | String | Always | +| guest | String | Always | +| duration | Integer | Always | +| repeats | Integer | Always | +| average | Integer | Always | + +### Example output + +```csv +zkvm,sp1 +guest,fibonacci +duration,4 +repeats,1 +average,4 +``` + +## Limitations + +Due to the complicated ways in which Nix ([craneLib](https://crane.dev/)) and cargo interact, most of the packages in this repository do **not** compile without Nix. +This also means that incremental/debug builds are not really possible. + +As of writing, user-defined input types (argument types to the entrypoint function) are not supported. +This also includes types defined by any libraries you may use. -- cgit v1.2.3