aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorKamen Mladenov <kamen@syndamia.com>2025-02-11 17:58:29 +0200
committerKamen Mladenov <kamen@syndamia.com>2025-02-13 12:58:08 +0200
commit99b5db7522929e9157a5ab20f2bd3c4997ca67ae (patch)
tree878325798df1fb5f24ee94d2527dd0c78ab189b8
parent32a586b87928ea7e23ff3484e416053325a5b068 (diff)
downloadzkVMs-benchmarks-99b5db7522929e9157a5ab20f2bd3c4997ca67ae.tar
zkVMs-benchmarks-99b5db7522929e9157a5ab20f2bd3c4997ca67ae.tar.gz
zkVMs-benchmarks-99b5db7522929e9157a5ab20f2bd3c4997ca67ae.zip
feat: Add README
-rw-r--r--README.md171
1 files changed, 170 insertions, 1 deletions
diff --git a/README.md b/README.md
index 0c24556..ea696a8 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,170 @@
-# zkVMs-benchmarks \ No newline at end of file
+# zkVMs-benchmarks
+
+This is a repository with setups and programs for zero-knowledge virtual machine benchmarking.
+Its ultimate goal is to deliver reproducible and accurate performance metrics accross all zkVMs, so **you** can choose which technology suits your needs!
+
+Being made with reproducibility in mind, this project also serves as a good framework for running programs accross zkVMs without the complicated and everchanging setups required to do so.
+
+## Usage
+
+The backbones of this entire codebase are [Nix](https://nixos.org/) and Linux.
+MacOS is not supported!
+Windows is supported via WSL.
+
+**First**, install the Nix package manager, follow their [download instructions](https://nixos.org/download/).
+Generally it should be enough to:
+
+```sh
+sh <(curl -L https://nixos.org/nix/install) --daemon
+```
+
+> [!WARNING]
+> It is preferrable to use the nixos.org script, as shown above!
+> Certain systems provide Nix with their native package managers, however practice has shown those do not always lead to working setups!
+
+Now, what follows depends on your usecase.
+
+### Run/benchmark a "built-in" program
+
+The `guests` directory provides a variety of programs, all of which are proven to work with their default inputs.
+It is advisable to first try and run one of them, so you can make sure Nix is installed and working properly.
+
+The smallest one is `fibonacci`.
+To make *all* zkVMs generate a proof for it just run:
+
+```sh
+nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove
+```
+
+> [!NOTE]
+> Take notice of the space between `--` and `prove`!
+> It marks an "end of options", as specified by [the POSIX specification](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02).
+
+Or to make, for example, [SP1](https://docs.succinct.xyz/docs/sp1/introduction) generate a proof and verify it, you may run:
+
+```sh
+nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- verify
+```
+
+### Run/benchmark your own program
+
+1. Clone the [git repository](https://github.com/blocksense-network/zkVMs-benchmarks)
+
+ ```sh
+ git clone git@github.com:blocksense-network/zkVMs-benchmarks.git
+ ```
+
+2. Navigate to the `guests` directory
+
+ ```sh
+ cd zkVMs-benchmarks/guests
+ ```
+
+3. Follow the instructions inside the `guests/README.md` file to setup your program
+
+ > [!NOTE]
+ > Remember to `git add` your project!
+
+4. Use the `.` path as the `nix run` source.
+ So, for example, if you want to create a proof for your program with [Jolt](https://jolt.a16zcrypto.com/), you can run:
+
+ ```sh
+ nix run .#jolt/NAME -- prove
+ ```
+
+ Where `NAME` is the name of your program (inside `guests`).
+
+## Command arguments
+
+When issuing `nix run`, a number of binaries are built for the relevant zkVM(s) and program, and the "main" binary is ran.
+This program has a constant argument schema accross all possibile configurations.
+
+All arguments after `--` are passed to it.
+As a start, you should look at the built-in help message.
+Further in this section there are some common configurations of arguments you may want to use.
+
+```sh
+nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- --help
+```
+
+```
+A CLI tool for running and benchmarking guest programs inside a zkVM environment.
+This binary has been built with a single zkVM and guest program in mind. If you
+want to run or benchmark your own guest program inside a zkVM, head on over to
+https://github.com/blocksense-network/zkVMs-benchmarks
+
+Usage: host-sp1 [OPTIONS] <RUN_TYPE> [PRIVATE_INPUT] [PUBLIC_INPUT]
+
+Arguments:
+ <RUN_TYPE> What should the zkVM do with the guest
+ [possible values: execute, prove, verify]
+ [PRIVATE_INPUT] Path to private input file (in TOML format)
+ [PUBLIC_INPUT] Path to public input file (in TOML format)
+
+Options:
+ -b, --benchmark
+ Enable benchmark timer and formatted output
+ -r, --repeat <REPEAT>
+ Benchmark the given action multiple times
+ -m, --millis
+ Output timings as milliseconds instead of seconds
+ -o, --metrics-output <METRICS_OUTPUT>
+ Put the benchmark's formatted output into a file of the given path
+ -a, --append
+ Append the benchmark formatted output to the given file, instead of
+ replacing it
+ -h, --help
+ Print help
+```
+
+### Example: benchmark a single program
+
+As already mentioned, if you omit a zkVM when issuing `nix run`, all zkVMs will be ran for the given program.
+However, when benchmarking, to get a useable output, you need to use `--metrics-output` with `--append`:
+
+```sh
+nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove --benchmark --metrics-output result.csv --append
+```
+
+### Example: benchmark a single program with custom input and millisecond precision
+
+Extending on the previous example, we can pass public and private input TOML files as positional arguments, after `prove`:
+
+```sh
+nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove ./private.toml ./public.toml -bamo result.csv
+```
+
+Other formats and taking inputs from standard input is not supported.
+
+## Benchmark metrics output format
+
+When running with the `--benchmark` attribute, additional data is emitted - the metrics output.
+This is a very simple CSV content, with two columns: the first a name and the second a value.
+
+Here is a table with the currently available pairs:
+
+| Name | Value type | Shown |
+| -------- | ---------- | ------ |
+| zkvm | String | Always |
+| guest | String | Always |
+| duration | Integer | Always |
+| repeats | Integer | Always |
+| average | Integer | Always |
+
+### Example output
+
+```csv
+zkvm,sp1
+guest,fibonacci
+duration,4
+repeats,1
+average,4
+```
+
+## Limitations
+
+Due to the complicated ways in which Nix ([craneLib](https://crane.dev/)) and cargo interact, most of the packages in this repository do **not** compile without Nix.
+This also means that incremental/debug builds are not really possible.
+
+As of writing, user-defined input types (argument types to the entrypoint function) are not supported.
+This also includes types defined by any libraries you may use.