1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
# zkVMs-benchmarks
This is a repository with setups and programs for zero-knowledge virtual machine benchmarking.
Its ultimate goal is to deliver reproducible and accurate performance metrics accross all zkVMs, so **you** can choose which technology suits your needs!
Being made with reproducibility in mind, this project also serves as a good framework for running programs accross zkVMs without the complicated and everchanging setups required to do so.
## Usage
The backbones of this entire codebase are [Nix](https://nixos.org/) and Linux.
MacOS is not supported!
Windows is supported via WSL.
**First**, install the Nix package manager, follow their [download instructions](https://nixos.org/download/).
Generally it should be enough to:
```sh
sh <(curl -L https://nixos.org/nix/install) --daemon
```
> [!WARNING]
> It is preferrable to use the nixos.org script, as shown above!
> Certain systems provide Nix with their native package managers, however practice has shown those do not always lead to working setups!
Now, what follows depends on your usecase.
### Run/benchmark a "built-in" program
The `guests` directory provides a variety of programs, all of which are proven to work with their default inputs.
It is advisable to first try and run one of them, so you can make sure Nix is installed and working properly.
The smallest one is `fibonacci`.
To make *all* zkVMs generate a proof for it just run:
```sh
nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove
```
> [!NOTE]
> Take notice of the space between `--` and `prove`!
> It marks an "end of options", as specified by [the POSIX specification](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02).
Or to make, for example, [SP1](https://docs.succinct.xyz/docs/sp1/introduction) generate a proof and verify it, you may run:
```sh
nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- verify
```
### Run/benchmark your own program
1. Clone the [git repository](https://github.com/blocksense-network/zkVMs-benchmarks)
```sh
git clone git@github.com:blocksense-network/zkVMs-benchmarks.git
```
2. Navigate to the `guests` directory
```sh
cd zkVMs-benchmarks/guests
```
3. Follow the instructions inside the `guests/README.md` file to setup your program
> [!NOTE]
> Remember to `git add` your project!
4. Use the `.` path as the `nix run` source.
So, for example, if you want to create a proof for your program with [Jolt](https://jolt.a16zcrypto.com/), you can run:
```sh
nix run .#jolt/NAME -- prove
```
Where `NAME` is the name of your program (inside `guests`).
## Command arguments
When issuing `nix run`, a number of binaries are built for the relevant zkVM(s) and program, and the "main" binary is ran.
This program has a constant argument schema accross all possibile configurations.
All arguments after `--` are passed to it.
As a start, you should look at the built-in help message.
Further in this section there are some common configurations of arguments you may want to use.
```sh
nix run github:blocksense-network/zkVMs-benchmarks#sp1/fibonacci -- --help
```
```
A CLI tool for running and benchmarking guest programs inside a zkVM environment.
This binary has been built with a single zkVM and guest program in mind. If you
want to run or benchmark your own guest program inside a zkVM, head on over to
https://github.com/blocksense-network/zkVMs-benchmarks
Usage: host-sp1 [OPTIONS] <RUN_TYPE> [PRIVATE_INPUT] [PUBLIC_INPUT]
Arguments:
<RUN_TYPE> What should the zkVM do with the guest
[possible values: execute, prove, verify]
[PRIVATE_INPUT] Path to private input file (in TOML format)
[PUBLIC_INPUT] Path to public input file (in TOML format)
Options:
-b, --benchmark
Enable benchmark timer and formatted output
-r, --repeat <REPEAT>
Benchmark the given action multiple times
-m, --millis
Output timings as milliseconds instead of seconds
-o, --metrics-output <METRICS_OUTPUT>
Put the benchmark's formatted output into a file of the given path
-a, --append
Append the benchmark formatted output to the given file, instead of
replacing it
-h, --help
Print help
```
### Example: benchmark a single program
As already mentioned, if you omit a zkVM when issuing `nix run`, all zkVMs will be ran for the given program.
However, when benchmarking, to get a useable output, you need to use `--metrics-output` with `--append`:
```sh
nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove --benchmark --metrics-output result.csv --append
```
### Example: benchmark a single program with custom input and millisecond precision
Extending on the previous example, we can pass public and private input TOML files as positional arguments, after `prove`:
```sh
nix run github:blocksense-network/zkVMs-benchmarks#fibonacci -- prove ./private.toml ./public.toml -bamo result.csv
```
Other formats and taking inputs from standard input is not supported.
## Benchmark metrics output format
When running with the `--benchmark` attribute, additional data is emitted - the metrics output.
This is a very simple CSV content, with two columns: the first a name and the second a value.
Here is a table with the currently available pairs:
| Name | Value type | Shown |
| -------- | ---------- | ------ |
| zkvm | String | Always |
| guest | String | Always |
| duration | Integer | Always |
| repeats | Integer | Always |
| average | Integer | Always |
### Example output
```csv
zkvm,sp1
guest,fibonacci
duration,4
repeats,1
average,4
```
## Limitations
Due to the complicated ways in which Nix ([craneLib](https://crane.dev/)) and cargo interact, most of the packages in this repository do **not** compile without Nix.
This also means that incremental/debug builds are not really possible.
As of writing, user-defined input types (argument types to the entrypoint function) are not supported.
This also includes types defined by any libraries you may use.
|
