Generated from pytest-benchmark JSON output
main@e9b775ac
Write operation benchmarks across all providers.; Read operation benchmarks across all providers.; Benchmarks through the full Index API.
Parameters: n = entries to store, any_provider = storage backend
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_store_single_entry[sqlite]
Store single entry [sqlite storage backend] |
197.60 μs | 309.84 μs | 3.04 Kops/s |
|
test_store_single_entry[redis]
Store single entry [redis storage backend] |
163.29 μs | 18.64 μs | 5.93 Kops/s |
|
test_store_entries_bulk[sqlite-10]
Store entries bulk [sqlite storage backend, 10 entries to store] |
2.00 ms | 752.24 μs | 337.0 ops/s |
|
test_store_entries_bulk[sqlite-100]
Store entries bulk [sqlite storage backend, 100 entries to store] |
21.84 ms | 5.82 ms | 41.0 ops/s |
|
test_store_entries_bulk[sqlite-1k]
Store entries bulk [sqlite storage backend, 1000 entries to store] |
244.06 ms | 47.92 ms | 4.3 ops/s |
|
test_store_entries_bulk[redis-10]
Store entries bulk [redis storage backend, 10 entries to store] |
1.65 ms | 75.47 μs | 589.6 ops/s |
|
test_store_entries_bulk[redis-100]
Store entries bulk [redis storage backend, 100 entries to store] |
16.75 ms | 762.98 μs | 58.2 ops/s |
|
test_store_entries_bulk[redis-1k]
Store entries bulk [redis storage backend, 1000 entries to store] |
168.53 ms | 6.85 ms | 5.9 ops/s |
|
test_store_schema[sqlite]
Store schema [sqlite storage backend] |
158.05 μs | 287.80 μs | 3.85 Kops/s |
|
test_store_schema[redis]
Store schema [redis storage backend] |
155.06 μs | 19.94 μs | 6.21 Kops/s |
|
test_store_schema_versions[sqlite-10v]
Store schema versions [sqlite storage backend, 10 entries to store] |
1.71 ms | 2.89 ms | 378.6 ops/s |
|
test_store_schema_versions[sqlite-50v]
Store schema versions [sqlite storage backend, 50 entries to store] |
8.36 ms | 2.37 ms | 92.9 ops/s |
|
test_store_schema_versions[redis-10v]
Store schema versions [redis storage backend, 10 entries to store] |
1.64 ms | 252.17 μs | 595.5 ops/s |
|
test_store_schema_versions[redis-50v]
Store schema versions [redis storage backend, 50 entries to store] |
8.16 ms | 1.10 ms | 120.3 ops/s |
|
test_get_entry_by_name[sqlite]
Get entry by name [sqlite storage backend] |
9.02 μs | 139.5 ns | 107.56 Kops/s |
|
test_get_entry_by_name[redis]
Get entry by name [redis storage backend] |
56.97 ms | 1.04 ms | 17.4 ops/s |
|
test_get_entry_by_cid[sqlite]
Get entry by cid [sqlite storage backend] |
9.12 μs | 121.0 ns | 106.26 Kops/s |
|
test_get_entry_by_cid[redis]
Get entry by cid [redis storage backend] |
171.13 μs | 25.66 μs | 5.58 Kops/s |
|
test_iter_entries[sqlite-10]
Iter entries [sqlite storage backend, 10 entries to store] |
36.98 μs | 261.0 ns | 26.55 Kops/s |
|
test_iter_entries[sqlite-100]
Iter entries [sqlite storage backend, 100 entries to store] |
323.26 μs | 12.57 μs | 3.04 Kops/s |
|
test_iter_entries[sqlite-1k]
Iter entries [sqlite storage backend, 1000 entries to store] |
3.35 ms | 48.96 μs | 251.0 ops/s |
|
test_iter_entries[redis-10]
Iter entries [redis storage backend, 10 entries to store] |
192.07 ms | 11.48 ms | 5.1 ops/s |
|
test_iter_entries[redis-100]
Iter entries [redis storage backend, 100 entries to store] |
191.97 ms | 5.61 ms | 5.2 ops/s |
|
test_iter_entries[redis-1k]
Iter entries [redis storage backend, 1000 entries to store] |
199.54 ms | 4.17 ms | 5.0 ops/s |
|
test_get_schema_json[sqlite]
Get schema json [sqlite storage backend] |
5.24 μs | 70.0 ns | 187.04 Kops/s |
|
test_get_schema_json[redis]
Get schema json [redis storage backend] |
151.25 μs | 18.46 μs | 6.37 Kops/s |
|
test_find_latest_version[sqlite-5v]
Find latest version [sqlite storage backend, 5 entries to store] |
10.48 μs | 131.0 ns | 93.55 Kops/s |
|
test_find_latest_version[sqlite-20v]
Find latest version [sqlite storage backend, 20 entries to store] |
24.48 μs | 270.0 ns | 40.02 Kops/s |
|
test_find_latest_version[sqlite-50v]
Find latest version [sqlite storage backend, 50 entries to store] |
52.47 μs | 591.0 ns | 18.80 Kops/s |
|
test_find_latest_version[redis-5v]
Find latest version [redis storage backend, 5 entries to store] |
16.44 ms | 423.04 μs | 60.1 ops/s |
|
test_find_latest_version[redis-20v]
Find latest version [redis storage backend, 20 entries to store] |
19.03 ms | 368.73 μs | 51.7 ops/s |
|
test_find_latest_version[redis-50v]
Find latest version [redis storage backend, 50 entries to store] |
26.73 ms | 3.00 ms | 39.0 ops/s |
|
test_iter_schemas[sqlite]
Iter schemas [sqlite storage backend] |
17.13 μs | 191.0 ns | 57.59 Kops/s |
|
test_iter_schemas[redis]
Iter schemas [redis storage backend] |
18.95 ms | 430.34 μs | 52.6 ops/s |
|
test_index_insert_dataset
Index insert dataset |
499.24 μs | 615.67 μs | 1.41 Kops/s |
|
test_index_get_dataset
Index get dataset |
18.05 μs | 230.0 ns | 53.80 Kops/s |
|
test_index_list_datasets
Index list datasets |
8.71 μs | 121.0 ns | 112.47 Kops/s |
|
test_index_publish_schema
Index publish schema |
290.57 μs | 412.45 μs | 2.03 Kops/s |
Shard writing throughput benchmarks.; Shard reading and iteration benchmarks.; Full write-then-read round-trip benchmarks.
Parameters: n = samples per shard, batch_size = samples per batch
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_write_basic_shard[100]
Write basic shard [100 samples per shard] |
8.21 ms | 171.82 μs | 120.3 ops/s | 82.07 μs | 12.18 Kops/s |
|
test_write_basic_shard[1k]
Write basic shard [1000 samples per shard] |
77.21 ms | 575.08 μs | 12.9 ops/s | 77.21 μs | 12.95 Kops/s |
|
test_write_basic_shard[10k]
Write basic shard [10000 samples per shard] |
763.17 ms | 32.87 ms | 1.3 ops/s | 76.32 μs | 13.10 Kops/s |
|
test_write_numpy_shard[100]
Write numpy shard [100 samples per shard] |
33.03 ms | 1.03 ms | 30.1 ops/s | 330.33 μs | 3.03 Kops/s |
|
test_write_numpy_shard[1k]
Write numpy shard [1000 samples per shard] |
350.12 ms | 32.15 ms | 2.8 ops/s | 350.12 μs | 2.86 Kops/s |
|
test_write_large_numpy_shard
Write large numpy shard |
8.168 s | 57.55 ms | 0.1 ops/s | 816.82 ms | 1.2 ops/s |
|
test_write_with_manifest[100]
Write with manifest [100 samples per shard] |
13.83 ms | 96.60 μs | 72.2 ops/s | 138.33 μs | 7.23 Kops/s |
|
test_write_with_manifest[1k]
Write with manifest [1000 samples per shard] |
96.81 ms | 668.32 μs | 10.3 ops/s | 96.81 μs | 10.33 Kops/s |
|
test_write_multi_shard
Write multi shard |
753.78 ms | 2.69 ms | 1.3 ops/s | 75.38 μs | 13.27 Kops/s |
|
test_read_ordered[100]
Read ordered [100 samples per shard] |
7.18 ms | 110.38 μs | 138.5 ops/s | 71.80 μs | 13.93 Kops/s |
|
test_read_ordered[1k]
Read ordered [1000 samples per shard] |
67.77 ms | 461.91 μs | 14.7 ops/s | 67.77 μs | 14.75 Kops/s |
|
test_read_ordered[10k]
Read ordered [10000 samples per shard] |
670.15 ms | 7.37 ms | 1.5 ops/s | 67.01 μs | 14.92 Kops/s |
|
test_read_shuffled[100]
Read shuffled [100 samples per shard] |
6.96 ms | 61.38 μs | 141.5 ops/s | 69.62 μs | 14.36 Kops/s |
|
test_read_shuffled[1k]
Read shuffled [1000 samples per shard] |
69.51 ms | 646.16 μs | 14.4 ops/s | 69.51 μs | 14.39 Kops/s |
|
test_read_batched[batch32]
Read batched [32 samples per batch] |
65.85 ms | 502.64 μs | 15.1 ops/s | 65.85 μs | 15.19 Kops/s |
|
test_read_batched[batch128]
Read batched [128 samples per batch] |
64.79 ms | 568.26 μs | 15.4 ops/s | 64.79 μs | 15.43 Kops/s |
|
test_read_numpy_ordered[100]
Read numpy ordered [100 samples per shard] |
14.48 ms | 313.70 μs | 68.8 ops/s | 144.82 μs | 6.91 Kops/s |
|
test_read_numpy_ordered[1k]
Read numpy ordered [1000 samples per shard] |
144.56 ms | 840.65 μs | 6.9 ops/s | 144.56 μs | 6.92 Kops/s |
|
test_roundtrip_basic[100]
Roundtrip basic [100 samples per shard] |
15.13 ms | 144.68 μs | 66.0 ops/s | 151.26 μs | 6.61 Kops/s |
|
test_roundtrip_basic[1k]
Roundtrip basic [1000 samples per shard] |
142.57 ms | 904.65 μs | 7.0 ops/s | 142.57 μs | 7.01 Kops/s |
|
test_roundtrip_numpy[100]
Roundtrip numpy [100 samples per shard] |
41.95 ms | 342.87 μs | 23.8 ops/s | 419.51 μs | 2.38 Kops/s |
|
test_roundtrip_numpy[500]
Roundtrip numpy [500 samples per shard] |
209.16 ms | 388.69 μs | 4.8 ops/s | 418.32 μs | 2.39 Kops/s |
Benchmark different query predicate types on a medium dataset.; Benchmark iterating through query results to access sample locations.; Benchmark query performance at different scales.; Benchmark manifest loading from disk.; Benchmark manifest construction from samples.
Parameters: n_shards = number of shards (100 samples each), n = samples in manifest
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_query_simple_equality
Query simple equality |
8.86 ms | 144.85 μs | 111.6 ops/s | — | — |
|
test_query_numeric_range
Query numeric range |
9.46 ms | 169.79 μs | 104.9 ops/s | — | — |
|
test_query_combined
Query combined |
4.60 ms | 52.91 μs | 216.8 ops/s | — | — |
|
test_query_isin
Query isin |
15.07 ms | 351.66 μs | 65.3 ops/s | — | — |
|
test_query_no_results
Query no results |
1.88 ms | 23.31 μs | 529.7 ops/s | — | — |
|
test_query_all_results
Query all results |
32.38 ms | 262.78 μs | 30.8 ops/s | — | — |
|
test_iterate_equality_results
Iterate equality results |
8.86 ms | 118.90 μs | 112.5 ops/s | 44.28 μs | 22.58 Kops/s |
|
test_iterate_range_results
Iterate range results |
19.52 ms | 245.72 μs | 51.0 ops/s | 35.50 μs | 28.17 Kops/s |
|
test_iterate_large_result_set
Iterate large result set |
302.02 ms | 17.21 ms | 3.2 ops/s | 30.20 μs | 33.11 Kops/s |
|
test_query_small
Query small |
2.14 ms | 37.54 μs | 464.7 ops/s | — | — |
|
test_query_medium
Query medium |
19.59 ms | 247.43 μs | 51.1 ops/s | — | — |
|
test_query_large
Query large |
169.62 ms | 893.20 μs | 5.9 ops/s | — | — |
|
test_load_from_directory[2s]
Load from directory [2 number of shards (100 samples each)] |
3.31 ms | 27.16 μs | 300.6 ops/s | — | — |
|
test_load_from_directory[5s]
Load from directory [5 number of shards (100 samples each)] |
8.27 ms | 55.45 μs | 120.6 ops/s | — | — |
|
test_load_from_directory[10s]
Load from directory [10 number of shards (100 samples each)] |
16.53 ms | 127.42 μs | 58.1 ops/s | — | — |
|
test_load_from_directory[20s]
Load from directory [20 number of shards (100 samples each)] |
33.25 ms | 267.11 μs | 29.9 ops/s | — | — |
|
test_load_from_shard_urls[2s]
Load from shard urls [2 number of shards (100 samples each)] |
3.25 ms | 26.91 μs | 306.2 ops/s | — | — |
|
test_load_from_shard_urls[5s]
Load from shard urls [5 number of shards (100 samples each)] |
8.21 ms | 51.96 μs | 116.7 ops/s | — | — |
|
test_load_from_shard_urls[10s]
Load from shard urls [10 number of shards (100 samples each)] |
16.42 ms | 84.27 μs | 60.9 ops/s | — | — |
|
test_manifest_build[100]
Manifest build [100 samples in manifest] |
1.35 ms | 22.70 μs | 736.1 ops/s | 13.54 μs | 73.87 Kops/s |
|
test_manifest_build[1k]
Manifest build [1000 samples in manifest] |
8.38 ms | 74.20 μs | 114.8 ops/s | 8.38 μs | 119.30 Kops/s |
|
test_manifest_build[5k]
Manifest build [5000 samples in manifest] |
39.88 ms | 453.55 μs | 21.8 ops/s | 7.98 μs | 125.37 Kops/s |
|
test_manifest_write[100]
Manifest write [100 samples in manifest] |
3.71 ms | 70.04 μs | 268.6 ops/s | 37.05 μs | 26.99 Kops/s |
|
test_manifest_write[1k]
Manifest write [1000 samples in manifest] |
7.05 ms | 72.30 μs | 141.5 ops/s | 7.05 μs | 141.90 Kops/s |
S3 shard writing benchmarks via moto mock.
Parameters: n = samples per shard
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_s3_write_shards[100]
S3 write shards [100 samples per shard] |
19.81 ms | 625.45 μs | 48.4 ops/s | 198.07 μs | 5.05 Kops/s |
|
test_s3_write_shards[500]
S3 write shards [500 samples per shard] |
86.99 ms | 732.38 μs | 11.5 ops/s | 173.99 μs | 5.75 Kops/s |
|
test_s3_write_with_manifest
S3 write with manifest |
58.42 ms | 2.47 ms | 15.4 ops/s | 292.11 μs | 3.42 Kops/s |
|
test_s3_write_cache_local
S3 write cache local |
43.37 ms | 745.37 μs | 21.0 ops/s | 216.83 μs | 4.61 Kops/s |
|
test_s3_write_direct
S3 write direct |
36.46 ms | 563.65 μs | 27.4 ops/s | 182.32 μs | 5.48 Kops/s |
|
test_s3_write_numpy
S3 write numpy |
44.03 ms | 698.27 μs | 22.5 ops/s | 440.29 μs | 2.27 Kops/s |
Pure serialization/deserialization without disk I/O.
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_serialize_basic_sample
Serialize basic sample |
1.69 μs | 41.0 ns | 576.95 Kops/s |
|
test_deserialize_basic_sample
Deserialize basic sample |
1.97 μs | 60.0 ns | 489.35 Kops/s |
|
test_serialize_numpy_sample
Serialize numpy sample |
19.65 μs | 310.0 ns | 49.55 Kops/s |
|
test_deserialize_numpy_sample
Deserialize numpy sample |
5.74 μs | 101.0 ns | 169.84 Kops/s |
|
test_serialize_large_numpy
Serialize large numpy |
356.81 ms | 2.46 ms | 2.8 ops/s |
|
test_deserialize_large_numpy
Deserialize large numpy |
26.74 ms | 814.46 μs | 37.1 ops/s |
|
test_as_wds_basic
As wds basic |
5.12 μs | 71.0 ns | 190.19 Kops/s |
|
test_as_wds_numpy
As wds numpy |
25.03 μs | 501.0 ns | 38.88 Kops/s |