Skip to content

Reproducibility hub

Run a real baseline in five minutes

Exact commands, pinned environment, deterministic seeds and expected outputs. The quickstart needs no network and no dataset download — it runs on tiny synthetic fixtures shipped with the benchmark.

Environment

5-minute quickstart (no network)

Verified to run
# 1. clone and install the benchmark package
git clone https://github.com/mandip42/rirmega-benchmark.git
cd rirmega-benchmark/rirbench
pip install -e ".[dev]"

# 2. generate the tiny synthetic fixtures (3 RIRs, known reverberation)
python fixtures/make_fixtures.py

# 3. run the full test + smoke suite
pytest -q

# 4. compute every reference metric for one RIR
rirbench metrics fixtures/medium_t60_0p6.wav

Expected output for step 4 (the medium fixture targets T60 = 0.6 s; the estimator recovers it to within ~2%):

{
  "rt60_t20_s": 0.6129,
  "rt60_t30_s": 0.6046,
  "edt_s": 0.5802,
  "drr_db": -11.95,
  "c50_db": 3.32,
  "c80_db": 7.52,
  "d50": 0.683
}

Run each track

# Track A — acoustic validity (recompute + agreement)
rirbench track-a --smoke

# Track B — synthetic↔real generalization protocol
rirbench track-b --smoke

# Track C — downstream ASR (WER/CER path)
rirbench track-c --smoke

Each --smoke run finishes in seconds and emits JSON in the exact shape used on the Results page.

Full run against the real dataset

Install the [hub] extra, then point Track A at a local copy of the RIR waveforms and the published metadata to measure agreement:

pip install -e ".[hub]"

# download metadata (honors HF_HUB_DOWNLOAD_TIMEOUT so it fails fast, not hangs)
python -c "from rirbench.data import download_rirmega_metadata as d; print(d())"

# recompute metrics for your local RIR .wav dir and compare to published metadata
rirbench track-a --dir /path/to/rir_wavs --meta /path/to/metadata_v2.parquet \
  --out track_a_full.json

Dataset: mandipgoswami/rirmega, revision main, license CC-BY-NC-4.0.

Checksums & versions

Troubleshooting