Benchmark results
Published, reproduced and new — kept separate
Results are grouped by provenance. The numbers below the fold are genuine rirbench outputs — on the shipped fixtures and on a bounded real sample of the RIR-Mega test split; full-dataset and licensed-corpus runs are marked pending until executed and committed.
Published baselines
The RIR-Mega paper and repo report parameter-estimation baselines (RT60/DRR/C50 regression; classical, 1D CNN, 1D Transformer). See the paper and repo for the authors' numbers.
Independently reproduced
Results a maintainer or third party reproduced from submitted instructions, within the stated uncertainty.
New experiments
Fresh results generated by this suite. The fixture-scale runs below are real; full-dataset runs are marked pending.
Track A — acoustic validity (fixture-scale, real output)
Reference metrics recomputed from 3 synthetic fixture RIRs with known target reverberation. The estimator recovers RT60 within a few percent, which validates the Schroeder pipeline end-to-end.
| sample_id | RT60 T20 (s) | RT60 T30 (s) | EDT (s) | DRR (dB) | C50 (dB) | D50 |
|---|---|---|---|---|---|---|
| dry_t60_0p2 | 0.201 | 0.197 | 0.212 | -7.01 | 15.01 | 0.969 |
| medium_t60_0p6 | 0.613 | 0.605 | 0.580 | -11.95 | 3.32 | 0.683 |
| wet_t60_1p2 | 1.203 | 1.197 | 1.197 | -14.56 | -0.95 | 0.446 |
Fixtures target T60 = 0.2 / 0.6 / 1.2 s. Sample size n = 3. Hardware/compute: trivial (CPU, seconds). Caveat: synthetic fixtures, not RIR-Mega samples — they validate the pipeline, not the dataset.
Track A — acoustic validity (bounded real sample, recomputed vs. published)
Reference metrics recomputed directly from 100 real RIR waveforms in the RIR-Mega rir_output_1k test split, compared against the dataset's own published metrics. This is an internal-consistency / validity check — not a claim that the published values are wrong.
| metric | n | bias | MAE | RMSE | MAE 95% CI | recomp. mean | pub. mean |
|---|---|---|---|---|---|---|---|
| rt60_broadband_vs_t20 | 100 | -0.025 | 0.029 | 0.043 | [0.024, 0.036] | 0.425 | 0.449 |
| rt60_broadband_vs_t30 | 100 | -0.001 | 0.001 | 0.002 | [0.001, 0.002] | 0.448 | 0.449 |
| drr_db | 100 | -1.198 | 1.728 | 2.126 | [1.487, 1.982] | -2.193 | -0.994 |
| c50_db | 100 | 10.345 | 10.345 | 28.542 | [5.880, 15.711] | 12.794 | 2.450 |
| c80_db | 100 | 3.565 | 3.565 | 4.287 | [3.117, 4.007] | 18.441 | 14.876 |
Estimator: rirbench Schroeder backward-integration (T20/T30), pure-numpy LS slope. Channel convention: first channel of linear8 array. Sample size n = 100 (entire test split). Hardware/compute: CPU, < 1 min. CI method: bootstrap (1000 resamples, seed 0).
Reading the numbers honestly: recomputed RT60 (T30) agrees with the published broadband RT60 to within ~0.001 s on average — strong validity evidence. DRR agrees within ~1.7 dB MAE. C50/C80 differ by several dB: the published clarity figures use a different direct-path window / band convention than this estimator's, so the gap reflects a definitional difference to document, not a dataset error. Published rt60 is a single broadband value; recomputed T20/T30 are independent estimates from the waveform. Agreement reflects estimator + definitional differences, not dataset error.
Track C — downstream ASR path (real WER/CER)
The RIR-Mega-Speech paper (arXiv:2601.19949) reports a real ASR-robustness baseline: Whisper small on 1,500 paired clean↔reverberant utterances. These are the author's published figures with 95% CIs — reproduced here as a reference, not run by this portal.
Clean 5.20% (95% CI 4.69–5.78) → reverberant 7.70% (95% CI 7.04–8.35), a paired increase of 2.50 pp (95% CI 2.06–2.98) — 48% relative degradation. WER increases monotonically with RT60 and decreases with DRR.
Word- and character-error-rate on a documented example clean↔reverberant transcript pair, proving the scoring path runs with no model download. Clean is recognized perfectly; the reverberant hypothesis degrades, as expected.
Full Whisper runs reproducing the paper's numbers (model versions, LibriSpeech splits, compute) are documented in whisper-rirmega-bench. A large-scale run committed to this portal is pending.
Track B — generalization
Track B is a protocol: it compares RIR-Mega against measured corpora (ACE, REVERB) the researcher supplies under their own licenses. Distribution and domain-shift tables are pending a run against a measured corpus. The smoke run (self-comparison, shift ≈ 0) confirms the harness is correct.