A Bad Day to Die Hard: Correcting the Dieharder Battery

We analyze Dieharder statistical randomness tests according to accuracy and correct interpretation of their results. We used all tests, processed 8 TB of quantum-generated data, and obtained null distributions of first-level and second-level p-values. We inspected whether the p-values are uniformly distributed. The analysis showed that more than half (out of 110) of Dierharder atomic tests (test with particular setting) produce null distributions of p-values that are biased from the expected uniform one. Additional analysis of the Kolmogorov–Smirnov (KS) test showed that the key KS test is also biased. This increases the probability of false positives (in the right tail) for all Dieharder tests as KS is used to post-process their results. Moreover, 12 tests (22 atomic) produce results significantly biased from the null distribution of the KS test which may suggest problems with the implementation of these tests.