Introducing DIAD: A Novel Metric for Assessing the Difficulty of Anomaly Detection Problems

Assessing the complexity of anomaly detection tasks is essential for benchmarking datasets and models as well as for understanding the problem domains. While numerous anomaly detection methods have been developed, there remains a need for a simple, learning-free metric to estimate the difficulty of a given anomaly detection task. In this paper, we introduce DIAD (Difficulty Index for Anomaly Detection), a lightweight metric designed to quantify task difficulty without requiring model training or inference. DIAD builds upon and extends the recently proposed AD3 metric by incorporating both the saliency of anomalies and the heterogeneity of normal appearance across the dataset. We evaluate DIAD on five widely used visual anomaly detection datasets and compare its scores with the observed performance of three state-of-the-art detection models. Results show that DIAD correlates more consistently with model performance than AD3, offering a practical and interpretable tool for assessing the complexity of anomaly detection problems.