Scalable robust covariance and correlation estimates for data mining

Covariance and correlation estimates have important applications in data mining. In the presence of outliers, classical estimates of covariance and correlation matrices are not reliable. A small fraction of outliers, in some cases even a single outlier, can distort the classical covariance and correlation estimates making them virtually useless. That is, correlations for the vast majority of the data can be very erroneously reported; principal components transformations can be misleading; and multidimensional outlier detection via Mahalanobis distances can fail to detect outliers. There is plenty of statistical literature on robust covariance and correlation matrix estimates with an emphasis on affineequivariant estimators that possess high breakdown points and small worst case biases. All such estimators have unacceptable exponential complexity in the number of variables and quadratic complexity in the number of observations. In this paper we focus on several variants of robust covar...

• Type: Conference
• Year: 2002
• Where: KDD
• Authors: Fatemah A. Alqallaf, Kjell P. Konis, R. Douglas Martin, Ruben H. Zamar
• Keywords: Correlation Matrix Estimates, Data Mining, KDD 2002, Pairwise Robust Covariance, Robust Estimators