What is Outlier Detection?
Outlier detection is the process of identifying data points that deviate significantly from the rest of the observations in a dataset. These unusual values can represent measurement errors, data entry mistakes, genuine extreme values, or the most interesting cases in your dataset.
This tool implements the two most widely used methods: the IQR method (Tukey fences) and the Z-score method. By running both methods simultaneously and comparing results, you get a more reliable picture of which points truly warrant investigation.
Methods Used
- IQR Method (Tukey Fences): Values outside [Q1 − 1.5×IQR, Q3 + 1.5×IQR] are flagged. Robust to non-normal distributions, this is the standard method taught in introductory statistics.
- Z-Score Method: Values with |z| > 3 are considered outliers (more than 3 standard deviations from the mean). Best suited for approximately normally distributed data.
What to Do with Outliers
- Investigate. Check if the outlier is a data entry error or measurement artifact. Fix or remove if it is clearly erroneous.
- Keep. If the outlier represents a genuine, valid observation, consider keeping it — especially if later analysis will use robust methods.
- Winsorize. Replace the extreme value with the nearest non-outlier value. This retains the observation without letting it dominate statistics like the mean.
- Transform. Apply log, square root, or Box-Cox transformation to reduce the influence of extreme values across the dataset.
Frequently Asked Questions
Which outlier detection method should I use?
What should I do with detected outliers?
Can I adjust the outlier detection sensitivity?
Key Terms
- IQR Method
- Data points outside [Q1−1.5×IQR, Q3+1.5×IQR] are flagged as outliers.
- Z-Score Method
- Values with |z| > 3 are considered statistical outliers.
- Tukey Fences
- The boundary defined by Q1−1.5×IQR and Q3+1.5×IQR.
Computation Method
- Quartiles are calculated using linear interpolation (method 7 from Hyndman & Fan, 1996), consistent with Python's NumPy and pandas defaults.
- Outlier detection uses the Tukey fences method: lower fence = Q1 − 1.5 × IQR, upper fence = Q3 + 1.5 × IQR.
- Chart scaling caps the display axis at the upper fence value so the visualization remains clear even when extreme outliers are present.
References
- Hyndman, R. J. & Fan, Y. (1996). “Sample Quantiles in Statistical Packages.” The American Statistician, 50(4), 361–365.
- Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.