Outlier Detection and Treatment

What Is an Outlier?

An outlier is an observation that lies an unusual distance from other values in a dataset. Outliers are not automatically errors — they may be genuine extreme values, data-entry mistakes, or signals of exactly the event you are trying to detect. Because outliers can distort means, standard deviations, regression slopes, and machine-learning model weights, you must decide what to do with them before analysis — and document every decision.

Method 1: The IQR Fence Rule

The interquartile range (IQR) measures the spread of the middle 50 percent of a distribution.

1. Compute Q1 (25th percentile) and Q3 (75th percentile) for the column.

2. IQR = Q3 − Q1

3. Lower fence = Q1 − 1.5 × IQR

4. Upper fence = Q3 + 1.5 × IQR

Any value below the lower fence or above the upper fence is flagged as a mild outlier. Multiplying by 3.0 instead of 1.5 gives the "extreme outlier" fences. This rule is nonparametric — it makes no assumption about the distribution shape — and it is the same rule seaborn uses when drawing a boxplot's whiskers.

In pandas:

```python

Q1 = df['salary'].quantile(0.25)

Q3 = df['salary'].quantile(0.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR

upper = Q3 + 1.5 * IQR

outliers = df[(df['salary'] < lower) | (df['salary'] > upper)]

```

Method 2: Z-Scores

A z-score expresses how many standard deviations a value sits from the column mean. The formula is:

> z = (x − mean) / standard deviation

Values with |z| > 3 are commonly flagged as outliers in normally distributed data. The z-score method is sensitive to the very outliers it is trying to detect because extreme values inflate the mean and standard deviation. For that reason, use it alongside the IQR method, not instead of it.

In pandas/numpy:

```python

from scipy import stats

df['salary_z'] = stats.zscore(df['salary'])

outliers_z = df[df['salary_z'].abs() > 3]

```

Method 3: Boxplot Inspection

A seaborn boxplot draws the box from Q1 to Q3, a line at the median, whiskers at the fences, and individual dots for values beyond the fences. Visual inspection lets you see the count and distribution of outliers at a glance. Always produce boxplots before and after any treatment so reviewers can verify your work.

```python

import seaborn as sns, matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

sns.boxplot(y=df['salary'], ax=axes[0]).set_title('Before – n outliers=12')

sns.boxplot(y=df_clean['salary'], ax=axes[1]).set_title('After removal – n outliers=0')

plt.tight_layout()

```

Annotate each plot with the count of affected records so any reader can audit your choices.

Choosing a Treatment

The right treatment depends on the analysis goal, not just the data.

| Goal | Recommended treatment | Reasoning |

|---|---|---|

| Estimate a population mean (e.g., average city salary) | Remove or Winsorize | Extreme values pull the mean away from the typical worker |

| Detect fraud or anomalies | Flag and retain | The outlier is the signal; removing it destroys evidence |

| Build a predictive model | Investigate first, then decide | Could be noise or a rare but real pattern |

Removal deletes the flagged rows. Document how many rows were removed and confirm the remaining dataset is still large enough to support your analysis.

Flagging adds a boolean column (`is_outlier = True/False`) and keeps the row. Downstream analysis can then include or exclude outliers as needed.

Winsorization caps values at the fence boundaries rather than removing them. The row count stays the same, but extreme values are pulled in to the fence. This preserves sample size while limiting distortion.

```python

from scipy.stats.mstats import winsorize

import numpy as np

df['salary_w'] = winsorize(df['salary'], limits=[0.05, 0.05])

```

`limits=[0.05, 0.05]` caps the bottom 5 percent and the top 5 percent at their respective 5th- and 95th-percentile values. Adjust the limits to match the actual fence positions in your data.

Comparing Before and After

After any treatment, produce a summary table of mean, median, and standard deviation before and after. A well-executed Winsorization typically:

• Lowers the mean slightly if there were high outliers
• Leaves the median nearly unchanged (the median is already resistant to outliers)
• Reduces the standard deviation

If the median changes substantially, investigate — you may have over-Winsorized.

Data Safety and Documentation Standards

Data cleaning is irreversible if you overwrite the original file. Always work on a copy of the dataset and keep the raw CSV untouched. Use version control (Git) or clearly named file versions (`salary_raw.csv`, `salary_clean_v1.csv`). In a professional setting, every outlier treatment decision should be recorded in a data-cleaning log or notebook markdown cell, stating the method used, the threshold applied, the number of records affected, and the business reason for the choice. This is an auditable record that protects you and your organization if results are questioned later.