Exploratory Data Analysis with pandas and NumPy

What Is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the structured practice of summarizing, visualizing, and questioning a dataset before building models or drawing conclusions. The goal is to understand what the data actually contains — its shape, distributions, missing values, and anomalies — so that every downstream decision rests on solid ground. EDA is a core competency tested by the Google Data Analytics Certificate, the IBM Data Science Professional Certificate, and CompTIA Data+.

Loading Data with pandas

pandas is the standard Python library for working with tabular data. You load a CSV file into a DataFrame with `pd.read_csv('filename.csv')`. Once loaded, call `df.shape` to confirm the number of rows and columns, and `df.dtypes` to see how pandas has interpreted each column's data type. A quick `df.head()` lets you inspect the first few rows.

For each column, four pandas methods form the backbone of your data dictionary:

• `df.describe()` — returns count, mean, standard deviation, min, quartiles, and max for numeric columns.
• `df['col'].value_counts()` — counts how often each unique value appears; essential for categorical columns.
• `df.isnull().sum()` — counts missing values in every column at once.
• `df.nunique()` — counts distinct values per column, helping you distinguish true categorical columns from columns that look numeric.

Write one paragraph per column in a Markdown cell. State the column's name, data type, number of unique values, the most common value (for categoricals) or the descriptive statistics (for numerics), and the count of missing values. This is your data dictionary, and it is the artifact that stakeholders can read without opening your code.

Computing Statistics with NumPy

NumPy operates on arrays and is highly efficient for numerical computation. After isolating numeric columns with `df.select_dtypes(include='number')`, you can compute:

• Mean: `np.mean(arr)` — the arithmetic average.
• Median: `np.median(arr)` — the middle value; robust to extreme values.
• Standard deviation: `np.std(arr, ddof=1)` — use `ddof=1` for sample standard deviation.
• Interquartile range (IQR): `np.percentile(arr, 75) - np.percentile(arr, 25)` — the spread of the middle 50 percent of values.

Flagging Outliers with the Three-Sigma Rule

A common, defensible method is to flag any observation whose value falls more than three standard deviations from the column mean. For a column named `response_time`:

```python

mean = np.mean(df['response_time'])

std = np.std(df['response_time'], ddof=1)

df['outlier_response_time'] = (np.abs(df['response_time'] - mean) > 3 * std)

```

The result is a Boolean column: `True` means the value is a potential outlier. Always inspect flagged rows manually — an outlier may be a data-entry error, a legitimate extreme event, or evidence of a second population in your data. Do not delete outliers without documenting your reasoning.

A note on the IQR method: Some analysts prefer flagging values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. Both methods are valid; document which you chose and why.

Writing the EDA Narrative

A Jupyter Markdown cell can contain formatted text alongside your code. Your five-paragraph narrative must cover:

1. Source — Where did the data come from? State the URL, agency, and date accessed.

2. Structure — How many rows and columns? What time period or geography does it cover?

3. Key distributions — Which numeric columns are skewed? Which categories dominate? Mention specific numbers.

4. Missing-data pattern — Which columns have missing values? Is missingness random or concentrated in specific categories or time periods?

5. Stakeholder questions — Pose two concrete, answerable questions. For example: "Does average response time differ by neighborhood?" or "Which request type generates the most repeat calls?"

Writing the narrative forces you to translate code output into plain language — a professional skill valued in every data role.

Data Integrity and Professional Ethics

Before publishing any EDA, verify that your dataset's terms of use permit analysis and sharing. Open-government datasets (data.gov, city open-data portals) are generally public, but confirm the license. Never re-publish personally identifiable information (PII). If a column contains names, addresses, or phone numbers, note its presence in your data dictionary but mask or drop values in any shared artifact.

Workspace Safety Note

Data science work is primarily a computer workstation environment. Follow your institution's acceptable-use policy for networked systems. When downloading files from external sources, save them to a designated project folder — do not run files from unknown sources, as malicious code can be embedded in data files. Use virtual environments to isolate package dependencies and prevent conflicts across projects.