Exploratory Data Analysis on Urban Datasets

What Is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the structured practice of examining a dataset before drawing conclusions or building models. The goal is not to prove a hypothesis — it is to understand what you actually have. In urban data work, raw datasets from city open-data portals, census sources, or municipal APIs often arrive with missing values, inconsistent formatting, and distributions that look nothing like a textbook example. EDA surfaces those problems early so they do not corrupt later analysis.

The Google Data Analytics Certificate frames EDA within the Ask and Prepare phases: you define what question you are trying to answer, then rigorously assess whether the data you have is fit to answer it. CompTIA Data+ Domain 2 (data mining) reinforces this by requiring analysts to profile data, assess completeness, and detect anomalies before any transformation or modeling step.

The Standard EDA Workflow

A reproducible EDA follows a consistent sequence. Working inside a Jupyter Notebook keeps your code, outputs, and narrative in one place, which is essential for a project portfolio.

Step 1 — Data overview. Load the dataset with `pandas` and immediately check its shape (`df.shape`), column names, and data types (`df.dtypes`). Confirm that numeric columns are actually stored as numeric types, not as strings.

Step 2 — Missing-data assessment. Calculate the null percentage for every column: `df.isnull().sum() / len(df) * 100`. Any column missing more than roughly 20–30 percent of its values needs a documented decision — impute, drop, or flag — before analysis proceeds.

Step 3 — Descriptive statistics. Run `df.describe()` for numeric columns and `df.value_counts()` for categorical ones. Note the minimum, maximum, mean, median, and standard deviation. A large gap between mean and median signals a skewed distribution; a minimum of zero in a variable like housing units often indicates a data-entry placeholder rather than a true zero.

Step 4 — Distribution plots. Histograms (via `matplotlib` or `seaborn`) show the shape of each numeric variable. Boxplots reveal the interquartile range and flag potential outliers as points beyond 1.5 times the IQR from the quartile boundaries. For urban datasets — rental prices, commute times, population density — right-skewed distributions are common and meaningful.

Step 5 — Correlation heatmap. Use `seaborn.heatmap(df.corr())` to display pairwise Pearson correlation coefficients. Values near +1 or −1 indicate strong linear relationships; values near 0 suggest little linear association. Remember that correlation does not imply causation, and that non-linear relationships will appear weak here even if they are real.

Interpreting Urban Context

A correlation coefficient or a skewed histogram has no meaning until you connect it to the city question you are investigating. If you are studying neighborhood rent burden and you observe a right-skewed rent distribution, the plain statistical fact is that a small number of high-rent observations pull the mean well above the median. The urban interpretation is that median rent is the more honest measure of what a typical renter pays, and that reporting the mean would overstate affordability. Annotating every plot with one sentence of urban-context interpretation is not optional decoration — it is the analytical habit that separates a data scientist from a spreadsheet operator.

Identifying Data-Quality Issues

At minimum, your EDA report must document at least two data-quality issues. Common ones in municipal datasets include:

• Structural zeros vs. true zeros — a zero in a population column may mean the tract is unpopulated or that the record was not collected.
• Inconsistent geographic identifiers — FIPS codes, ZIP codes, and neighborhood names may overlap or conflict across joined tables.
• Temporal mismatches — datasets collected in different years joined as if they are contemporaneous.
• Outliers from entry error — a property listed at a rent of $1 or $99,999 likely reflects a data-entry convention, not a market price.

Document each issue, note which column it affects, and state your planned treatment.

Revising the Research Question

EDA sometimes reveals that the original research question cannot be answered with the available data. That is a legitimate and important finding, not a failure. If a key variable has 40 percent missing values or is available only at the county level when your question requires the tract level, you must revise the question to match what the data can support. Document the original question and the revised version in your EDA report; reviewers and future employers read this section to assess your analytic judgment.

Data Safety and Professional Responsibility

Urban datasets frequently include personally identifiable information (PII) or quasi-identifiers — small census tracts, exact addresses, income bands — that can re-identify individuals when combined. Before publishing or sharing any EDA output, confirm that your dataset is either fully aggregated or has been cleared for public release by the data provider. Never post raw record-level city data to a public GitHub repository without explicit permission. Store working data files in the designated project folder with access limited to team members. These are professional obligations, not suggestions.