Exploratory Data Analysis with the Tidyverse

Exploratory Data Analysis with the Tidyverse

In the previous lesson you explored a dataset using pandas in Python. Now you will perform the same exploration in R using the tidyverse — a collection of packages designed around a consistent philosophy of tidy data. By the end of this lesson you will be able to group and summarize data with dplyr, clean messy text with stringr, and reshape tables with tidyr. Comparing your R results to the pandas output you already produced is a built-in accuracy check: if the numbers agree (within rounding caused by floating-point differences), both analyses are likely correct.

Why the Tidyverse?

The tidyverse packages share a common grammar and pipe-friendly design. A pipe (`|>` in base R 4.1+ or `%>%` from magrittr) passes the output of one function directly into the next, making code read like a series of English instructions. This reduces the need for intermediate variables and makes the data-transformation story easier to audit — an important quality in professional analytics work.

Grouping and Summarizing with dplyr

The two workhorses are `group_by()` and `summarize()` (also spelled `summarise()`). `group_by()` tags each row with its group membership without changing the data; `summarize()` then collapses each group into a single row of statistics.

```r

library(tidyverse)

df <- read_csv("lesson6_dataset.csv")

summary_tbl <- df |>

group_by(region) |>

summarize(

n = n(),

mean_sales = mean(sales, na.rm = TRUE),

med_sales = median(sales, na.rm = TRUE)

) |>

arrange(desc(mean_sales))

write_csv(summary_tbl, "region_summary.csv")

```

Key points:

• `na.rm = TRUE` tells R to skip missing values when calculating the mean and median — the same behavior you achieved with `skipna=True` in pandas.
• `arrange(desc(...))` sorts from largest to smallest, matching a pandas `sort_values(ascending=False)` call.
• Repeat the `group_by()` block for at least two additional categorical columns (for example, product category and customer segment) to satisfy the three-grouping requirement.

When you compare these results to your pandas output, minor decimal differences (e.g., 1234.56 vs. 1234.5600000001) are acceptable; they arise from floating-point arithmetic and do not indicate an error. Larger discrepancies should be investigated — a common cause is one tool dropping NAs while the other includes them.

Cleaning Text with stringr

Real datasets often contain inconsistent formatting: city names like `"New York"`, `"new york"`, and `" New York "` are all different strings to a computer. The stringr package provides a predictable, pipe-friendly interface to string operations.

| Function | Purpose |

|---|---|

| `str_to_lower()` | Convert all characters to lowercase |

| `str_trim()` | Remove leading and trailing whitespace |

| `str_replace_all()` | Replace a pattern with a new string |

```r

before_cities <- n_distinct(df$city)

df <- df |>

mutate(

city = str_trim(city),

city = str_to_lower(city)

)

after_cities <- n_distinct(df$city)

cat("Unique city values before:", before_cities, "\n")

cat("Unique city values after: ", after_cities, "\n")

```

Record both counts in your lab log. A decrease in distinct values after cleaning confirms that duplicates caused by formatting differences have been merged. This documentation is important for reproducibility: a future analyst (or your future self) can see exactly what changed and why.

Reshaping Data with tidyr

Data comes in two shapes. Wide format places each variable in its own column — convenient for reading but awkward for many ggplot2 visualizations and statistical models. Long format stacks variable names into one column and their values into another — ideal for grouped operations and plotting.

• `pivot_longer()` converts wide → long.
• `pivot_wider()` converts long → wide.

```r

long_tbl <- summary_tbl |>

pivot_longer(

cols = c(mean_sales, med_sales),

names_to = "metric",

values_to = "value"

)

wide_again <- long_tbl |>

pivot_wider(

names_from = "metric",

values_from = "value"

)

stopifnot(nrow(wide_again) == nrow(summary_tbl))

stopifnot(ncol(wide_again) == ncol(summary_tbl))

```

The `stopifnot()` calls act as automatic checks: if row or column counts do not match, R stops and reports an error so you catch the problem immediately.

Documenting in R Markdown

Your deliverable is an R Markdown (`.Rmd`) file that knits to HTML or PDF without errors. Each code chunk should be followed by a plain-English explanation. For the reshape section, write at least one sentence explaining when long format is preferable (modeling, plotting) and when wide format is preferable (display, export to spreadsheets).

Before submitting, knit the document and confirm it completes without errors. A clean knit is the R equivalent of a Python script that runs from top to bottom without exceptions.

Data Integrity and Professional Practice

Although this lesson involves no electrical equipment or chemicals, professional data work carries its own integrity obligations. Never alter the original source file; always read from it and write outputs to new files. Keep your lab log entries factual — record what the data showed, not what you hoped it would show. Misrepresenting data quality or cleaning steps is an ethical violation with real consequences in business and public-sector analytics.