Data Preparation and Feature Engineering for ML

Why Data Preparation Matters in Machine Learning

A machine learning model learns patterns from numbers. Raw, real-world datasets almost never arrive in that ideal state. They contain missing values, text categories that algorithms cannot read, and numerical columns measured on wildly different scales. Skipping preparation steps does not just reduce accuracy — it can produce models that appear to work during training but fail silently in production. Disciplined data preparation is therefore the foundation of every reliable ML workflow.

The Train / Validation / Test Split

Before touching any data transformations, you must split your dataset. The standard three-way split serves distinct purposes:

• Training set — the data the model learns from.
• Validation set — held out during training; used to tune hyperparameters and compare model versions.
• Test set — touched exactly once, after all decisions are final, to report honest performance.

A common ratio is 70 % training, 15 % validation, 15 % test. Use `train_test_split` from scikit-learn twice: first to carve off the test set, then to split the remainder into training and validation. Always set a `random_state` value so your split is reproducible. Stratify on the target column when the classes are imbalanced so each split contains a representative proportion of each class.

Data Leakage — The Critical Safety Rule

Data leakage means information from outside the training set influences the model during fitting. The most common form is fit-on-test leakage: fitting a scaler or imputer on the full dataset before splitting, so statistics from test rows contaminate the training process. This produces optimistic evaluation scores that will not hold in production — a silent, expensive failure.

Rule: fit all transformers only on training data. Transform validation and test data using those already-fitted transformers.

This is not optional. Treat it the same way you would treat a workplace safety rule: violating it creates harm that may not be visible until later, and at significant cost.

Handling Missing Values with SimpleImputer

Missing values appear as `NaN` in a pandas DataFrame. Two strategies cover most cases:

• Mean or median imputation — replace each missing number with the column mean (sensitive to outliers) or median (more robust). Fit the imputer on training data; it stores that mean or median and applies it to new data.
• Most-frequent imputation — replace missing categories with the most common value in that column.

Never drop rows with missing values from your test set; real production data will have gaps too, and your pipeline must handle them.

Encoding Categorical Variables with OneHotEncoder

Algorithms need numbers. A column containing values like `"Male"`, `"Female"` must be converted. One-hot encoding creates one binary column per unique category. For a column with values `{"Red", "Blue", "Green"}`, the encoder produces three columns: `is_Red`, `is_Blue`, `is_Green`. Set `handle_unknown='ignore'` so the encoder does not crash when a category appears in the test set that was not seen during training.

For ordinal categories where order matters (e.g., `"Low" < "Medium" < "High"`), use `OrdinalEncoder` and specify the order explicitly.

Scaling Numerical Features with StandardScaler

Many algorithms — including logistic regression, support vector machines, and k-nearest neighbors — are sensitive to feature scale. A column ranging from 0 to 1 and a column ranging from 0 to 1,000,000 will not be treated equally without scaling.

StandardScaler subtracts the column mean and divides by the standard deviation, producing a distribution with mean 0 and standard deviation 1. It stores the training mean and standard deviation; those same values are applied to validation and test data — never recalculated on them.

Building a scikit-learn Pipeline with ColumnTransformer

A `Pipeline` chains multiple steps so they behave as a single object. A `ColumnTransformer` applies different transformers to different columns in parallel and concatenates the results. Combining them gives you a single, safe, saveable object.

```python

from sklearn.pipeline import Pipeline

from sklearn.compose import ColumnTransformer

from sklearn.impute import SimpleImputer

from sklearn.preprocessing import StandardScaler, OneHotEncoder

numeric_transformer = Pipeline(steps=[

('imputer', SimpleImputer(strategy='median')),

('scaler', StandardScaler())

])

categorical_transformer = Pipeline(steps=[

('imputer', SimpleImputer(strategy='most_frequent')),

('onehot', OneHotEncoder(handle_unknown='ignore'))

])

preprocessor = ColumnTransformer(transformers=[

('num', numeric_transformer, numeric_cols),

('cat', categorical_transformer, categorical_cols)

])

```

Call `preprocessor.fit(X_train)` and then `preprocessor.transform(X_val)` and `preprocessor.transform(X_test)`. Never call `fit` on validation or test data.

Saving the Pipeline

Save the fitted pipeline with `joblib.dump(preprocessor, 'pipeline.pkl')`. This lets you apply the exact same transformations to new data in production without recomputing statistics. Load it later with `joblib.load('pipeline.pkl')`.

Workspace Habits

Always document which columns are numeric and which are categorical at the top of your notebook. Version-control your notebook alongside your pickle file. During peer review, confirm that no `fit` or `fit_transform` call is made on validation or test arrays — this is the leakage-check step and it must be completed before a notebook is considered complete.