Decision Trees and Random Forests

Decision Trees and Random Forests

How a Decision Tree Works

A decision tree is a flowchart-style model that splits data into smaller groups by asking a series of yes/no questions about feature values. At each internal node the algorithm chooses the feature and threshold that best separates the classes — in scikit-learn the default criterion for classification is Gini impurity, which measures how often a randomly chosen item would be mislabeled if it were randomly classified according to the distribution at that node. The tree keeps splitting until it reaches a leaf node, which returns a predicted class.

Because a tree can keep splitting until every training sample is correctly classified, an unpruned tree almost always achieves near-perfect training accuracy but performs poorly on new data. That gap between training performance and validation (or test) performance is called overfitting. The model has memorized the training set instead of learning general patterns.

Controlling Overfitting with Hyperparameters

The most direct way to limit overfitting in a single decision tree is to restrict how deep the tree can grow using the `max_depth` hyperparameter in scikit-learn. A shallow tree (for example, `max_depth=3`) forces the model to capture only the most important splits and tends to generalize better. You compare training accuracy versus validation accuracy across several depth values; when training accuracy climbs much faster than validation accuracy, the model is overfitting.

Other useful hyperparameters include `min_samples_split` (the minimum number of samples required to split a node) and `min_samples_leaf`. Adjusting these can also reduce overfitting without imposing a hard depth limit.

From One Tree to a Forest

A random forest is an ensemble of decision trees. It reduces the variance that makes a single tree unstable through two mechanisms:

1. Bootstrap aggregation (bagging): Each tree is trained on a random sample of the training data drawn with replacement. No single tree sees the full dataset.

2. Feature randomness: At every split, each tree considers only a random subset of features rather than all of them. This decorrelates the trees so their errors are not all in the same direction.

The final prediction is determined by a majority vote across all trees. Because individual trees err in different directions, averaging their votes cancels out much of the random noise, producing a model with lower variance than any single tree — even though each individual tree may still overfit its bootstrap sample.

Key hyperparameters for `RandomForestClassifier` in scikit-learn:

• `n_estimators`: number of trees. More trees generally improve stability up to a point; runtime increases proportionally.
• `max_depth`: maximum depth of each individual tree.
• `max_features`: number of features considered at each split (default `"sqrt"` for classification).

Feature Importances

Scikit-learn calculates feature importance for tree-based models by measuring the total reduction in impurity (Gini or entropy) that a feature produces across all splits and all trees, weighted by the number of samples that reach those splits. The values are normalized so they sum to 1.0. A higher value means that feature contributed more to splitting the data cleanly. Feature importances are a fast way to understand which variables drive the model's decisions and to communicate results to stakeholders.

Important caveat: tree-based importances can be biased toward high-cardinality features (features with many unique values). When you need rigorous feature attribution, complement this with permutation importance or SHAP values.

Validation Curves and Hyperparameter Tuning

A validation curve plots a performance metric (such as accuracy) on the y-axis against a hyperparameter value (such as `max_depth`) on the x-axis, showing separate lines for training and validation scores. The ideal operating point is where validation accuracy is highest before it begins to decline — the point just before the training-validation gap widens significantly.

A manual grid search means you loop over a list of hyperparameter combinations, train a model for each, record validation metrics, and select the combination with the best validation score. Scikit-learn's `GridSearchCV` automates this with cross-validation, but doing it manually in a loop first helps you understand exactly what is happening.

Responsible and Safe Data Practices

When working with medical or financial datasets — such as the Heart Disease UCI or Credit Approval UCI datasets used in this lab — you must follow data-handling protocols:

• Data privacy: Never upload patient or financial records to a public repository or share them outside approved channels. Work only within the environment your program or employer has authorized.
• Dataset provenance: Confirm the dataset's license before use. UCI Machine Learning Repository datasets are provided for research and educational purposes; review any terms before applying them commercially.
• Bias awareness: Medical and financial datasets frequently contain historical biases. Feature importances can surface proxy variables (for example, a ZIP code that correlates with race). Document these risks and flag them in your analysis.
• Reproducibility: Set a random seed (`random_state` parameter) in every model you train so results can be reproduced exactly by an instructor or future colleague.

Connecting to the Credentials

Decision trees and random forests appear in the IBM Data Science Professional Certificate curriculum and are tested indirectly in CompTIA Data+ through questions on modeling concepts and data analysis techniques. Understanding variance, bias, and model validation is also foundational knowledge for the Google Data Analytics Certificate capstone and the data modeling content of Microsoft PL-300.