🌳 Random Decision Trees & Forests

Decision Tree Basics

A decision tree is a flowchart: start at the root, ask a yes/no question about a feature, follow a branch, repeat until reaching a leaf (prediction).

Example: Predict if loan will default. Split on income: "income > $50k?" If yes, check credit score. If no, check employment status. Leaf: "approve" or "deny".

Trees are interpretable (you can trace the logic) but prone to overfitting (too many splits memorize training data noise).

The Forest: Ensemble of Trees

Build many trees, each slightly different (random subsets of data, random feature splits). Prediction: average (regression) or majority vote (classification) across all trees.

Why it works: individual trees are imperfect, but averaging independent imperfect models reduces variance and overfitting.

Random Forests: Key Ideas

1. Bootstrapping (bagging): For each tree, sample training data with replacement. Each tree sees a slightly different dataset.

2. Random feature selection: At each split in each tree, choose best split from a random subset of features (not all features). Forces diversity; prevents one dominant feature from controlling all trees.

3. Shallow trees: Usually grow trees to full depth but stop before massive overfitting. Combine many weak learners.

4. Out-of-bag (OOB) error: Data not selected for a tree's bootstrap sample can estimate generalization error without a separate test set.

Hyperparameters to Tune

• n_estimators: Number of trees. More = better (up to a point), but slower. Usually 100–1000.
• max_depth: Max depth of each tree. Shallow trees reduce overfitting. No limit (None) = grow until pure.
• min_samples_split: Min samples to split a node. Higher = shallower trees, less overfitting.
• max_features: Number of random features to consider per split. Lower = more randomness = less correlation between trees.
• random_state: Seed for reproducibility.

Advantages

• Low bias, low variance: Ensemble reduces overfitting compared to single trees.
• Handles mixed data: Numeric and categorical features, no scaling needed.
• Feature importance: Rank features by how much they reduce impurity across the forest.
• Robust: Works with missing data, outliers, correlated features.
• Parallelizable: Trees are independent; train on many cores.

Disadvantages

• Less interpretable than a single tree: Hard to explain "why" when averaging 1000 trees.
• Memory and computation: Storing many trees costs memory; prediction slower than single tree.
• Still overfits on very noisy data: No algorithm is magic.
• Biased on highly imbalanced data: Majority class dominates splits.

When to Use Random Forests

• Mixed numeric/categorical features without manual feature engineering
• Unknown feature interactions (trees learn them automatically)
• Non-linear relationships
• Need feature importance rankings
• Baseline model (hard to beat for tabular data)

Variants & Extensions

Extremely Randomized Trees (Extra Trees): Use random split thresholds instead of optimized splits. Faster training, can reduce overfitting more.

Gradient Boosting (XGBoost, LightGBM): Build trees sequentially, each correcting errors of the previous. Often outperforms random forests but slower to train and more tuning required.

Random Forests for regression: Predict continuous values instead of classes. Average tree predictions instead of voting.