Target Encoding

How Target Encoding Works

+------------------------+
|  Raw Categorical Data  |
+-----------+------------+
            |
            v
+-----------+------------+
| Calculate Target Mean  |
|  for Each Category     |
+-----------+------------+
            |
            v
+-----------+------------+
|  Apply Smoothing (α)   |
+-----------+------------+
            |
            v
+-----------+------------+
| Replace Categories     |
| with Encoded Values    |
+-----------+------------+
            |
            v
+-----------+------------+
| Model Training Stage   |
+------------------------+

Overview of Target Encoding

Target Encoding transforms categorical features into numerical values by replacing each category with the average of the target variable for that category. This allows models to leverage meaningful numeric signals instead of arbitrary categories.

Calculating Category Averages

First, compute the mean of the target (e.g., probability of class or average outcome) for each category in the training data. These values reflect the relationship between category and target, capturing predictive information.

Smoothing to Prevent Overfitting

Target Encoding often applies smoothing, blending the category mean with the global target mean. A smoothing parameter (α) controls how much weight is given to category-specific versus global information, reducing noise in rare categories.

Integration into Model Pipelines

Once encoded, the transformed numerical feature replaces the original category in the dataset. This new representation is used in model training and inference, providing richer and more informative inputs for both regression and classification models.

🎯 Target Encoding: Core Formulas and Concepts

1. Basic Target Encoding Formula

For a categorical value c in feature X, the encoded value is:

TE(c) = mean(y | X = c)

2. Global Mean Encoding

Used to reduce overfitting, especially for rare categories:

TE_smooth(c) = (∑ y_c + α * μ) / (n_c + α)

Where:

y_c = sum of target values for category c
n_c = count of samples with category c
μ = global mean of target variable
α = smoothing factor

3. Regularized Encoding with K-Fold

To avoid target leakage, encoding is done using out-of-fold mean:

TE_kfold(c) = mean(y | X = c, excluding current fold)

4. Log-Transformation for Classification

For binary classification (target 0 or 1):

TE_log(c) = log(P(y=1 | c) / (1 − P(y=1 | c)))

5. Final Feature Vector

The encoded column replaces or augments the original categorical feature:

X_encoded = [TE(x₁), TE(x₂), ..., TE(xₙ)]

Practical Use Cases for Businesses Using Target Encoding

• Customer Segmentation. Target encoding helps identify segments based on behavioral patterns by translating categorical demographics into meaningful numerical metrics.
• Churn Prediction. Businesses can effectively model customer churn by encoding customer features to understand which demographic groups are at higher risk.
• Sales Forecasting. Utilizing target encoding allows businesses to incorporate qualitative sales factors and improve forecasts on revenue generation.
• Fraud Detection. By encoding categorical data about transactions, organizations can better identify patterns associated with fraudulent activities.
• Risk Assessment. Target encoding is useful in risk assessment applications, helping in quantifying the impact of categorical risk factors on future outcomes.

Example 1: Simple Mean Encoding

Feature: “City”

London → [y = 1, 0, 1], mean = 0.67
Paris → [y = 0, 0], mean = 0.0
Berlin → [y = 1, 1], mean = 1.0

Target Encoded values:

London → 0.67
Paris → 0.00
Berlin → 1.00

Example 2: Smoothed Encoding

Global mean μ = 0.6, smoothing α = 5

Category A: 2 samples, total y = 1

TE = (1 + 5 * 0.6) / (2 + 5) = (1 + 3) / 7 = 4 / 7 ≈ 0.571

Smoothed encoding stabilizes values for low-frequency categories

Example 3: K-Fold Encoding to Prevent Leakage

5-fold cross-validation

When encoding “Region” feature, mean target is computed excluding the current fold:

Fold 1: TE(region X) = mean(y) from folds 2-5
Fold 2: TE(region X) = mean(y) from folds 1,3,4,5
...

This ensures that the target encoding is unbiased and generalizes better

import pandas as pd

# Sample dataset
df = pd.DataFrame({
    'color': ['red', 'blue', 'red', 'green', 'blue'],
    'purchased': [1, 0, 1, 0, 1]
})

# Compute mean target for each category
target_mean = df.groupby('color')['purchased'].mean()

# Map means to the original column
df['color_encoded'] = df['color'].map(target_mean)

print(df)
  

def target_encode_smooth(df, cat_col, target_col, alpha=10):
    global_mean = df[target_col].mean()
    agg = df.groupby(cat_col)[target_col].agg(['mean', 'count'])
    smoothing = (agg['count'] * agg['mean'] + alpha * global_mean) / (agg['count'] + alpha)
    return df[cat_col].map(smoothing)

df['color_encoded_smooth'] = target_encode_smooth(df, 'color', 'purchased', alpha=5)
print(df)
  

Types of Target Encoding

• Mean Target Encoding. This method replaces each category with the mean of the target variable for that category. It effectively captures the relationship between the categorical feature and the target but can lead to overfitting if not managed carefully.
• Weighted Target Encoding. This approach combines the mean of the target variable with a global mean in order to reduce the impact of noise from categories with few samples. It balances the insights captured from individual category means with overall trends.
• Leave-One-Out Encoding. Each category is replaced with the average of the target variable from the other samples while excluding the current sample. This reduces leakage but increases computational complexity.
• Target Encoding with Smoothing. This technique blends the category mean with the overall target mean using a predefined ratio. Smoothing is useful when categories have very few observations, helping to prevent overfitting.
• Cross-Validation Target Encoding. Here, target encoding is applied within a cross-validation framework, ensuring that the encoding values are derived only from the training data. This significantly reduces the risk of data leakage.

Conclusion

Target encoding is a powerful technique that transforms categorical variables into a format suitable for machine learning. By effectively creating numerical representations, it enables models to learn from data efficiently, leading to better predictive performance. As the technology continues to develop, its applications and value in AI will only increase.

Top Articles on Target Encoding