XGBoost Classifier

How XGBoost Classifier Works

          +-------------------+
          |   Input Features  |
          +--------+----------+
                   |
                   v
          +--------+----------+
          |  Initial Prediction |
          +--------+----------+
                   |
                   v
          +--------+----------+
          |  Compute Residuals |
          +--------+----------+
                   |
                   v
        +----------+-----------+
        | Train Decision Tree 1 |
        +----------+-----------+
                   |
                   v
        +----------+-----------+
        | Update Prediction    |
        +----------+-----------+
                   |
                   v
        +----------+-----------+
        | Train Decision Tree 2 |
        +----------+-----------+
                   |
                  ...
                   |
                   v
        +----------+-----------+
        | Final Output (Ensemble) |
        +------------------------+

📊 XGBoost Classifier: Core Formulas and Concepts

1. Model Structure

XGBoost builds an additive model composed of K decision trees:

ŷ_i = ∑_{k=1}^K f_k(x_i), where f_k ∈ F

Here, F is the space of regression trees.

2. Objective Function

The learning objective is composed of a loss function and regularization term:

Obj(θ) = ∑ l(y_i, ŷ_i) + ∑ Ω(f_k)

3. Regularization Term

To prevent overfitting, XGBoost uses the following regularization:

Ω(f) = γT + (1/2) λ ∑ w_j²

Where T is the number of leaves, and w_j is the score on each leaf.

4. Gradient and Hessian

To optimize the objective, it uses second-order Taylor approximation:

g_i = ∂_{ŷ} l(y_i, ŷ_i)
h_i = ∂²_{ŷ} l(y_i, ŷ_i)

5. Tree Structure Score

To choose a split, the gain is computed as:

Gain = 1/2 * [ (G_L² / (H_L + λ)) + (G_R² / (H_R + λ)) - (G² / (H + λ)) ] - γ

Where G = ∑ g_i and H = ∑ h_i in respective branches.

Practical Use Cases for Businesses Using XGBoost Classifier

• Churn Prediction. Companies analyze customer behavior to predict churn rate, enabling proactive retention strategies tailored to at-risk customers.
• Credit Scoring. Financial institutions use XGBoost to assess risk accurately, determining creditworthiness for loans while minimizing defaults.
• Sales Forecasting. Businesses leverage historical sales data processed with XGBoost to predict future sales trends, allowing for better inventory and resource management.
• Fraud Detection. XGBoost assists financial firms in identifying fraudulent transactions through anomaly detection, ensuring security and trust in financial operations.
• Image Classification. Companies apply XGBoost in machine learning for image recognition tasks, such as sorting images or detecting objects within them, enhancing automation processes.

Example 1: Binary Classification with Log Loss

Loss function:

l(y, ŷ) = -[y log(ŷ) + (1 - y) log(1 - ŷ)]

For a sample with y = 1 and ŷ = 0.7:

Loss = -[1 * log(0.7) + 0 * log(0.3)] = -log(0.7) ≈ 0.357

Example 2: Computing Gain for a Tree Split

Suppose:

G_L = 10, H_L = 4
G_R = 6,  H_R = 2
λ = 1, γ = 0.1

Compute total gain:

Gain = 1/2 * [ (100 / 5) + (36 / 3) - (256 / 7) ] - 0.1
     = 1/2 * [20 + 12 - 36.57] - 0.1
     = 1/2 * -4.57 - 0.1 ≈ -2.385

Since gain is negative, this split would be rejected.

Example 3: Predicting with Final Model

Suppose the final boosted model includes 3 trees:

Tree 1: output = 0.3
Tree 2: output = 0.25
Tree 3: output = 0.4

Sum of outputs:

ŷ = 0.3 + 0.25 + 0.4 = 0.95

If using logistic sigmoid for binary classification:

σ(ŷ) = 1 / (1 + exp(-0.95)) ≈ 0.721

Final predicted probability = 0.721

import xgboost as xgb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load data and split
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train XGBoost Classifier
model = xgb.XGBClassifier(use_label_encoder=False, eval_metric='mlogloss')
model.fit(X_train, y_train)

# Predict and evaluate
y_pred = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
  

# Train with early stopping
model = xgb.XGBClassifier(use_label_encoder=False, eval_metric='mlogloss')
model.fit(
    X_train, y_train,
    early_stopping_rounds=10,
    eval_set=[(X_test, y_test)],
    verbose=False
)
  

Types of XGBoost Classifier

• Binary Classifier. The binary classifier is used for tasks where there are two possible output classes, such as spam detection in emails. It learns from labeled examples to predict one of two classes.
• Multi-Class Classifier. This type can classify instances into multiple categories, such as classifying images into different objects. The multi-class classifier supports various models and enables accurate predictions across multiple classes.
• Ranking Classifier. Ranking classifiers are useful in applications where the order or importance of items matters, such as search results. This type ranks items based on their predicted relevance.
• Regression Classifier. Although primarily a classification tool, XGBoost can also be adapted for regression tasks. This classifier predicts continuous values, like house prices based on certain features.
• Scalable Classifier. The scalable classifier leverages distributed computing to handle extremely large datasets. It is optimized for use on modern cloud computing platforms, allowing businesses to analyze vast amounts of data quickly.

Conclusion

XGBoost Classifier remains a powerful tool in artificial intelligence, favored for its accuracy and efficiency in various applications. As industries continue to evolve, XGBoost’s capabilities will adapt and expand, ensuring that it remains relevant in the face of technological advancements.

Top Articles on XGBoost Classifier