optuna
**XGBoost: eXtreme Gradient Boosting**
**Overview**
XGBoost is a scalable, distributed gradient-boosted decision tree (GBDT) library. For nearly a decade, it has been the "King of Kaggle," winning more competitions than any other algorithm on tabular data.
**Why is it so good?**
**1. Regularization**
It includes L1 and L2 regularization in the objective function, preventing overfitting better than standard Gradient Boosting.
**2. Speed**
- **Column Block Structure**: Parallelizes tree construction.
- **Hardware Optimization**: Cache-aware access patterns.
**3. Handling Missing Values**
It automatically learns the best direction (left or right) to handle missing values ('NaN') in the data.
**Usage (Python)**
```python
import xgboost as xgb
# DMatrix (Internal efficient format)
dtrain = xgb.DMatrix(X_train, label=y_train)
# Parameters
param = {'max_depth': 2, 'eta': 1, 'objective': 'binary:logistic'}
# Train
bst = xgb.train(param, dtrain, num_boost_round=10)
# Predict
preds = bst.predict(dtest)
```
**Competition**
Recently, **LightGBM** (Microsoft) and **CatBoost** (Yandex) have challenged XGBoost's dominance by offering faster training speeds and better categorical handling, but XGBoost remains the gold standard baseline.