Gradient-Boosting anything (alert: high performance): Part3, Histogram-based boosting

A few weeks ago, I intoduced a model-agnostic gradient boosting procedure, that can use any base learner (available in R and Python package mlsauce):

• https://thierrymoudiki.github.io/blog/2024/10/06/python/r/genericboosting
• https://thierrymoudiki.github.io/blog/2024/10/14/r/genericboosting-r

The rationale is different from other histogram-based gradient boosting algorithms, as histograms are only used here for feature engineering of continuous features. So far, I don’t see huge differences with the original implementation of the GenericBooster, but it’s still a work in progress. I envisage to try it out on a data set that contains a ‘higher’ mix of continuous and categorical features (as categorical features are not histogram-engineered).

Here are a few results that can give you an idea of the performance of the algorithm:

!pip install git+https://github.com/Techtonique/mlsauce.git --verbose --upgrade --no-cache-dir

import os
import mlsauce as ms
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from time import time

load_models = [load_breast_cancer, load_iris, load_wine, load_digits]

for model in load_models:

    data = model()
    X = data.data
    y= data.target

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 13)

    clf = ms.LazyBoostingClassifier(verbose=0, ignore_warnings=True, #n_jobs=2,
                                    custom_metric=None, preprocess=False)

    start = time()
    models, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=True)
    models2, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=False)
    print(f"\nElapsed: {time() - start} seconds\n")

    display(models)
    display(models2)

2it [00:00,  2.27it/s]
100%|██████████| 38/38 [00:41<00:00,  1.09s/it]
2it [00:00,  5.14it/s]
100%|██████████| 38/38 [00:43<00:00,  1.14s/it]


Elapsed: 85.95083284378052 seconds

2it [00:00,  6.46it/s]
100%|██████████| 38/38 [00:12<00:00,  3.11it/s]
2it [00:00, 10.38it/s]
100%|██████████| 38/38 [00:11<00:00,  3.18it/s]


Elapsed: 24.71835470199585 seconds

2it [00:00,  5.45it/s]
100%|██████████| 38/38 [00:14<00:00,  2.63it/s]
2it [00:00,  9.26it/s]
100%|██████████| 38/38 [00:14<00:00,  2.58it/s]


Elapsed: 29.76035761833191 seconds

2it [00:01,  1.90it/s]
100%|██████████| 38/38 [09:30<00:00, 15.02s/it]
2it [00:01,  1.03it/s]
100%|██████████| 38/38 [09:27<00:00, 14.94s/it]


Elapsed: 1141.7054164409637 seconds