Benchmarking 30 statistical/Machine Learning models on the VN1 Forecasting — Accuracy challenge

This post is about the VN1 Forecasting – Accuracy challenge. The aim is to accurately forecast future sales for various products across different
clients and warehouses, using historical sales and pricing data.

Phase 1 was a warmup to get an idea of what works and what wouldn’t (and… for overfitting the validation set, so that the leaderboard is almost meaningless). It’s safe to say, based on empirical observations, that an advanced artillery would be useless here. In phase 2 (people are still welcome to enter the challenge, no pre-requisites from phase 1 needed), the validation set is provided, and there’s no leaderboard for the test set (which is great, basically: “real life”; no overfitting).

My definition of “winning” the challenge will be to have an accuracy close to the winning solution by a factor of 1% (2 decimals). Indeed, the focus on accuracy means: we are litterally targetting a point on the real line (well, the “real line”, the interval is probably bounded but still contains an infinite number of points). If the metric was a metric for quantifying uncertainty… there would be too much winners 🙂

In the examples, I show you how you can start the competition by benchmarking 30 statistical/Machine Learning models on a few products, based on the validation set provided yesterday. No tuning, no overfitting. Only hold-out set validation. You can notice, on some examples, that a model can be the most accurate on point forecasting, but completely off-track when trying to capture the uncertainty aroung the point forecast. Food for thought.

0 – Functions and packages

!pip uninstall nnetsauce --yes
!pip install nnetsauce --upgrade --no-cache-dir

import numpy as np
import pandas as pd

def rm_leading_zeros(df):
    if 'y' in df.columns and (df['y'] == 0).any():
        first_non_zero_index_y = (df['y'] != 0).idxmax()
        df = df.loc[first_non_zero_index_y:].reset_index(drop=True)
    return df.dropna().reset_index(drop=True)

# Read price data
price = pd.read_csv("/kaggle/input/2024-10-02-vn1-forecasting/Phase 0 - Price.csv", na_values=np.nan)
price["Value"] = "Price"
price = price.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Read sales data
sales = pd.read_csv("/kaggle/input/2024-10-02-vn1-forecasting/Phase 0 - Sales.csv", na_values=np.nan)
sales["Value"] = "Sales"
sales = sales.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Read price validation data
price_test = pd.read_csv("/kaggle/input/2024-10-02-vn1-forecasting/Phase 1 - Price.csv", na_values=np.nan)
price_test["Value"] = "Price"
price_test = price_test.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Read sales validation data
sales_test = pd.read_csv("/kaggle/input/2024-10-02-vn1-forecasting/Phase 1 - Sales.csv", na_values=np.nan)
sales_test["Value"] = "Sales"
sales_test = sales_test.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Create single dataframe
df = pd.concat([price, sales]).unstack("Value").reset_index()
df.columns = ["Client", "Warehouse", "Product", "ds", "Price", "y"]
df["ds"] = pd.to_datetime(df["ds"])
df = df.astype({"Price": np.float32,
                "y": np.float32,
                "Client": "category",
                "Warehouse": "category",
                "Product": "category",
                })

df_test = pd.concat([price_test, sales_test]).unstack("Value").reset_index()
df_test.columns = ["Client", "Warehouse", "Product", "ds", "Price", "y"]
df_test["ds"] = pd.to_datetime(df_test["ds"])
df_test = df_test.astype({"Price": np.float32,
                "y": np.float32,
                "Client": "category",
                "Warehouse": "category",
                "Product": "category",
                })

display(df.head())
display(df_test.head())

df.describe()
df_test.describe()

display(df.info())
display(df_test.info())

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2559010 entries, 0 to 2559009
Data columns (total 6 columns):
 #   Column     Dtype         
---  ------     -----         
 0   Client     category      
 1   Warehouse  category      
 2   Product    category      
 3   ds         datetime64[ns]
 4   Price      float32       
 5   y          float32       
dtypes: category(3), datetime64[ns](1), float32(2)
memory usage: 51.6 MB



None


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 195689 entries, 0 to 195688
Data columns (total 6 columns):
 #   Column     Non-Null Count   Dtype         
---  ------     --------------   -----         
 0   Client     195689 non-null  category      
 1   Warehouse  195689 non-null  category      
 2   Product    195689 non-null  category      
 3   ds         195689 non-null  datetime64[ns]
 4   Price      85630 non-null   float32       
 5   y          195689 non-null  float32       
dtypes: category(3), datetime64[ns](1), float32(2)
memory usage: 4.3 MB



None

1 – AutoML for a few products

1 – 1 Select a product

np.random.seed(413)
#np.random.seed(13) # uncomment to select a different product
#np.random.seed(1413) # uncomment to select a different product
#np.random.seed(71413) # uncomment to select a different product
random_series = df.sample(1).loc[:, ['Client', 'Warehouse', 'Product']]
client = random_series.iloc[0]['Client']
warehouse = random_series.iloc[0]['Warehouse']
product = random_series.iloc[0]['Product']
df_filtered = df[(df.Client == client) & (df.Warehouse == warehouse) & (df.Product == product)]
df_filtered = rm_leading_zeros(df_filtered)
display(df_filtered)
df_filtered_test = df_test[(df_test.Client == client) & (df_test.Warehouse == warehouse) & (df_test.Product == product)]
display(df_filtered_test)

df_selected = df_filtered[['y', 'ds']].set_index('ds')
df_selected.index = pd.to_datetime(df_selected.index)
display(df_selected)

df_selected_test = df_filtered_test[['y', 'ds']].set_index('ds')
df_selected_test.index = pd.to_datetime(df_selected_test.index)
display(df_selected_test)

1 – 2 AutoML (Hold-out set)

import nnetsauce as ns
import numpy as np
from time import time

# Custom error metric 
def custom_error(objective, submission):
    try: 
        pred = submission.mean.values.ravel()
        true = objective.values.ravel()
        abs_err = np.nansum(np.abs(pred - true))
        err = np.nansum((pred - true))
        score = abs_err + abs(err)
        score /= true.sum().sum()
    except Exception:
        score = 1000
    return score

regr_mts = ns.LazyMTS(verbose=0, ignore_warnings=True, 
                          custom_metric=custom_error,                      
                          type_pi = "scp2-kde", # sequential split conformal prediction
                          lags = 1, n_hidden_features = 0,
                          sort_by = "Custom metric",
                          replications=250, kernel="tophat",
                          show_progress=False, preprocess=False)
models, predictions = regr_mts.fit(X_train=df_selected.values.ravel(), 
                                   X_test=df_selected_test.values.ravel())

100%|██████████| 32/32 [00:24<00:00,  1.28it/s]

1 – 3 models leaderboard

display(models)

2 – Best model

best_model = regr_mts.get_best_model()
display(best_model)

DeepMTS(kernel='tophat', n_hidden_features=0, n_layers=1,
        obj=RANSACRegressor(random_state=42), replications=250,
        show_progress=False, type_pi='scp2-kde')

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.