Last year, in a previous post,
I’ve used Python package the-teller to explain an xgboost model’s predictions.
After reading today’s post, you’ll be able to use that same package, the-teller, to explain
predictions of a Keras neural network trained on tabular data.
It’s worth mentioning that I’m not using autokeras here to obtain a perfect model (try a Random Forest in the
same setting as the one described below 😉 ). Rather,
I’m using it to obtain a relatively good Keras model without much manual tuning.
A wrapper that allows to use Keras models as scikit-learn models (fit, predict, model selection, pipelines, etc.):
pip install scikeras
Scientific computing/data wrangling in Python:
pip install scipy==1.4.1
pip install numpy
pip install pandas
Tensorflow (Keras is built on top of this package)
A tool for explaining predictions of Statistical/Machine Learning models on tabular data:
pip install the-teller
After the installation, we import these packages into Python:
import numpy as np
import pandas as pd
import autokeras as ak
import teller as tr
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from scikeras.wrappers import KerasRegressor
The dataset used for this demo, the California housing dataset (imported by sklearn’s fetch_california_housing), has the following description:
- __Response__ / __target__ to be explained: median __house value for California districts__, in hundreds of thousands of dollars ($100,000)
- __MedInc__: median income in block group
- __HouseAge__: median house age in block group
- __AveRooms__: average number of rooms per household
- __AveBedrms__: average number of bedrooms per household
- __Population__: block group population
- __AveOccup__: average number of household members
- __Latitude__: block group latitude
- __Longitude__: block group longitude
# Input data from california housing
X, y = fetch_california_housing(return_X_y=True, as_frame=False)
# Columns names
X_names = fetch_california_housing(return_X_y=True, as_frame=True)[0].columns
# Split data into a training test and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2, random_state=13)
# Initialize autokeras's structured data regressor.
reg = ak.StructuredDataRegressor(
overwrite=True, max_trials=100, loss="mean_squared_error",
) # It tries 100 different models. Try a lower `max_trials` for a faster result.
# Feed the structured data regressor with training data, and train on 20 epochs.
reg.fit(x=X_train, y=y_train, epochs=20)
# Predict with the _best_ model found by autokeras.
predicted_y = reg.predict(X_test)
# Out-of-sample error (Root Mean Squared Error)
print(mean_squared_error(y_true=y_test, y_pred=predicted_y.flatten(), squared=False))
The model found by autokeras, reg, is exported to a Keras model, whose summary
of layers and parameters can be printed:
All the ingredients for feeding the-teller’s Explainer are now gathered:
# creating the explainer
explainer = tr.Explainer(obj=reg2)
# fitting the explainer to unseen data
explainer.fit(X_test, y_test, X_names=X_names, method="avg")
explainer.plot(what="average_effects")
According to this Keras neural network, all else held equal, the average number of bedrooms
and the median income in block are the most important drivers for an increase in
housing value. Surprisingly too (or not?), when the housing age in block group is increased by a little \(\epsilon\),
the housing value does not change on average – all else held equal.
explainer.summary()
Heterogeneity of marginal effects:
mean std median min max
AveBedrms 1.461185 1.491522 1.241837 -2.834498 7.180917
MedInc 0.412377 0.251765 0.394124 -0.215032 1.737655
Population 0.000037 0.000209 0.000026 -0.000666 0.001251
HouseAge 0.000000 0.000000 0.000000 0.000000 0.000000
Longitude 0.000000 0.000000 -0.000000 -0.000000 -0.000000
Latitude -0.042189 0.164907 -0.039731 -0.743647 0.643677
AveRooms -0.085101 0.228191 -0.056002 -0.938256 0.783281
AveOccup -0.567745 0.487438 -0.422143 -2.381372 0.105577
