This post implements the standard matrix based estimation of multiple linear regression model using Tensorflow. With this example, we can learn some basic vector or matrix operations in Tensorflow and also Python.

Linear Regression using Tensorflow

To study some basic vector or matrix operations in Tensorflow which is not familiar to us, we take the linear regression model as an example, which is familiar to us.

Linear Regression model

Multiple linear regression model has the following expression. \((t=1,2,…,n)\)

\[\begin{align} Y_t&=\beta_0+\beta_1 X_{1t}+\beta_2 X_{2t}+⋯+\beta_{p-1} X_{p-1,t}+\epsilon_t \end{align}\]
Here \(Y_t\) is the dependent variable and \( X_t = (1,X_{1t},X_{2t},…,X_{p-1,t})\) is a set of independent variables. \(\beta=(\beta_0,\beta_1,\beta_2,…,\beta_{p-1} )^{‘}\) is a vector of parameters and \(\epsilon_t\) is a vector or stochastic disturbances.

It is worth noting that the number of parameters is \(p\) and the number of variables is \(p-1\)

Stochastic error term \(\epsilon_t\) is assumed in the following way.

1. \(E[\epsilon_t ]=0\), \(\text{var}[\epsilon_t ]=\sigma_\epsilon^2\)
2. For all \(t_1\) and \(t_2\), \((t_1≠t_2)\), \(cov[\epsilon_{t1}, \epsilon_{t2}]=E[\epsilon_{1t} \epsilon_{2t} ]=0\)
3. \(\epsilon_t\) follows a normal distribution

Least Squares Estimator

To estimate the regression coefficients \(\beta\), we use least squares which minimize the sum of squared residuals. In a matrix notation, the least squares estimator is calculated in the following way. \[\begin{align} S(\beta)&=\sum_{t=1}^{n}\epsilon_t^2=\epsilon^{‘} \epsilon \\ &=(Y-X\beta)^{‘} (Y-X\beta) \\ &=(Y^{‘}-\beta^{‘} X^{‘} )(Y-X\beta) \\ &=Y^{‘} Y-Y^{‘} X\beta-\beta^{‘} X^{‘} Y+\beta^{‘} X^{‘} X\beta \\ &=Y^{‘} Y-2\beta^{‘} X^{‘} Y+\beta^{‘} X^{‘} X\beta \\ \end{align}\]
Differentiating \(S(\beta)\) with respect to \(\beta\) and set to zero results in the following the normal equation. \[ \frac{\partial S(\beta)}{\partial \beta} = -2X^{‘} Y+2X^{‘} X \beta=0 \] Hence the least squares estimator of \(\beta\) is \[ \hat{\beta}=(X^{‘} X)^{-1} X^{‘} Y \]

Standard Errors

\(\hat{\beta}\) follows the following distribution
\[\begin{align} \hat{\beta} \sim N \left(\beta, \sigma_{\epsilon}^2 (X^{‘} X)^{-1} \right) \end{align}\] \[\begin{align} \text{cov}[\hat{\beta}] &= \text{cov}[ (X^{‘} X)^{-1} X^{‘} Y ] \\ &= (X^{‘} X)^{-1} X^{‘} \text{cov}[Y] X(X^{‘} X)^{-1} \\ &= (X^{‘} X)^{-1} X^{‘} \sigma_{\epsilon}^2 I X(X^{‘} X)^{-1} \\ &= \sigma_{\epsilon}^2 (X^{‘} X)^{-1} X^{‘} I X(X^{‘} X)^{-1} \\ &= \sigma_{\epsilon}^2 (X^{‘} X)^{-1} \end{align}\] Since \(\sigma_{\epsilon}^2\) is the population parameter we don’t know, \(\sigma_{\epsilon}^2\) is replaced with \(s^2\), the sample estimate of \(\sigma^2\). \[\begin{align} \widehat{\text{cov}[\hat{\beta}]} &= s^2 (X^{‘} X)^{-1} \\ \\ s^2 &= \frac{e^{‘}e}{n-\color{red}{p}} \end{align}\] To estimate \(\sigma^2\), we need to estimate \(p\) parameters (1 intercept + (p-1) coefficients),in other words, \(p\) degree of freedom is lost.

Python code using Tensorflow

The purpose of this post is to learn some basic vector or matrix operations (matrix multiplication, transpose, inverse, etc.) in Tensorflow. As an example, we use the diabetes data from sklearn package which has 10 explanatory variables and 1 response variable.

To check the estimation accuracy, regression outputs from sklearn, statsmodels are also considered. pd.DataFrame() from pandas package is used for make a table from np.array or Tenslorflow objects.

We can easily find that all results are the same as expected. In particular, two approaches using np.array and Tensorflow has the nearly same structure.

Concluding Remarks

This post dealt with how to use some basic vector or matrix operations of Tensorflow. It is similar to that of np.arrary but there are some subtle differences about manipulating array objects. Based on these, if we know remaining subjects such as for-loop, function definition, and optimization, it is expected that we can implement state space model using Kalman filter and estimate its parameters. \(\blacksquare\)