Page 38 - Artificial Intelligence for the Internet of Everything

Uncertainty Quantification in Internet of Battlefield Things 25

Algorithm 2.1 Generic SGD

The basic SGD algorithm sets n k ¼ 1, α k ¼ ð1=kÞ and computes the
stochastic direction as:

k
w k +1 ¼ w k α k rFðw k ,ξ Þ
This is essentially the full-gradient method only evaluated for one sample
point. Using the standard full-gradient method would require n gradient
evaluations every iteration, but the basic SGD algorithm only requires the
evaluation of one. We briefly discuss an implementation for logistic
regression.

2.3.3 Example: Logistic Regression
To make this algorithm more concrete, consider the case of binary logistic
regression. When the amount of data is manageable, a standard way
(Hastie, Tibshirani, & Friedman, 2009) to find the optimal parameters is
to apply Newton’s method (since there is no closed-form solution) to the
log-likelihood for the model:
n
X T T
y i w x i logð1+ e w x i
‘ðw;x,yÞ¼ Þ
i¼1
However, when n is large, this quickly becomes unfeasible. Instead, we
could use a simple SGD implementation. In this case we set the stochastic