Gradient

$$ \nabla E(\mathbf{w}) = \frac{\partial E(\mathbf{w})}{\partial \mathbf{w}} = \begin{bmatrix} \frac{\partial E(\mathbf{w})}{\partial w_0} \ \frac{\partial E(\mathbf{w})}{\partial w_1} \end{bmatrix} $$

Polynomial regression

• polynomiální regrese je speciální případ lineární regrese, protože $\mathbf{x}$ jen expanduje na požadovaný řád $k$, ale je to lineární kombinace $\hat{\mathbf{x}} \mathbf{w}$

Sum of squares

• vychází z toho jaké bereme rozložení chyby, pak najít maximum log likelihood, zderivovat a položit 0

Linear logistic regression

• $\sigma(a)$ = logistic sigmoid function
• $t_n$ = jestli je správná třída (1 nebo 0)
• $P(\mathbf{t}|\mathbf{X}) = \Pi_n \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right)^{t_n}\left( 1 - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right)^{(1-t_n)}$
• cross-entropy (cross-entropie): $E(\mathbf{w}) = - \ln P(\mathbf{t}|\mathbf{X})$

$$ E(w) = \sum_n \left( t_n \ln \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) + (1-t_n) \ln \left( 1 \red{-} \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) \right) \ \sigma’(a) = \sigma(a) \left(1 - \sigma(a) \right) = \sigma(a) \sigma(-a) \ {\partial E(w) \over \partial w} = - \sum_n \left( { t_n \over \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) } \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \left( 1 - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) x_n \red{-} { (1 - t_n) \left( 1 - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \over 1 - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) } x_n \right) \ = -\sum_n \left( t_n \left( 1 - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) x_n - (1 - t_n) \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) x_n \right) \ = -\sum_n \left( t_n x_n - t_n x_n \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) - x_n \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) + t_n x_n \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) \ = -\sum_n \left( \left( t_n - \sigma \left( \hat{\mathbf{x}}^T_n \mathbf{w} \right) \right) x_n \right) $$

Gradient descent

• $w^{t+1} = w^t - \eta \nabla E(w)$

Summary

• TODO: