Backpropagation_decoupled backpropagation-CSDN博客

本文链接：https://blog.csdn.net/github_29374279/article/details/50776265

Backpropagation

@[深度学习, 向后传播算法]

Backpropagation

$w_jk^l$ 表示 $(l-1)^{th}$ 层的第 $k$ 个神经元和第 $l^{th}$ 层的第j个元素的连接

神经网络权重

enter image description here

$b_j^l$ 第 $l$ 层神经元的bias
$a_j^i$ 第 $l$ 层神经元的activation

a l j = σ (\sum k w l j k a l - 1 k + b l j), (1)

$\begin{eqnarray} a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right), \tag{1}\end{eqnarray}$

$w_{jk}^l$ : $j$ 的范围是第l层神经元的个数， $k$ 的范围是第 $(l-1)$ 层神经元的个数，这个表示方便将公式表示为矩阵的形式

a l = σ (w l a l - 1 + b l) . (2)

$\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l). \tag{2}\end{eqnarray}$
这里 $a^{l-1}$ 是第 $l$ 层的激活神经元

代价函数相关的两个假设

backpropagation的目标就是计算代价函数对 $w$ 和 $b$ 的偏导

二次代价函数的形式：

C = 1 2 n \sum x ∥ y (x) - a L (x) ∥ 2, (3)

$\begin{eqnarray} C = \frac{1}{2n} \sum_x \|y(x)-a^L(x)\|^2, \tag{3}\end{eqnarray}$

假设1：代价函数能够被写成 $C = \frac {1}{n}\sum_x C_x$ ,需要这个假设的原因是backpropagation实际上需要我们计算的是对单个训练样本的偏导（ $\frac {\partial C_x} {\partial w}$ 和 $\frac {\partial C_x} {\partial b}$ ）
假设2：代价函数能够被写成神经网络输出的函数

例如：二次代价函数能够写成：

C = 1 2 ∥ y - a L ∥ 2 = 1 2 \sum j (y j - a L j) 2, (4)

$\begin{eqnarray} C = \frac{1}{2} \|y-a^L\|^2 = \frac{1}{2} \sum_j (y_j-a^L_j)^2, \tag{4}\end{eqnarray}$

Hadamard product

[12] ⊙ [34] = [1 * 3 2 * 4] = [38] . (5)

$\begin{eqnarray} \left[\begin{array}{c} 1 \\ 2 \end{array}\right] \odot \left[\begin{array}{c} 3 \\ 4\end{array} \right] = \left[ \begin{array}{c} 1 * 3 \\ 2 * 4 \end{array} \right] = \left[ \begin{array}{c} 3 \\ 8 \end{array} \right]. \tag{5}\end{eqnarray}$

Backpropagation的四个基本等式

backpropagation是为了理解在神经网络中改变weights和biases是怎样改变代价函数，最终，意味着计算偏导 $\frac {\partial C} {\partial w_{jk}^l}$ 和 $\frac {\partial C} {\partial b_j^l}$

为了计算偏导，我们首先计算中间量， $\delta_j^l$ ,表示 $l^{th}$ 层第 $j$ 个神经元的error

z l j = (\sum k w l j k a l - 1 k + b l j), (6)

$\begin{eqnarray} z^{l}_j = \left (\sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right) , \tag{6}\end{eqnarray}$

$δ l j \equiv \partial C \partial z l j . (7)$ $\begin{eqnarray} \delta^l_j \equiv \frac{\partial C}{\partial z^l_j}. \tag{7}\end{eqnarray}$

backpropagation给了一个对每层计算 $\delta^l$ 的方法

误差在输出层的等式

$δ L j = \partial C \partial a L j σ' (z L j) . (BP1)$ $\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j). \tag{BP1}\end{eqnarray}$
* $\partial C / {\partial a_j^L}$ 衡量了以第 $j^{th}$ 个激活元为函数的变化速率
* $\sigma'(z^L_j)$ 衡量了sigmoid函数对 $z_j^L$ 的变化速率
* $\partial C / {\partial a_j^L}$ 的精确形式取决于代价函数的选择，例如针对二次代价函数而言
* 当 $\sigma(z^L_j)$ 趋近于0或1的时候， $\sigma'(z^L_j) \approx 0$ ， $\delta^L_j$ 也会变的很小，可以说输出神经元已经饱和，weight开始停止学习或学习的很慢

$C= \frac {1} {2} \sum_j (y_j - a_j)^2$
$\frac {\partial C }{ \partial a_j^L} = (a_j - y_j)$

$δ L = \nabla a C ⊙ σ' (z L) . (BP1a)$ $\begin{eqnarray} \delta^L = \nabla_a C \odot \sigma'(z^L). \tag{BP1a}\end{eqnarray}$
二次代价函数的 $\delta^L$
$δ L = (a L - y) ⊙ σ' (z L)$ $\delta^L = (a^L - y) \odot \sigma'(z^L)$

在式子中都有较好的向量形式，因此容易利用Numpy等库进行计算

下一层误差等式, $\delta^{l+1}$

$δ l = ((w l + 1) T δ l + 1) ⊙ σ' (z l), (BP2)$ $\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l), \tag{BP2}\end{eqnarray}$

总结而言：
* 当输出神经元的状态是low-activation或者sturated时，weight将会缓慢的学习
* 这四个公式对任何形式的激活函数都有用

An equation for the rate of change of the cost with respect to any bias in the network:

$\partial C \partial b l j = δ l j (BP3)$ $\frac {\partial C} {\partial b^l_j} = \delta ^l _j\tag {BP3}$

$\partial C \partial b = δ, (BP3a)$ $\begin{eqnarray} \frac{\partial C}{\partial b} = \delta, \tag{BP3a}\end{eqnarray}$
An equation for the rate of change of the cost with respect to any weight in the network
$\partial C \partial w l j k = a l - 1 k δ l j . (BP4)$ $\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j. \tag{BP4}\end{eqnarray}$

$\partial C \partial w = a i n δ o u t, (BP4a)$ $\begin{eqnarray} \frac{\partial C}{\partial w} = a_{\rm in} \delta_{\rm out}, \tag{BP4a}\end{eqnarray}$

证明

（BP1）

$δ L j = \partial C \partial a L j σ' (z L j) . (BP1)$ $\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j). \tag{BP1}\end{eqnarray}$

$δ L j = \partial C \partial z L j$ $\delta^L_j = \frac{\partial C}{\partial z^L_j}$
链式法则
$δ L j = \partial C \partial a L j \partial a L j \partial z L j$ $\delta^L_j = \frac {\partial C}{\partial a^L_j} \frac {\partial a^L_j} {\partial z^L_j} \\$

(BP2)

$δ l = ((w l + 1) T δ l + 1) ⊙ σ' (z l), (BP2)$ $\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l), \tag{BP2}\end{eqnarray}$

$δ l j = \partial C \partial z l j = \sum k \partial C \partial z l + 1 k \partial z l + 1 k \partial z l j = \sum k \partial z l + 1 k \partial z l j δ l + 1 k, (链式法则) (a) (b)$ $\begin{align} \delta^l_j &= \frac{\partial C}{\partial z^l_j} \tag{链式法则} \\ &= \sum_k \frac{\partial C}{\partial z^{l+1}_k} \frac{\partial z^{l+1}_k}{\partial z^l_j} \tag{a}\\ &= \sum_k \frac{\partial z^{l+1}_k}{\partial z^l_j} \delta^{l+1}_k, \tag{b}\end{align}$
$z l + 1 k = \sum j w l + 1 k j a l j + b l + 1 k = \sum j w l + 1 k j σ (z l j) + b l + 1 k$ $z^{l+1}_k = \sum_j w^{l+1}_{kj} a^l_j +b^{l+1}_k = \sum_j w^{l+1}_{kj} \sigma(z^l_j) +b^{l+1}_k$

(BP3)

$\partial C \partial b l j = δ l j (BP3)$ $\frac {\partial C} {\partial b^l_j} = \delta ^l _j\tag {BP3}$

$\partial C \partial b l j = \partial C \partial z l j \partial z l j \partial b l j$ $\begin{align}\frac {\partial C} {\partial b^l_j} = \frac {\partial C} {\partial z^l_j} \frac {\partial z^l_j}{\partial b^l_j} \end{align}$
$\partial z l j \partial b l j = 1$ $\frac {\partial z^l_j}{\partial b^l_j} = 1$

(BP4)

$\partial C \partial w l j k = a l - 1 k δ l j . (BP4)$ $\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j. \tag{BP4}\end{eqnarray}$

$\partial C \partial w l j k = \partial C \partial z l j \partial z l j \partial w l j k (链式法则)$ $\begin{align}\frac {\partial C} {\partial w^l_{jk}} = \frac {\partial C} {\partial z^l_j} \frac {\partial z^l_j}{\partial w^l_{jk}} \tag{链式法则}\end{align}$
$z l j = \sum k w l j k a l - 1 k + b l j$ $z^l_j = \sum_k {w^l_{jk} a^{l-1}_k} +b_j^l$

backpropagation算法

backpropagation equations 提供了一个计算代价函数梯度的方式

输入 x :设置相应的 activation $a^1$ 为输入层
Feedforward： 对每一层 $l = 2,3,...,L$ 计算 $z^l = w_la^{l-1} + b^l$ 和 $a^l = \sigma (z^l)$
Output error (输出层误差) ： 计算向量 $\delta^L = \nabla_a C \odot \sigma'(z^L)$
Backpropagate the error : 对每一层 $l = L-1,L-2,...,2$ 计算 $\delta^l = ((w^{l+1})^T\delta^{l+1}) \odot \sigma'(z^l)$
输出 : 计算代价函数的梯度，通过 $\frac {\partial C}{\partial w^l_{jk}} = a_k^{l-1} \delta^l_j 和 \frac {\partial C}{\partial b^l_j} = \delta_l^j$

mini-batch:(随机梯度下降结合backpropagation)
1. 输入一组训练样本
2. 对每个训练样本：设置相应的输入激活元 $a^{x,1}$
- Feedforward： 对每一层 $l = 2,3,...,L$ 计算 $z^{x,l} = w_la^{x,l-1} + b^l$ 和 $a^{x,l} = \sigma (z^{x,l})$
- Output error (输出层误差) ： 计算向量 $\delta^{x,L} = \nabla_a C_x \odot \sigma'(z^{x,L})$
- Backpropagate the error : 对每一层 $l = L-1,L-2,...,2$ 计算 $\delta^{x,l} = ((w^{l+1})^T\delta^{x,l+1}) \odot \sigma'(z^{x,l})$
3. 梯度下降： 对每一层 $l = L,L-1,...,2$ ,更新权重weights，根据规则
$w l \to w l - η m \sum x δ x, l (a x, l - 1) T$ $w^l \rightarrow w^l - \frac {\eta}{m} \sum_x \delta^{x,l}(a^{x,l-1})^T$ $b l \to b l - η m \sum x δ x, l$ $b^l \rightarrow b^l - \frac {\eta}{m} \sum_x \delta^{x,l}$