当前位置：首页 > news >正文

反向传播notes

news 2025/7/16 10:30:51

~~谁敢相信我马上要秋招了且有两段算法实习到现在才算真的理解透彻反向传播，这世界就是个巨大的ctbz…~~

首先理解链式法则，假设有两个可微的函数 $f (x)$ 和 $g (x)$ ， $h (x) = f (g (x))$ ，记 $u = g (x)$ , $f (u) = h (x)$ ，则 $∂h(x)∂x=∂f(u)∂u∂g(x)∂x\frac{\partial h(x)}{\partial x}=\frac{\partial f(u)}{\partial u}\frac{\partial g(x)}{\partial x}$ .
复合函数的导数可以逐步分解求导再相乘，而神经网络里的基本单元就是线性层+激活函数，假设输入x有两层网络： $z1=W1x+b1,a1=σ(z1)z^1=W^1x+b^1,a^1=\sigma(z^1)$
$z2=W2a1+b2,a2=σ(z2)z^2=W^2a^1+b^2,a^2=\sigma(z^2)$
最终输出 $y^{pred}=a^2$ ，定义损失函数为MSE， $L=12(ypred−y)2L=\frac12(y^{pred}-y)^2$ ， $y$ 是label，有 $∂L∂ypred=ypred−y\frac{\partial L}{\partial y^{pred}}=y^{pred}-y$ ，
初始随机化参数，梯度下降更新参数值，有 $W2=W2−α∂L∂W2W^2=W^2-\alpha\frac{\partial L}{\partial W^2}$ ， $∂L∂W2=∂L∂ypred∂ypred∂z2∂z2∂W2=(ypred−y)σ′(z2)a1\frac{\partial L}{\partial W^2}=\frac{\partial L}{\partial y^{pred}}\frac{\partial y^{pred}}{\partial z^2}\frac{\partial z^2}{\partial W^2}=(y^{pred}-y)\sigma'(z^2)a^1$ ，依次更新反向传播。
$∂L∂b2=∂L∂ypred∂ypred∂z2∂z2∂b2=(ypred−y)σ′(z2)\frac{\partial L}{\partial b^2}=\frac{\partial L}{\partial y^{pred}}\frac{\partial y^{pred}}{\partial z^2}\frac{\partial z^2}{\partial b^2}=(y^{pred}-y)\sigma'(z^2)$
$∂L∂b1=∂L∂ypred∂ypred∂z2∂z2∂a1∂a1∂z1∂z1∂b1=(ypred−y)σ′(z2)W2σ′(z1)\frac{\partial L}{\partial b^1}=\frac{\partial L}{\partial y^{pred}}\frac{\partial y^{pred}}{\partial z^2}\frac{\partial z^2}{\partial a^1}\frac{\partial a^1}{\partial z^1}\frac{\partial z^1}{\partial b^1}=(y^{pred}-y)\sigma'(z^2)W^2\sigma'(z^1)$
$∂L∂W1=∂L∂ypred∂ypred∂z2∂z2∂a1∂a1∂z1∂z1∂W1=(ypred−y)σ′(z2)W2σ′(z1)x\frac{\partial L}{\partial W^1}=\frac{\partial L}{\partial y^{pred}}\frac{\partial y^{pred}}{\partial z^2}\frac{\partial z^2}{\partial a^1}\frac{\partial a^1}{\partial z^1}\frac{\partial z^1}{\partial W^1}=(y^{pred}-y)\sigma'(z^2)W^2\sigma'(z^1)x$

todo
函数求导的转置变换

实现代码

import numpy as npclass NeuralNetwork:def __init__(self, input_size, hidden_size, output_size):self.input_size = input_sizeself.hidden_size = hidden_sizeself.output_size = output_size# Initialize weightsself.weights_input_hidden = np.random.randn(self.input_size, self.hidden_size)self.weights_hidden_output = np.random.randn(self.hidden_size, self.output_size)# Initialize the biasesself.bias_hidden = np.zeros((1, self.hidden_size))self.bias_output = np.zeros((1, self.output_size))def sigmoid(self, x):return 1 / (1 + np.exp(-x))def sigmoid_derivative(self, x):return x * (1 - x)def feedforward(self, X):# Input to hiddenself.hidden_activation = np.dot(X, self.weights_input_hidden) + self.bias_hiddenself.hidden_output = self.sigmoid(self.hidden_activation)# Hidden to outputself.output_activation = np.dot(self.hidden_output, self.weights_hidden_output) + self.bias_outputself.predicted_output = self.sigmoid(self.output_activation)return self.predicted_outputdef backward(self, X, y, learning_rate):# Compute the output layer erroroutput_error = y - self.predicted_outputoutput_delta = output_error * self.sigmoid_derivative(self.predicted_output)# Compute the hidden layer errorhidden_error = np.dot(output_delta, self.weights_hidden_output.T)hidden_delta = hidden_error * self.sigmoid_derivative(self.hidden_output)# Update weights and biasesself.weights_hidden_output += np.dot(self.hidden_output.T, output_delta) * learning_rateself.bias_output += np.sum(output_delta, axis=0, keepdims=True) * learning_rateself.weights_input_hidden += np.dot(X.T, hidden_delta) * learning_rateself.bias_hidden += np.sum(hidden_delta, axis=0, keepdims=True) * learning_ratedef train(self, X, y, epochs, learning_rate):for epoch in range(epochs):output = self.feedforward(X)self.backward(X, y, learning_rate)if epoch % 4000 == 0:loss = np.mean(np.square(y - output))print("Epoch{epoch}, Loss:{loss}")X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])nn = NeuralNetwork(input_size=2, hidden_size=4, output_size=1)
nn.train(X, y, epochs=10000, learning_rate=0.1)# Test the trained model
output = nn.feedforward(X)
print(output)