Back propagation mathematics
2.0
2003/04/22
2003/04/22
Krzysztof
A.
Cyran
chrisc1@rice.edu
Krzysztof
A.
Cyran
chrisc1@rice.edu
back propagation
first order rule
gradient based learning
gradient optimization
learning methods
neural networks
steepest descent method
This module explains mathematical background of the most popular learning algorithm, called back propagation. It is widely used in multilayer perceptrons training but it's usage is not limited to this particular type of neural network.
Error definition
The back propagation method is the example of the wide class of training methods based on the information covered in the gradient of error function. The independent variables in this minimization are weights of neural network and the considered error to be minimized is the root mean square one.
Let us consider the training set composed of L ordered pairs, of the following form:
{
(
x
(
1
)
,
d
(
1
)
),(
x
(
2
)
,
d
(
2
)
),...,(
x
(
L
)
,
d
(
L
)
)
}
Furthermore, let us define the total error E generated on outputs of neural network after presenting the entire training set, as:
E=
∑
l=1
L
E
(
l
)
where:
E
(
l
)
=
∑
m=1
M
E
m
(
l
)
=
1
2
∑
m=1
M
(
d
m
(
l
)
−
y
m
(
l
)
)
2
As was already told, the independent variables in the minimization of error E are weights
w
ij
Subscript
w
ij
Since even for the relatively small networks the number of weigths is big, in real applications, the training of the neural network is the minimization of the scalar field over the vector space with hundreds or (more often) thousands dimensions. One of the minizmiazation techniques for such problem is the steapest descent method
x
0
1
x
2
n 1
2
2
12
26390 n 1103
4
n
9801
396
4
n
n
4
-1
∑
n
=
1
∞
2
2
(
26390
n
+
1103
)
(
4
n
)
!
9801
396
4
n
n
!
4
∑
n
=
1
∞
2
2
(
26390
n
+
1103
)
(
4
n
)
!
9801
396
4
n
n
!
4
n
1
2
2
12
26390
n
1103
4
n
9801
396
4
n
n
4
-1
n
1
2
2
12
26390
n
1103
4
n
9801
396
4
n
n
4
-1