The back propagation method is the example of the wide class of training methods based on the information covered in the gradient of error function. The independent variables in this minimization are weights of neural network and the considered error to be minimized is the root mean square one.

Let us consider the training set composed of LL ordered pairs, of the following form: { ( x ( 1 ) , d ( 1 ) ),( x ( 2 ) , d ( 2 ) ),...,( x ( L ) , d ( L ) ) } { ( x ( 1 ) , d ( 1 ) ),( x ( 2 ) , d ( 2 ) ),...,( x ( L ) , d ( L ) ) } Furthermore, let us define the total error EE generated on outputs of neural network after presenting the entire training set, as: E= ∑ l=1 L E ( l ) E= ∑ l=1 L E ( l ) where: E ( l ) = ∑ m=1 M E m ( l ) = 1 2 ∑ m=1 M ( d m ( l ) − y m ( l ) ) 2 E ( l ) = ∑ m=1 M E m ( l ) = 1 2 ∑ m=1 M ( d m ( l ) − y m ( l ) ) 2 As was already told, the independent variables in the minimization of error EE are weights w ij w ij Subscript w ij Since even for the relatively small networks the number of weigths is big, in real applications, the training of the neural network is the minimization of the scalar field over the vector space with hundreds or (more often) thousands dimensions. One of the minizmiazation techniques for such problem is the steapest descent method