Design Time Series NARX Feedback Neural Networks
All the specific dynamic networks discussed so far have either been focused networks,
with the dynamics only at the input layer, or feedforward networks. The nonlinear
autoregressive network with exogenous inputs (NARX) is a recurrent dynamic network,
with feedback connections enclosing several layers of the network. The NARX model is
based on the linear ARX model, which is commonly used in time-series modeling.
The defining equation for the NARX model is
y(t) = f (y(t −1), y(t − 2), , y(t − ny ),u(t −1),u(t − 2), ,u(t − nu)) … …
where the next value of the dependent output signal y(t) is regressed on previous values
of the output signal and previous values of an independent (exogenous) input signal. You
can implement the NARX model by using a feedforward neural network to approximate
the function f. A diagram of the resulting network is shown below, where a two-layer
feedforward network is used for the approximation. This implementation also allows for a
vector ARX model, where the input and output can be multidimensional.
There are many applications for the NARX network. It can be used as a predictor, to
predict the next value of the input signal. It can also be used for nonlinear filtering, in
which the target output is a noise-free version of the input signal. The use of the NARX
network is shown in another important application, the modeling of nonlinear dynamic
systems.
Design Time Series NARX Feedback Neural Networks
3-23
Before showing the training of the NARX network, an important configuration that is
useful in training needs explanation. You can consider the output of the NARX network
to be an estimate of the output of some nonlinear dynamic system that you are trying
to model. The output is fed back to the input of the feedforward neural network as part
of the standard NARX architecture, as shown in the left figure below. Because the true
output is available during the training of the network, you could create a series-parallel
architecture (see [NaPa91]), in which the true output is used instead of feeding back the
estimated output, as shown in the right figure below. This has two advantages. The first
is that the input to the feedforward network is more accurate. The second is that the
resulting network has a purely feedforward architecture, and static backpropagation can
be used for training.
The following shows the use of the series-parallel architecture for training a NARX
network to model a dynamic system.
The example of the NARX network is the magnetic levitation system described beginning
in “Use the NARMA-L2 Controller Block” on page 4-18. The bottom graph in the
following figure shows the voltage applied to the electromagnet, and the top graph shows
the position of the permanent magnet. The data was collected at a sampling interval of
0.01 seconds to form two time series.
The goal is to develop a NARX model for this magnetic levitation system.
3 Dynamic Neural Networks
3-24
First, load the training data. Use tapped delay lines with two delays for both the input
and the output, so training begins with the third data point. There are two inputs to the
series-parallel network, the u(t) sequence and the y(t) sequence.
load magdata
y = con2seq(y);
u = con2seq(u);
Create the series-parallel NARX network using the function narxnet. Use 10 neurons in
the hidden layer and use trainlm for the training function, and then prepare the data
with preparets:
d1 = [1:2];
d2 = [1:2];
narx_net = narxnet(d1,d2,10);
narx_net.divideFcn = '';
narx_net.trainParam.min_grad = 1e-10;
[p,Pi,Ai,t] = preparets(narx_net,u,{},y);
(Notice that the y sequence is considered a feedback signal, which is an input that is
also an output (target). Later, when you close the loop, the appropriate output will be
connected to the appropriate input.) Now you are ready to train the network.
narx_net = train(narx_net,p,t,Pi);
Design Time Series NARX Feedback Neural Networks
3-25
You can now simulate the network and plot the resulting errors for the series-parallel
implementation.
yp = sim(narx_net,p,Pi);
e = cell2mat(yp)-cell2mat(t);
plot(e)
You can see that the errors are very small. However, because of the series-parallel
configuration, these are errors for only a one-step-ahead prediction. A more stringent test
would be to rearrange the network into the original parallel form (closed loop) and then
to perform an iterated prediction over many time steps. Now the parallel operation is
shown.
3 Dynamic Neural Networks
3-26
There is a toolbox function (closeloop) for converting NARX (and other) networks from
the series-parallel configuration (open loop), which is useful for training, to the parallel
configuration (closed loop), which is useful for multi-step-ahead prediction. The following
command illustrates how to convert the network that you just trained to parallel form:
narx_net_closed = closeloop(narx_net);
To see the differences between the two networks, you can use the view command:
view(narx_net)
view(narx_net_closed)
Design Time Series NARX Feedback Neural Networks
3-27
All of the training is done in open loop (also called series-parallel architecture), including
the validation and testing steps. The typical workflow is to fully create the network in
open loop, and only when it has been trained (which includes validation and testing
steps) is it transformed to closed loop for multistep-ahead prediction. Likewise, the R
values in the GUI are computed based on the open-loop training results.
You can now use the closed-loop (parallel) configuration to perform an iterated prediction
of 900 time steps. In this network you need to load the two initial inputs and the two
initial outputs as initial conditions. You can use the preparets function to prepare the
data. It will use the network structure to determine how to divide and shift the data
appropriately.
y1 = y(1700:2600);
u1 = u(1700:2600);
[p1,Pi1,Ai1,t1] = preparets(narx_net_closed,u1,{},y1);
yp1 = narx_net_closed(p1,Pi1,Ai1);
TS = size(t1,2);
plot(1:TS,cell2mat(t1),'b',1:TS,cell2mat(yp1),'r')
3 Dynamic Neural Networks
3-28
The figure illustrates the iterated prediction. The blue line is the actual position of the
magnet, and the red line is the position predicted by the NARX neural network. Even
though the network is predicting 900 time steps ahead, the prediction is very accurate.
In order for the parallel response (iterated prediction) to be accurate, it is important that
the network be trained so that the errors in the series-parallel configuration (one-stepahead
prediction) are very small.
You can also create a parallel (closed loop) NARX network, using the narxnet command
with the fourth input argument set to 'closed', and train that network directly.
Generally, the training takes longer, and the resulting performance is not as good as that
obtained with series-parallel training.
Design Time Series NARX Feedback Neural Networks
3-29
Each time a neural network is trained, can result in a different solution due to different
initial weight and bias values and different divisions of data into training, validation,
and test sets. As a result, different neural networks trained on the same problem can
give different outputs for the same input. To ensure that a neural network of good
accuracy has been found, retrain several times.
There are several other techniques for improving upon initial solutions if higher accuracy
is desired. For more information, see “Improve Neural Network Generalization and Avoid
Overfitting” on page 8-31.
Multiple External Variables
The maglev example showed how to model a time series with a single external input
value over time. But the NARX network will work for problems with multiple external
input elements and predict series with multiple elements. In these cases, the input and
target consist of row cell arrays representing time, but with each cell element being an Nby-
1 vector for the N elements of the input or target signal.
For example, here is a dataset which consists of 2-element external variables predicting a
1-element series.
[X,T] = ph_dataset;
The external inputs X are formatted as a row cell array of 2-element vectors, with each
vector representing acid and base solution flow. The targets represent the resulting pH of
the solution over time.
You can reformat your own multi-element series data from matrix form to neural
network time-series form with the function con2seq.
The process for training a network proceeds as it did above for the maglev problem.
net = narxnet(10);
[x,xi,ai,t] = preparets(net,X,{},T);
net = train(net,x,t,xi,ai);
y = net(x,xi,ai);
e = gsubtract(t,y);
To see examples of using NARX networks being applied in open-loop form, closedloop
form and open/closed-loop multistep prediction see “Multistep Neural Network
Prediction” on page 3-51.
All the specific dynamic networks discussed so far have either been focused networks,
with the dynamics only at the input layer, or feedforward networks. The nonlinear
autoregressive network with exogenous inputs (NARX) is a recurrent dynamic network,
with feedback connections enclosing several layers of the network. The NARX model is
based on the linear ARX model, which is commonly used in time-series modeling.
The defining equation for the NARX model is
y(t) = f (y(t −1), y(t − 2), , y(t − ny ),u(t −1),u(t − 2), ,u(t − nu)) … …
where the next value of the dependent output signal y(t) is regressed on previous values
of the output signal and previous values of an independent (exogenous) input signal. You
can implement the NARX model by using a feedforward neural network to approximate
the function f. A diagram of the resulting network is shown below, where a two-layer
feedforward network is used for the approximation. This implementation also allows for a
vector ARX model, where the input and output can be multidimensional.
There are many applications for the NARX network. It can be used as a predictor, to
predict the next value of the input signal. It can also be used for nonlinear filtering, in
which the target output is a noise-free version of the input signal. The use of the NARX
network is shown in another important application, the modeling of nonlinear dynamic
systems.
Design Time Series NARX Feedback Neural Networks
3-23
Before showing the training of the NARX network, an important configuration that is
useful in training needs explanation. You can consider the output of the NARX network
to be an estimate of the output of some nonlinear dynamic system that you are trying
to model. The output is fed back to the input of the feedforward neural network as part
of the standard NARX architecture, as shown in the left figure below. Because the true
output is available during the training of the network, you could create a series-parallel
architecture (see [NaPa91]), in which the true output is used instead of feeding back the
estimated output, as shown in the right figure below. This has two advantages. The first
is that the input to the feedforward network is more accurate. The second is that the
resulting network has a purely feedforward architecture, and static backpropagation can
be used for training.
The following shows the use of the series-parallel architecture for training a NARX
network to model a dynamic system.
The example of the NARX network is the magnetic levitation system described beginning
in “Use the NARMA-L2 Controller Block” on page 4-18. The bottom graph in the
following figure shows the voltage applied to the electromagnet, and the top graph shows
the position of the permanent magnet. The data was collected at a sampling interval of
0.01 seconds to form two time series.
The goal is to develop a NARX model for this magnetic levitation system.
3 Dynamic Neural Networks
3-24
First, load the training data. Use tapped delay lines with two delays for both the input
and the output, so training begins with the third data point. There are two inputs to the
series-parallel network, the u(t) sequence and the y(t) sequence.
load magdata
y = con2seq(y);
u = con2seq(u);
Create the series-parallel NARX network using the function narxnet. Use 10 neurons in
the hidden layer and use trainlm for the training function, and then prepare the data
with preparets:
d1 = [1:2];
d2 = [1:2];
narx_net = narxnet(d1,d2,10);
narx_net.divideFcn = '';
narx_net.trainParam.min_grad = 1e-10;
[p,Pi,Ai,t] = preparets(narx_net,u,{},y);
(Notice that the y sequence is considered a feedback signal, which is an input that is
also an output (target). Later, when you close the loop, the appropriate output will be
connected to the appropriate input.) Now you are ready to train the network.
narx_net = train(narx_net,p,t,Pi);
Design Time Series NARX Feedback Neural Networks
3-25
You can now simulate the network and plot the resulting errors for the series-parallel
implementation.
yp = sim(narx_net,p,Pi);
e = cell2mat(yp)-cell2mat(t);
plot(e)
You can see that the errors are very small. However, because of the series-parallel
configuration, these are errors for only a one-step-ahead prediction. A more stringent test
would be to rearrange the network into the original parallel form (closed loop) and then
to perform an iterated prediction over many time steps. Now the parallel operation is
shown.
3 Dynamic Neural Networks
3-26
There is a toolbox function (closeloop) for converting NARX (and other) networks from
the series-parallel configuration (open loop), which is useful for training, to the parallel
configuration (closed loop), which is useful for multi-step-ahead prediction. The following
command illustrates how to convert the network that you just trained to parallel form:
narx_net_closed = closeloop(narx_net);
To see the differences between the two networks, you can use the view command:
view(narx_net)
view(narx_net_closed)
Design Time Series NARX Feedback Neural Networks
3-27
All of the training is done in open loop (also called series-parallel architecture), including
the validation and testing steps. The typical workflow is to fully create the network in
open loop, and only when it has been trained (which includes validation and testing
steps) is it transformed to closed loop for multistep-ahead prediction. Likewise, the R
values in the GUI are computed based on the open-loop training results.
You can now use the closed-loop (parallel) configuration to perform an iterated prediction
of 900 time steps. In this network you need to load the two initial inputs and the two
initial outputs as initial conditions. You can use the preparets function to prepare the
data. It will use the network structure to determine how to divide and shift the data
appropriately.
y1 = y(1700:2600);
u1 = u(1700:2600);
[p1,Pi1,Ai1,t1] = preparets(narx_net_closed,u1,{},y1);
yp1 = narx_net_closed(p1,Pi1,Ai1);
TS = size(t1,2);
plot(1:TS,cell2mat(t1),'b',1:TS,cell2mat(yp1),'r')
3 Dynamic Neural Networks
3-28
The figure illustrates the iterated prediction. The blue line is the actual position of the
magnet, and the red line is the position predicted by the NARX neural network. Even
though the network is predicting 900 time steps ahead, the prediction is very accurate.
In order for the parallel response (iterated prediction) to be accurate, it is important that
the network be trained so that the errors in the series-parallel configuration (one-stepahead
prediction) are very small.
You can also create a parallel (closed loop) NARX network, using the narxnet command
with the fourth input argument set to 'closed', and train that network directly.
Generally, the training takes longer, and the resulting performance is not as good as that
obtained with series-parallel training.
Design Time Series NARX Feedback Neural Networks
3-29
Each time a neural network is trained, can result in a different solution due to different
initial weight and bias values and different divisions of data into training, validation,
and test sets. As a result, different neural networks trained on the same problem can
give different outputs for the same input. To ensure that a neural network of good
accuracy has been found, retrain several times.
There are several other techniques for improving upon initial solutions if higher accuracy
is desired. For more information, see “Improve Neural Network Generalization and Avoid
Overfitting” on page 8-31.
Multiple External Variables
The maglev example showed how to model a time series with a single external input
value over time. But the NARX network will work for problems with multiple external
input elements and predict series with multiple elements. In these cases, the input and
target consist of row cell arrays representing time, but with each cell element being an Nby-
1 vector for the N elements of the input or target signal.
For example, here is a dataset which consists of 2-element external variables predicting a
1-element series.
[X,T] = ph_dataset;
The external inputs X are formatted as a row cell array of 2-element vectors, with each
vector representing acid and base solution flow. The targets represent the resulting pH of
the solution over time.
You can reformat your own multi-element series data from matrix form to neural
network time-series form with the function con2seq.
The process for training a network proceeds as it did above for the maglev problem.
net = narxnet(10);
[x,xi,ai,t] = preparets(net,X,{},T);
net = train(net,x,t,xi,ai);
y = net(x,xi,ai);
e = gsubtract(t,y);
To see examples of using NARX networks being applied in open-loop form, closedloop
form and open/closed-loop multistep prediction see “Multistep Neural Network
Prediction” on page 3-51.
Комментариев нет:
Отправить комментарий