Available online at www.sciencedirect.

com
ScienceDirect

Mathematics and Computers in Simulation 202 (2022) 331–342

www.elsevier.com/locate/matcom

Original articles

Solving flows of dynamical systems by deep neural networks and a

novel deep learning algorithm✩
Guangyuan Liao, Limin Zhang ∗
Key Laboratory of Intelligent Analysis and Decision on Complex Systems, School of Science, Chongqing University of Posts and
Telecommunications, Chongqing 400065, China
Received 24 November 2021; received in revised form 12 May 2022; accepted 4 June 2022
Available online 14 June 2022

Abstract
Machine learning becomes popular and is used for a wide range of problems in various areas of applied sciences. In
dynamical systems, machine learning methods are applied to solve differential equations. In this paper, we develop an artificial
network to solve systems of ordinary differential equations. For the network, we use a multilayer perceptron networks, which is
a fully connected feedforward network to predict the flow of a specific system. In order to improve the long time estimation of
the method, we introduced an upper bound control strategy. To deal with stiff ODE systems(such as slow-fast coupled system),
a new algorithm, Finite Neural Element method, is introduced. By numerical simulation, the novel algorithm is proved to have
better efficiency and accuracy than direct machine learning method.
© 2022 International Association for Mathematics and Computers in Simulation (IMACS). Published by Elsevier B.V. All rights
reserved.
Keywords: Multilayer perceptron; Neural network; Differential equations; Finite neural element

1. Introduction
Finding solutions of differential equations are one of the most fundamental areas in analysis of dynamical
systems. And it is well established by various numerical methods, such as finite difference [6], finite element [12]
and so on. Most of them require numerical integrations, the results are discrete and non differentiable. These classical
methods are generally efficient for determining the state of a system for a sequence of times. However, when we
are only interested of a solution at a specific time, the computational work must still be done by determining all
the states at each time steps leading up to the state of interest. Additionally, the need of previous state solutions
also limits the parallelizability of these methods.
Another approach of solving differential equations is to use the artificial neural networks. The idea is first
proposed by Dissanayake and Phan-Thien [5], where they approximate the solution of the differential equations as
✩ The first author’s work was supported by the Scientific Research Starting Foundation of Chongqing University of Posts and
Telecommunications (Grant No. A2022-289). The corresponding author’s work was supported by the National Natural Science Foundation of
China (Grant No. 11901073), Science and technology research project of Chongqing Education Commission (Grant No. KJQN201900619)
and the Scientific Research Starting Foundation of Chongqing University of Posts and Telecommunications (Grant No. A2018-124).
∗ Corresponding author.
E-mail addresses: [email protected] (G. Liao), [email protected] (L. Zhang).

https://doi.org/10.1016/j.matcom.2022.06.004
0378-4754/© 2022 International Association for Mathematics and Computers in Simulation (IMACS). Published by Elsevier B.V. All rights
reserved.
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

a neural network, and they train the neural net to minimize the loss of the boundary conditions and the differential
equations. To minimize the loss function, taking gradient with respect to the parameters, Lagaris et al. [13]
showed that for a feedforward neural network, the derivatives with respect to these parameters can be explicitly
computed and the calculation can be applied on parallel hardware. After training, an advantage of network based
methods is that any approximate solution in the training range can be calculated directly without knowing the
information of previous states. Such method is applied to both ODEs and PDEs, and has many software packages
in development [2,14]. In those method and software packages for ODEs, a single solution is learned. Once we
change initial condition and parameters, retraining the network is required. There are also some other methods
in progress. For instance, some researches [3,9,15] approximated time series even chaotic time series by using
reservoir computing networks. Sirignano and Spiliopoulos [19] worked on combining Galerkin method with the
neural network solver. Ngom and Marin [16] used a Fourier neural network as function approximators. Cosmin,
et al. [1] presented an adaptive collocation strategy for solving PDEs. Guo et al. [11] proposed a Deep Collocation
Method(DCM) for thin plate bending problem, such method solved the difficulty of the C1 continuity requirement
in traditional mesh-based methods. Other applications of the DCM are also made on heterogeneous porous media,
and the groundwater flow problems [10]. Vien et al. and Samaniego et al. [17,18] proposed a deep energy method.
In this paper, we extend the method of [13] for learning the flow of a system of ODEs instead of learning a
single solution. By doing that, the trained network has a greater area of application. Then we propose a bounded
control strategy to make the training process converge faster, and the global error is smaller than those using neural
network directly. As far as we know, there are few works using control strategy to the trained network. Moreover,
we propose a new algorithm, Finite Neural Element (FNE) method, which is very effective in handling coupled
systems with multiple time scale (such as slow–fast neural networks), while the classical neural network method
fails.

2. Methods
In this section, we review the basic ideas of dynamical systems and the definition of the flow. Then we build
the deep neural network for computing the flow and derive the loss function analytically. Instead of applying such
network directly to solve dynamical system [13], a control method and a computing algorithm are introduced for
improving the efficiency of the network method.

2.1. Set up problem and neural network

Consider the following general system of ODEs

dx
= f (t, x), (1)
dt
where x ∈ Rn is the state variable, and t is the time variable. If the flow for (1) is defined as ϕ(t, x), then it satisfies
the following initial value problem
dϕ(t, x0 )
= f (t, ϕ(t, x0 )), ϕ(0, x) = x0 . (2)
dt
Our goal is to build a deep neural network N (t, x, p), such that
ϕ(t, x) = x + t N (t, x, p). (3)
In order to train the parameters p of the network N (t, x, p) such that (3) satisfies the differential equations (1),
we define a loss function as
dϕ(t, x)
J ( p) = ∥ − f (t, ϕ(t, x))∥2
dt (4)
∂ N (t, x, p)
= ∥N (t, x, p) + t − f (t, x + t N (t, x, p))∥2 .
∂t
In (4), the term ∂ N (t,x,
∂t
p)
is required, since the neural network N (t, x, p) is differentiable, we can calculate it by
chain rule. To do that, we first give a definition of the neural network in vector formation.
332
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 1. The schematic multilayer perceptron for solving 2D systems. The input layer with three neurons represents t, x and y. Two hidden
layers are shown, and the output layer has two neurons representing the state variables.

In Fig. 1, a schematic deep neural network for solving 2D systems with two hidden layers is presented. Notice
that, the input layer is filled with three neurons, so that the state variables and time variable are both included. We
define the number of neurons for each layer as N1 , N2 , . . . , Nn , the weight matrix and the bias from layer i to layer
i+1 as W (i) , B (i) , where i = 1, . . . , n − 1, W (i) is an Ni by Ni+1 matrix, and B (i) is an Ni+1 by 1 column vector.
We propagate the network and obtain

a (1) = X
z (i) = W (i)T a (i) + B (i) (5)
a (i+1)
= g(z ), i = 1, . . . , n − 1.
(i)

Notice that, X in the first equation of (5) is a combination of the time variable and the state variables, in other
words, X = (t, x)T , t ∈ R, x ∈ Rm , where m is the dimension of the system we aim to solve. g(x) is the activation
function, here we choose the sigmoid function g(x) = 1+e1−x . Then we obtain that N (t, x, p) = a (n) , therefore

∂ N (t, x, p) ∂a (n)
=
∂X ∂X
2
∏ ∂a (i) ∂z (i−1)
=
i=n
∂z (i−1) ∂a (i−1) (6)
2
∏
= g ′ (z (i−1) )W (i−1)T ,
i=n

which is a Jacobian matrix, and its first column is the partial derivative ∂ N (t,x,
∂t
p)
. Hence, we obtained the loss
function explicitly.
Now the neural network and the loss function are well defined, the problem is then to find a set of parameters p,
such that the loss function is optimized. To do that, we use the trust region reflective algorithm [4] and the gradient
of J ( p) is obtained by auto differentiation.
333
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

2.2. Upper bound control

It is necessary to validate if the method introduced above work correctly. Based on our analysis, Eq. (3) is the
approximation for the flow of (1), while the exact flow is
∫ t
ϕe (t, x) = x + f (s, x(s))ds. (7)
0
By the definition of the neural network, we have
N (t, x, p) = a (n) = g(z (i) ), (8)
and we know that |g(z (i) )| ≤ 1. Hence,
|N (t, x, p)| ≤ 1. (9)
Therefore,
|t N (t, x, p)| ≤ |t|. (10)
By (3), we have
|ϕ(t, x)| ≤ |x| + |t|. (11)
However, the exact flow is not guaranteed to be bounded by (11) if we do not know the upper bound of | f (t, x)|.
To avoid that, an upper bound control technique is introduced. Suppose
| f (t, x)| ≤ M, (t, x) ∈ I, (12)
where I is the parameter range we use for training, M is one upper bound of the vector field (for best approximation,
it is suggested to choose M close to sup| f (t, x)|). Define a new state variable
x
y= , (13)
M
such that
dy 1 dx f (t, M y)
= = . (14)
dt M dt M
For the exact flow of (14), we have
∫ t
f (s, y(s))
|ϕe (t, y)| = |y + ds|
0 M
M (15)
≤ |y| + |t|
M
≤ |y| + |t|,
which has a same bound as (11). We then define its neural network approximation as
ϕ̂(t, y) = y + t N̂ (t, y, p). (16)
Notice that, it is not hard to show
ϕ(t, x) = M ϕ̂(t, y). (17)
In the next section, an example is presented for the comparison of using upper bound control and not using
upper bound control.

2.3. Finite Neural Element algorithm

In practice, when the neural network method is applied to solve biological slow–fast oscillators, it did not work
well on a long time interval. For example, Flamant, et al. [8] use the neural network method to predict the solution
of the FitzHugh–Nagumo model for a long time interval, the training process might hit the nullclines and stay on
them. Which is because the points on nullclines also satisfy the loss function, and those points are easier for the
334
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

network to “learn”. To avoid it, they applied the curriculum learning. However, it requires more neural units and
more levels of neural layers, which leads a large memory cost and a lose of efficiency (the network they used has
4 dense layers of 128 neurons each, the training time is 34.8 h).
In this section, we introduce a Finite Neural Element (FNE) method, such method is in spirit of Finite Element
Method. In the Finite Element Method, the “element” represents that the domain which is divided into small pieces,
and the corner point of each element is known as a “node”. For each element, the interpolation functions are defined,
these interpolation functions are often known as basis functions. Similarly, in our FNE method, suppose we want
to solve (1) with an initial condition x0 and a time range [0, T ], instead of using (3) directly, we discretize the
whole time interval into finite elements of small intervals with distance t L . Let t1 , . . . , t N be the end point of each
interval, we define the neural element as
ϕi (t, x) = xi−1 + t Ni (t, x, pi ), (18)
where xi is the end point of the last prediction of the neural network and
xi = xi−1 + t L Ni (t L , xi−1 , pi ). (19)
Then we train each small network, and combine them together as our final trained network. The following
pseudocode provides more details for the FNE algorithm.

Algorithm 1: Finite Neural Element

FUNCTION [y, p] = FNE(T, x0 , odefun);
initialization for training parameters;
tspan = [t1 , ..., t N ];
x1 = x0 ;
i = 2;
while i ≤ N do
Train the small network to get parameters’ values: par s;
xi = xi−1 + t L N (t, xi−1 , par s);
i = i + 1;
SAVE p[i] ← par s, yi ← xi ;
end

Remark. Since the small network is restricted on a short interval [0, t L ], the long time evolution error is avoided,
and we can make the network as small as possible. For the FN model example, we found that the small network can
be made as just one hidden layer of one neuron, which greatly reduces the memory cost and increases efficiency.
Secondly, to use the FNE network for prediction, it is slight different than the deep neural network method. Instead
of storing a whole trained parameter set like the deep network, several small parameter sets are stored as a trained
set for the FNE method. It is possible to predict any part of trajectories between 0 and T with the saved parameters
of each network element. Additionally, we can also run the FNE method for a bunch of initial conditions, and use
the trained network to predict flows for an area of the phase space. In our FNE method, we split the whole time
interval into several small time interval, and we call each of them a neural element. The training parameters are
also separated for each small element, so they are trained independently. The global numerical error is bounded
by the maxima of all the local errors. So the convergence is also guaranteed. Numerical results are presented in
Section 3.3.

3. Numerical results
In this section, we solve several ODE examples with our method. The first example is an 1D ODE, where we
show the efficiency of upper bound control. The second example is a 2D ODE system which is known as simple
harmonic oscillators, where we show our method is applicable for prediction of flows. The third one is known as
van der pol oscillator or FitzHugh–Nagumo model, where we applied the FNE algorithm to show its advantages of
dealing with stiff systems.
335
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 2. 1D example: (a) and (b) Exact solutions versus neural network prediction. The x axis presents the initial condition x, and the y axis
represents the time t. The heat map color represents the value of the flow at (t,x). (c) and (d) The absolute errors comparison, no bound
control results vs. bound control results. (For interpretation of the references to color in this figure legend, the reader is referred to the web
version of this article.)

3.1. 1D example

The 1D example we choose is as follows

dx
= −2t x. (20)
dt
We have the exact solutions for such system as
2
ϕ(t, x) = xe−t . (21)
In practice, the network is built with two hidden layers, each of which has eight neurons. The sampling points
are selected from (t, x) ∈ [−1, 1] × [−2, 2]. The upper bound M = 4 is selected. In Fig. 2, 20 samples of time
between [−1,1] are selected, and 500 samples of state variables between [−2,2] are selected. So our training size is
10,000. The training time is 68.6 mins. From the third panel, the error plot, we can see that the max error occurs on
the boundary of our time and state range. In Fig. 2, a comparison of the training without upper bound control. The
exact solution is presented in (a), and the neural net solution is presented in (b). In Fig. 2(c), the max absolute error
336
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 3. For both training, we run the training process with 15,000 function evaluations. The one with control is more accurate than the one
without control.

of training results without bound control is around 0.25, which is occurred on the four corners of the boundary.
However, in Fig. 2(d), the same network is applied for training, but with upper bound control, the absolute error is
bounded by 0.02. The training processes in both case are continued until 15,000 function evaluations. For training
without upper bound control, the last value of loss is 3.178e−2 in Fig. 3(a), while the value is 2.99e−4 in Fig. 3(b).

3.2. 2D example: harmonic oscillators

In two dimensions, we test our method with a simple harmonic oscillators

dx
=y
dt (22)
dy
= −x.
dt
In 2D, our input neurons increase to three and the output neurons increase to two. We first use the network
structure same as Fig. 1, which is two hidden layers of eight neurons each. The training samples’ range are
(t, x, y) ∈ [−.5, .5] × [−2, 2] × [−2, 2], so we choose upper bound M = 2. The exact flow is
( )
x cos t + y sin t
ϕ(t, x, y) = . (23)
y cos t − x sin t
For training, the schematic network shown in Fig. 1 is used. In Fig. 4(a), we show the evolution of the loss
function as the number of iterations increase. In Fig. 4(b), the solution bundles are presented in red dashed curves,
and the exact solutions are presented in black curves for comparison.

3.3. Fitzhugh–Nagumo model

The FitzHugh–Nagumo model [7] is a simple model for biological neuron, it is a relaxation oscillator with
different time scales. It is useful to test the capability of our method on such system. Mathematically, the system
is written as
dx
= −2x 3 + 3x 2 − y
dt (24)
dy
= ϵ(ax − b − y).
dt
337
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 4. (a) The training loss versus number of function evaluations. (b) Comparison of the trained flow and the exact flow. (For interpretation
of the references to color in this figure legend, the reader is referred to the web version of this article.)

Where ϵ = 0.1, a = 2, b = 0.5, the initial condition range is (t, x, y) ∈ [0, 1] × [−0.8, 1.8] × [−0.8, 1.8]. The
upper bound is M = 30.
In Fig. 5, we applied the neural network of Fig. 1, trained it with 16,000 samples. Fig. 5(a) shows the training
results in the phase space, where each short curve is a flow of one initial condition, and the time range is from 0 to
1. For comparison, we plot the corresponding streamline (by classical numerical method) in Fig. 5(b). Comparing
(a) and (b), we see that even a short time interval is applied, the computing results are still not that accurate. We
also found that the numerical error cannot be decreased by increasing the neurons of hidden layers or the levels of
hidden layers. We also show the change of loss in Fig. 5(c). Even the global loss function is minimized to a relative
small error, the local errors on a lot of trajectories (those near the x-nullcline) in 5(a) are notably large. These
makes the neural network result useless in predicting long time behavior. Another problem we have mentioned as
the “nullcline stuck” (see [8]) cannot be solved by direct network training.
In Fig. 6(a), we applied the FNE algorithm for the FitzHugh–Nagumo model, where the end time is T = 30,
and the initial condition x0 = (0.65, 0.5). The time interval (0,30) is split into 30 finite element, each is solved
by a neural element. For comparison, a numerical result without FNE method is presented as Fig. 6(b). Where the
training time range is [0, 5], the trained trajectory failed to match the real solution, and is stuck to the x-nullcline
at time t = 5.

4. Conclusion and prospect

In this paper, we contribute to build a bridge between dynamical systems and machine learning. In particular, we
applied the deep neural networks to approximate system of ODEs and improved the algorithm. The deep learning
network is applied to compute flows of a specific dynamical system. In Section 2, the structure of the neural network
is established, the explicit expression (3) is introduced using the neural net. A customized loss function is given for
training the network. Using the differentiability of the neural network, the derivate of the flow is obtained explicitly,
and the gradient of the loss function is obtained by automatic differentiation. For novel points of this paper, an upper
bound control technique is introduced, which is proved to be efficient and useful in practical training. For instance,
see the 1D example in Section 3. For numerical experiment, we test the method with 1D and 2D examples. Solutions
and error bounds are presented. Additionally, a finite neural element (FNE) algorithm is introduced for computing
long time trajectories of the FitzHugh–Nagumo system.
Comparing to classical methods, such as Finite element methods(FEM), which is clearly faster and more accurate,
but the performance of our method is in the acceptance range. The FEM method can only deal with one initial
338
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 5. (a) Solution bundles in the phase plane. Each short curve represents a subset of the flow on a certain initial condition. (b) Streamline
plot of the model for comparison with the training results. (c) The training loss for the FN model.

condition at a time, our method can solve a bunch of initial conditions at a time, and once a trained network is
obtained, it is capable of predicting solutions with new initial conditions in the neighborhood of the original initial
conditions. Our method has the following advantages comparing with other classical methods. First, the trained
network is differentiable, while the numerical solutions of most classical methods are just discrete. Second, the
solution obtained by this method is reusable. In fact, once the trained network is obtained, other tasks requiring
such dynamic system flow will be simplified. Third, it may reduce the cost of redundant calculations if the optimally
trained networks for various differential equations are shared and made available for free. Fourth, the error at the
test points is smaller than the maximum error at the training points.
Comparing to existing machine learning solvers, In Cosmin, et al. [1]. A deep learning network for solving
PDEs is presented, an adaptive collocation strategy is proposed. The robustness of the neural network is increased
by such strategy. While in our paper, the training points are placed at the beginning, no new points are added. We
will consider this strategy in our future work. In Guo et al. [11], a Deep Collocation Method(DCM) is proposed for
thin plate bending problem, such method solved the difficulty of the C1 continuity requirement in traditional mesh-
based methods. In Guo et al. [10], another application of the DCM is made on heterogeneous porous media, and
the groundwater flow problems are also considered. However, the definition of flow in our paper is a mathematical
concept in the area of dynamical system, which is different than that in their paper. Our method trains the network
339
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 6. One trajectory example: (a) Trained trajectory, the different colored curves represent a trajectory of one neural element. (b) Same
initial condition without FNE method, the trajectory stuck to x-nullcline. (For interpretation of the references to color in this figure legend,
the reader is referred to the web version of this article.)

to fit the flow of the dynamical systems, which means that it satisfies a set of initial conditions. It will greatly
increase the areas of application. In Vien et al. and Samaniego et al. [17,18], a deep energy method is proposed
for solving PDEs, where they modified the loss function as the energetic structure of some BVPs. In our method,
strong form is used, however the discretization of the domain is different than that in those methods.
By applying a proper activation function, we obtained a nice upper bound control for the solutions of differential
equations. There are also other choices for activation function, such as tanh, ReLU, and so on. Three different
types of activation functions were implemented in our simulations, the sigmoid function, tanh function and sine
function. Our upper bound strategy still works for other activation functions as well as the absolute value of the
activation function is bounded. However, the accuracy is sensitive to the differential equation we aim to solve. In
the 1D example and the harmonic oscillators example, the tanh function is more accurate, and the sigmoid function
is failed to predict the solutions of the harmonic oscillators. In the FitzHugh–Nagumo model, the sigmoid function
is more accurate. In all three examples, the sine function has moderate accuracy. The absolute errors are shown in
Fig. 7.
In the numerical experiments of this paper, we fixed the neural architecture, and we found that there is no big
advantages by increasing the number of hidden layers or the neurons in each hidden layer. A study of discovering
the reason of such phenomenon is one of our future directions.
Another direction is the way of sampling grid points. In our experiments, the equidistant grid points are used
for training. If we could consider more training points at regions where we have higher errors, and use less training
points at regions of lower errors. Since the deep learning method can be used for domains of higher dimensions,
the way of assigning training points becomes more important to increase the efficiency of computations.
The idea of the FNE algorithm divided the large neural networks into several small neural networks, this greatly
increased the efficiency of the optimal computations, and it proved its capability of handling multiscale problems.
We are interested in extending the algorithm for other problems where the original deep network method fails, for
example eigenvalue problems.

Appendix. Parameters

We provide tables of hyperparameters that we used in all of those models (see Tables 1–3).

340
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

Fig. 7. Relative errors of different choices of activation functions. Notice that for the result of sigmoid activation function, the relative errors
of the 1D example and the harmonic oscillators are greater than 0.5, which is failed to predict the solutions.

Table 1
Hyperparameters for numerical simulations of the 1D example.
Input layer 2 neurons
Hidden layers 2 dense layers of 16 neurons
Output layer 1 neurons
Batch size 1000
Learning rate 0.1
Training time 169.36 s

Table 2
Hyperparameters for numerical simulations of the 2D harmonic oscillator.
Input layer 3 neurons
Hidden layers 2 dense layers of 16 neurons
Output layer 2 neurons
Batch size 10,000
Learning rate 0.1
Training time 12 653.49 s

Table 3
Hyperparameters for numerical simulations of the FitzHugh–Nagumo oscillator.
Input layer 3 neurons
Hidden layers 2 dense layers of 16 neurons
Output layer 2 neurons
Batch size 2000
Learning rate 0.1
Training time 415.87 s

References
[1] Cosmin Anitescu, Elena Atroshchenko, Naif Alajlan, Timon Rabczuk, Artificial neural network methods for the solution of second
order boundary value problems, Comput. Mater. Contin. 59 (1) (2019) 345–359.
[2] Feiyu Chen, David Sondak, Pavlos Protopapas, Marios Mattheakis, Shuheng Liu, Devansh Agarwal, Marco Di Giovanni, NeuroDiffEq:
A python package for solving differential equations with neural networks, J. Open Source Softw. 5 (46) (2020) 1931, http:
//dx.doi.org/10.21105/joss.01931.
341
G. Liao and L. Zhang Mathematics and Computers in Simulation 202 (2022) 331–342

[3] Xiaolu Chen, Tongfeng Weng, Huijie Yang, Changgui Gu, Jie Zhang, Michael Small, Mapping topological characteristics of dynamical
systems into neural networks: A reservoir computing approach, Phys. Rev. E 102 (3) (2020) 033314.
[4] Thomas F. Coleman, Yuying Li, An Interior Trust Region approach for nonlinear minimization subject to bounds, SIAM J. Optim. 6
(2) (1996) 418–445, http://dx.doi.org/10.1137/0806023.
[5] M.W.M.G. Dissanayake, N. Phan-Thien, Neural-network-based approximations for solving partial differential equations, Commun.
Numer. Methods. Eng. 10 (3) (1994) 195–201, http://dx.doi.org/10.1002/cnm.1640100303.
[6] Howard W. Emmons, The numerical solution of partial differential equations, Quart. Appl. Math. 2 (3) (1944) 173–195.
[7] Richard FitzHugh, Impulses and physiological states in theoretical models of nerve membrane, Biophys. J. 1 (6) (1961) 445–466.
[8] Cedric Flamant, Pavlos Protopapas, David Sondak, Solving differential equations using neural network solution bundles, 2020, arXiv
preprint arXiv:2006.14372.
[9] Vishwa Goudar, Dean V. Buonomano, Encoding sensory and motor patterns as time-invariant trajectories in recurrent neural networks,
ELife 7 (2018) http://dx.doi.org/10.7554/elife.31134.
[10] Hongwei Guo, Xiaoying Zhuang, Pengwan Chen, Naif Alajlan, Timon Rabczuk, Stochastic deep collocation method based on neural
architecture search and transfer learning for heterogeneous porous media, Eng. Comput. (2022) 1–26.
[11] Hongwei Guo, Xiaoying Zhuang, Timon Rabczuk, A deep collocation method for the bending analysis of Kirchhoff plate, 2021, arXiv
preprint arXiv:2102.02617.
[12] Hayrettin Kardestuncer, Douglas H. Norrie, Finite Element Handbook, McGraw-Hill, Inc., 1987.
[13] Isaac E. Lagaris, Aristidis Likas, Dimitrios I. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations,
IEEE Trans. Neural Netw. 9 (5) (1998) 987–1000.
[14] Lu Lu, Xuhui Meng, Zhiping Mao, George Em Karniadakis, DeepXDE: A deep learning library for solving differential equations,
SIAM Rev. 63 (1) (2021) 208–228, http://dx.doi.org/10.1137/19m1274067.
[15] Mantas Lukoševičius, Herbert Jaeger, Reservoir computing approaches to recurrent neural network training, Comp. Sci. Rev. 3 (3)
(2009) 127–149.
[16] Marieme Ngom, Oana Marin, Fourier neural networks as function approximators and differential equation solvers, Stat. Anal. Data
Mining ASA Data Sci. J. (2021).
[17] Vien Minh Nguyen-Thanh, Cosmin Anitescu, Naif Alajlan, Timon Rabczuk, Xiaoying Zhuang, Parametric deep energy approach for
elasticity accounting for strain gradient effects, Comput. Methods Appl. Mech. Engrg. 386 (2021) 114096.
[18] Esteban Samaniego, Cosmin Anitescu, Somdatta Goswami, Vien Minh Nguyen-Thanh, Hongwei Guo, Khader Hamdia, X Zhuang,
T Rabczuk, An energy approach to the solution of partial differential equations in computational mechanics via machine learning:
Concepts, implementation and applications, Comput. Methods Appl. Mech. Engrg. 362 (2020) 112790.
[19] Justin Sirignano, Konstantinos Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, J. Comput.
Phys. 375 (2018) 1339–1364, http://dx.doi.org/10.1016/j.jcp.2018.08.029.

342