entropy

Review
Information Geometry, Fluctuations, Non-Equilibrium
Thermodynamics, and Geodesics in Complex Systems
Eun-jin Kim

Center for Fluid and Complex Systems, Coventry University, Priory St, Coventry CV1 5FB, UK;
[email protected]

Abstract: Information theory provides an interdisciplinary method to understand important phe-

nomena in many research fields ranging from astrophysical and laboratory fluids/plasmas to bio-
logical systems. In particular, information geometric theory enables us to envision the evolution of
non-equilibrium processes in terms of a (dimensionless) distance by quantifying how information
unfolds over time as a probability density function (PDF) evolves in time. Here, we discuss some
recent developments in information geometric theory focusing on time-dependent dynamic aspects
of non-equilibrium processes (e.g., time-varying mean value, time-varying variance, or tempera-
ture, etc.) and their thermodynamic and physical/biological implications. We compare different
distances between two given PDFs and highlight the importance of a path-dependent distance for a
time-dependent PDF. We then discuss the role of the information rate Γ = ddtL and relative entropy
in non-equilibrium thermodynamic relations (entropy production rate, heat flux, dissipated work,
non-equilibrium free energy, etc.), and various inequalities among them. Here, L is the information
length representing the total number of statistically distinguishable states a PDF evolves through over
time. We explore the implications of a geodesic solution in information geometry for self-organization





 and control.

Citation: Kim, E.-j. Information

Keywords: information geometry; entropy; information rate; information length; fluctuations;
Geometry, Fluctuations,
Non-Equilibrium Thermodynamics,
Langevin equations; Fokker-planck equation; time-dependent probability density functions;
and Geodesics in Complex Systems. self-organization
Entropy 2021, 23, 1393. https://
doi.org/10.3390/e23111393

Academic Editor: Miguel Rubi 1. Introduction

Information geometry refers to the application of the techniques of differential geom-
Received: 29 September 2021 etry to probability and statistics. Specifically, it uses differential geometry to define the
Accepted: 19 October 2021
metric tensor that endows the statistical space (consisting of probabilities) with the notion
Published: 24 October 2021
of distance [1–31]. While seemingly too abstract, it permits us to measure quantitative
differences among different probabilities. It then makes it possible to link a stochastic
Publisher’s Note: MDPI stays neutral
process, complexity, and geometry, which is particularly useful in classifying a growing
with regard to jurisdictional claims in
number of data from different research areas (e.g., from astrophysical and laboratory
published maps and institutional affil-
systems to biosystems). Furthermore, it can be used to obtain desired outcomes [6–10,15]
iations.
or to understand statistical complexity [4].
For instance, the Wasserstein metric [6–10] was widely used in the optimal transport
problem where the main interest is to minimize transport cost which is a quadratic function
of the distance between two locations. It satisfies the Fokker-Planck equation for gradient
Copyright: © 2021 by the authors. flow which minimizes the entropy/energy functional [7]. For Gaussian distributions, the
Licensee MDPI, Basel, Switzerland.
Wasserstein metric space consists of physical distances—Euclidean and positive symmetric
This article is an open access article
matrices for the mean and variance, respectively (e.g., see [8]).
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

Entropy 2021, 23, 1393. https://doi.org/10.3390/e23111393 https://www.mdpi.com/journal/entropy

Entropy 2021, 23, 1393 2 of 24

In comparsion, the Fisher (Fisher-Rao) information [32] can be used to define a di-
mensionless distance in statistical manifolds [33,34]. For instance, the statistical distance ds
represents the number of indistinguishable states as [5,33]

dp2j ∂ ln p j ∂ ln p j α β
(ds)2 ≡ ∑ = ∑ p j (d ln p j )2 = ∑ pj dλ dλ = ∑ dλα gαβ dλβ . (1)
j
pj j j,α,β
∂λα ∂λ β α,β

∂ ln p ∂ ln p ∂ ln p ∂ ln p
Here the Fisher information metric gαβ = h ∂λα j ∂λ β j i = ∑ j p j ∂λα j ∂λ β j provides
natural (Riemannian) distinguishability metric on the space of probability distributions.
λα ’s are the parameters of the probability p j and the angular brackets represent the en-
semble average over p j . Note that Equation (1) is given for a discrete probability p j . For a
continuous Probability Density Function (PDF) p( x ) for a variable x, Equation (1) becomes
( p( x )) ∂ ln ( p( x ))
(ds)2 = dxp( x )[d ln ( p( x )]2 = ∑α,β dλα gαβ dλ β where gαβ = dx p( x ) ∂ ln∂λ
R R
α
∂λ β
.
For Gaussian processes, the Fisher metric is inversely proportional to the covariance
matrices of fluctuations in the systems. Thus, in thermodynamic equilibrium, strong
fluctuations lead to a strong correlation and a shorter distance between the neighboring
states [34,35]. Alternatively, fluctuations determine the uncertainty in measurements,
providing the resolution (the distance unit) that normalizes the distance between different
thermodynamic states.
To appreciate the meaning of fluctuation-based metric, let us consider the (equilibrium)
Maxwell-Boltzmann distribution p( Ej ) = βe− βEj for the energy state Ej

p( Ei )
= e− β(Ei −Ej ) . (2)
p( Ej )

Here β = 1/k B T is the inverse temperature; k B is the Boltzmann constant; T is the tem-
perature of the heat bath. In Equation (2), the thermal energy k B T = h Ei of the heat bath
(the width/uncertainty of the probability) provides the resolution to differentiate different
states ∆E = Ei − Ej . The smaller is the resolution (temperature), the more distinguishable
states (more accessible information in the system) there are. It agrees with the expectation
that a PDF gradient (the Fisher-information) increases with information [32].
This concept has been generalized to non-equilibrium systems [36–43], including the
utilization for controlling systems to minimize entropy production [38,40,42], the measure-
ment of the statistical distance in experiments to validate theoretical predictions [41], etc.
However, some of these works rely on the equilibrium distribution Equation (2) that is
valid only in or near equilibrium while many important phenomena in nature and labo-
ratories are often far from equilibrium with strong fluctuations, variability, heterogeneity,
or stochasticity [44–52]. Far from equilibrium, there is no (infinite-capacity) heat bath that
can maintain the system at a certain temperature, or constant fluctuation level. One of the
important questions far from equilibrium is indeed to understand how fluctuation level
β(t)−1 changes with time. Furthermore, PDFs no longer follow the Maxwell-Boltzmann
nor Gaussian distributions and can involve the contribution from (rare) events of large
amplitude fluctuations [53–62]. Therefore, the full knowledge of time-varying PDFs and
the application of information geometry to such PDFs have become of considerable interest.
Furthermore, while in equilibrium [63,64], information theoretical measures (e.g., Shan-
non information entropy) can be given thermodynamic meanings (e.g., heat), in non-
equilibrium such interpretations are not always possible and equilibrium thermodynamic
rules can break down locally (e.g., see [65,66] and references therein). Much progress
on these issues has been made by different authors (e.g., [65–82]) through the develop-
ment of information theory, stochastic thermodynamics, and non-equilibrium fluctua-
tion theorems with the help of the Fisher information [32], relative entropy [83], mutual
information [84,85], etc. Exemplary works include the Landauer’s principle which links
information loss to the ability to extract work [86,87]; the resolution of Maxwell’s de-
mon paradox [88]; black hole thermodynamics [89,90]; various thermodynamic inequal-
Entropy 2021, 23, 1393 3 of 24

ity/uncertainty relations [65,68,91–97]; and linking different research areas (e.g., non-
equilibrium processes to quantum mechanics [98–100], physics to biology [101]).
The paper aims to discuss some recent developments in the information geometric
theory of non-equilibrium processes. Since this would undoubtedly span a broad range of
topics, this paper will have to be selective and will focus on elucidating the dynamic aspect
of non-equilibrium processes and thermodynamic and physical/biological implications.
Throughout the paper, we highlight that time-varying measures (esp. variance) introduces
extra complication in various relations, in particular, between the information geometric
measure and entropy production rate. We make the efforts to make this paper self-contained
(e.g., by including the derivations of some well-known results) wherever possible.
The remainder of this paper is organized as follows. Section 2 discusses different
distances between two PDFs and the generalization for a time-dependent non-equilibrium
PDF. Section 3 compares the distancs from Section 2. Section 4 discusses key thermody-
namic relations that are useful for non-equilibrium processes. Section 5 establishes relations
between information geometric quantities (in Section 2) and thermodynamics (in Section 4).
In Section 6, we discuss the concept of a geodesic in information geometry and its impli-
cations for self-organization or designing optimal protocols for control. Conclusions are
provided in Section 7.

2. Distances/Metrics
This section discusses the distance defined between two probabilities (Section 2.1)
and along the evolution path of a time-dependent probability (Section 2.2). Examples and
comparisons of these distances are provided in Section 3.1. For illustration,
R we use a PDF
p( x, t) of a stochastic variable x and differential entropy S = − dx p( x, t) ln ( p( x, t)) by
using the unit k B = 1.

2.1. Distance Between Two PDFs

We consider the distance between two PDFs p1 = p( x, t1 ) and p2 = p( x, t2 ) of a
stochastic variable x at two times t1 and t2 , respectively where t1 = t2 or t1 6= t2 in general.

2.1.1. Wootters’ Distance

The Wootters’ distance [5,33] is defined in quantum mechanics by the shortest distance
between the two p1 and p2 that have the wave functions ψ1 and ψ2 (p1 = |ψ1 |2 and
p2 = |ψ2 |2 ), respectively. Specifically, for given p1 and p2 , the distance s( p1 , p2 ) between p1
and p2 can be parameterized by infinitely many different paths between p1 and p2 . Letting
z be the affine parameter of a path, we have
v
Z 2 u
ds(z) u dλα dλ β
Z Z
s ( p1 , p2 ) = ds = dz = dzt∑ gαβ , (3)
1 dz α,β
dz dz

where ds is given in Equation (1). Among all possible paths, the minimum of s( p1 , p2 ) is
obtained for a particular path that optimizes the quantum distinguishability; the (Hilbert-
space) angle between the two wave functions provides such minimum distance as
Z 
−1 1 1
W [ p1 , p2 ] = cos dx [ p( x, t1 )] [ p( x, t2 )] .
2 2 (4)

Equation (4) is for a pure state and has been generalized to mixed states (e.g., see [37,102]
and references therein). Note that the Wootters’ distance is related to the Hellinger dis-
tance [43].
Entropy 2021, 23, 1393 4 of 24

2.1.2. Kullback-Leibler (K-L) Divergence/Relative Entropy

Kullback-Leibler (K-L) divergence between the two PDFs [83], also called relative
entropy, is defined by
!
p( x, t1 )
Z
K( p1 | p2 ) = dx p( x, t1 ) ln . (5)
p( x, t2 )

Relative entropy quantifies the difference between a PDF p1 and another PDF p2 . It
takes the minimum zero value for identical two PDFs p1 = p2 and becomes large as p1 and
p2 become more different. However, as it is defined in Equation (5), it is not symmetric
between p1 and p2 and does not satisfy the triangle inequality. It is thus not a metric in a
strict sense.

2.1.3. Jensen Divergence

The Jensen divergence (also called Jensen distance) is the symmetrized Kullback–
Leibler divergence defined by

1h i
J ( p1 | p2 ) = K( p1 | p2 ) + K( p2 | p1 )) . (6)
2
p
While the square root of the Jensen-Shannon divergence J ( p1 | p2 ) is a metric [4,103],
J ( p1 | p2 ) itself has also been used in examining statistical complexity (e.g., see [43,104,105]).

2.1.4. Euclidean Norm

In analysis of big data, the Euclidean norm [5,106] is used, which is defined by
Z 2
| p1 − p2 |2

= dx p( x, t1 ) − p( x, t2 ) .
(7)

While Equation (7) has a direct analogy to the physical distance, it has a limitation
in measuring statistical complexity due to the neglect of the stochastic nature [5]. For
instance, the Wootters’ distance in Equation (4) was shown to work better than Equation (7)
in capturing complexity in the logistic map [5].

2.2. Distance along the Path

Equations (4)–(7) can be used to define the distance between the two given PDFs
p( x, t1 ) and p( x, t2 ) at times t1 and t2 (t2 > t1 ). However, p( x, t) at the intermediate time
t = (t1 , t2 ) can take an infinite number of different values depending on the exact path
that a system takes between p( x, t1 ) and p( x, t2 ). One example would be i) p( x, t1 ) =
p( x, t2 ) = p( x, t) for all t = (t1 , t2 ) and x, in comparison with ii) p( x, t1 ) = p( x, t2 ) but
p( x, t) 6= p( x, t1 ) and p( x, t) 6= p( x, t2 ). What is necessary is a path-dependent distance that
depends on the exact evolution and the form of p( x, t) for t = (t1 , t2 ).

2.2.1. Information Rate

Calculating a path-dependent distance for a time-dependent PDF p( x, t) requires the
generalization of the distance in Section 2.1. To this end, we consider two (temporally) ad-
jacent PDFs along the trajectory, say, p( x, t) and p( x, t + dt) and calculate the (infinitesimal)
relative entropy between them in the limit dt → 0 to the leading order in O(dt):

1 1
lim 2
K[ p( x, t + dt)| p( x, t)] = lim K[ p( x, t)| p( x, t + dt)]
dt→0 ( dt ) dt→0 ( dt )2
1 1 1
Z
= lim 2
J [ p ( x, t + dt )| p ( x, t )] = dxp( x, t)(∂t ln p( x, t))2 ≡ Γ2 . (8)
dt→0 ( dt ) 2 2
Entropy 2021, 23, 1393 5 of 24

Here, we used p( x, t + dt) = p( x, t) + (dt)∂t p + 12 (dt)2 (∂t p)2 + O((dt)3 ), ln(1 + r ) =

1 2
+ O(r3 ) for r  R1, and dx∂t p( x, t) = dx∂tt p( x, t) = 0 because of the total
R R
r− 2r
probability conservation dxp( x, t) = 1. Due to the symmetry of K[ p( x, t + dt)| p( x, t)] to
leading order O((dt)2 ), K[ p( x, t + dt)| p( x, t)] = J [ p( x, t + dt)| p( x, t)] to O((dt)2 ).
In Equation (8), the information rate Γ is defined by [15–29]

2
Z Z
Γ2 (t) = lim J [ p( x, t + dt)| p( x, t)] = dxp( x, t)(∂t ln p( x, t))2 = 4 dx (∂t q(t))2 .
dt→0 ( dt )2
(9)
√
Here, q = p, and Γ ≥ 0 by definition. We note that the last term in terms of q
in Equation (9) can be used when q = p = 0. The dimensions of Γ2 ≡ E and Γ are
(time)−2 and (time)−1 , respectively. They do not change their values under nonlinear, time-
independent transformation of variables (see Appendix A). Thus, using the unit where the
length is dimensionless, E and Γ can be viewed as the kinetic energy per mass and velocity,
respectively. For this reason, Γ was called the velocity (e.g., in [15,17]).
Note that E can be viewed as the Fisher information [32] if time is interpreted as
a parameter (e.g., [97]). However, time in classical mechanics is a passive quantity that
cannot be changed by an external control. Γ is also called the entropy production rate in
quantum mechanics [107]. However, as shown in Sections 4.1 and 4.4, the relation between
Γ and thermodynamic entropy production rate is more complicated (see Equation (28)).
Γ in Equation (9) is the information rate representing how quickly new information
is revealed as a PDF evolves in time. Here, Γ−1 = τ is the characteristic time scale of this
information change in time. To show that Γ is related to fluctuation’s smallest time scale [97],
we assume that λα ’s are the estimators (parameters) of a p( x, t) and use the Cramér-
−1
R
Rao bound on the Fisher information gαβ = dxp( x, t)∂λα [ln p( x, t)]∂λ β [ln p( x, t)] ≥ Cαβ
where C αβ ≡ hδλα δλ β i is the covariance matrix (e.g., see [32]); δλα = λα − hλα i denotes
d ln p ∂ ln p α
fluctuation. Using dt = ∂λα dλ dt then leads to

dλα dλ β dλα −1 dλ β
Γ2 = ∑ dt
gαβ
dt
≥ ∑ C
dt αβ dt
. (10)
αβ αβ

For the diagonal gαβ = gα δαβ , Equation (10) is simplified as

 2
1 dλα
∑ h(δλα )2 i dt
≤ Γ2 . (11)
α

Equation (11) shows how the RMS fluctuation-normalized rate at which the parameter
λα can change is bounded above by Γ. If there is only α = 1 (λα = λδα,1 ), Equation (11) is
further simplified:

1 dλ
p ≤ Γ, (12)
h(δλ)2 i dt

clearly showing that λ normalized by its RMS fluctuations cannot change faster than the
information rate.
Finally, it is worth highlighting that Equation (9) is general and can be used even when
the parameters λα ’s and gαβ in Γ2 in Equation (10) are unknown. Examples include the
cases where PDFs are empirically inferred from experimental/observational data. Readers
are referred to Refs. [21,23,28] for examples. It is only the special case where we have a
complete set of parameters λα ’s of a PDF that we can express Γ using the Fisher information
as in Equation (10). For instance, for a Gaussian p( x, t) that is fully described by the mean
1
value h x i and variance 2β , (λ1 , λ2 ) = (h x i, β).
Entropy 2021, 23, 1393 6 of 24

2.2.2. Information Length

Since Γ ∝ J [ p( x, t + dt)| p( x, t)] is a metric [103] as noted in Section 2.1, Γ is also a
p

metric. Thus, we sum Γ along the trajectory to define a finite distance. Specifically, starting
with an initial PDF p( x, t = 0), we integrate Γ(t) over time to obtain the dimensionless
number as a function time as
Z t
L(t) = dt1 Γ(t1 ). (13)
0

L is the information length [15–31] that quantifies the total change in information along
the trajectory of p( x, t) or the total number of statistically distinguishable states it evolves
through over time. [We note that different names (e.g., statistical length [108], or statistical
distance [97]) were also used for L.] It is important to note that unlike the Wootters’ distance
(the shortest distance among all possible paths between the two PDFs) in Equation (3)
(e.g., [5]), L(t) in Equation (13) is fixed for a given time-evolving PDF p( x, t).
By definition in Equation (13), L(t = 0) = 0 and L(t) monotonically increases with
time since Γ ≥ 0 (e.g., see Figure A2 in [22]). L(t) takes a constant value only in a stationary
state (Γ = ∂t p = 0). One of its important consequences is that when p( x, t) relaxes into a
stationary PDF in the long time limit t → ∞, Γ(t) → 0 and L(t) → L∞ as t → ∞ where L∞
is a constant depending on initial conditions and parameters. This property of L∞ was used
to understand attractor structure in a relaxation problem; specifically, Refs. [15,16,18,22,28]
calculated L∞ for different values of the mean position x0 of an initial PDF and examined
how L∞ depends on x0 . Furthermore, Γ and L were shown to be useful for quantifying
hysteresis in forward-backward processes [19], correlation and self-regulation among
different players [23,25], and predicting the occurrence of sudden events [27] and phase
transitions [23,25]. Some of these points are illustrated in Section 3.1.

3. Model and Comparison of Metrics

For discussing/comparing different metrics in Section 2 and statistical measures in
Section 4, we use the following Langevin model [109]

dx
= f ( x, t) + ξ = −∂ x V ( x, t) + ξ. (14)
dt
Here, V ( x, t) is, in general, a time-dependent potential which can include an internal
potential and an external force; ξ is a short (delta)-correlated Gaussian noise with the
following statistical property

hξ (t)ξ (t0 )i = 2Dδ(t − t0 ). (15)

Here, the angular brackets represent the ensemble average over the stochastic noise ξ;
D ≥ 0 is the amplitude of ξ. It is important to note that far from equilibrium, the average
(e.g., h x (t)i) is a function of time, in general.
The exact PDFs can be obtained for the Ornstein-Uhlenbeck (O-U) process which has
V = γ2 ( x − v(t))2 and f = −γ( x − v(t)) in Equation (14). Here, v(t) is a deterministic
function of time. Specifically, for the initial Gaussian PDF p( x, 0)
r
β 0 − β 0 ( x − x0 )2
p( x, 0) = e , (16)
π
a time-dependent PDF remains Gaussian at all time:
Z r
β − β( x−h xi)2
p( x, t) = dx1 p( x, t; x1 , 0) p( x1 , 0) = e , (17)
π
1 e−2γt D (1 − e−2γt )
= + , (18)
2β(t) 2β 0 γ
Entropy 2021, 23, 1393 7 of 24

Z t
h x (t)i = x0 e−γt + γ dt1 e−γ(t−t1 ) v(t1 ). (19)
0

1
In Equations (16)–(19), x0 = h x (t = 0)i, β 0 = β(t = 0), and h(δx )2 i = = σ2 . Here,
2β
β, σ and σ2 are the inverse temperature, standard deviation, and variance, respectively;
β 0 and x0 are the values of β and h x i, respectively, at t = 0. Equation (18) shows that as
t → ∞, β(t → ∞) = 2D γ
. Note that we use both β and σ here to clarify the connections to
the previous works [15,17,22,26–28].

3.1. Geometric Structure of Equilibrium/Attractors

To elucidate the main difference between the distances in Equations (4)–(6) and (13),
we consider the relaxation problem by assuming v(t) = 0. In the following, we compare
the distance between p( x, 0) and p( x, t → ∞) by using p1 ( x, t1 ) = p( x, 0) and p2 ( x, t2 ) =
p( x, t → ∞) in Equations (4)–(6) and Equation (13). Analytical expressions for these
distances are given in [22].
Each curve in Figure 1 shows how each distance depends on the initial mean position
x0 . The four different curves are for L∞ (in blue), Wootters’ distance (in orange), K-L
relative entropy (in green), and Jensen divergence (in red), respectively. The relative
entropy and Jensen divergence exhibit similar behavior, the red and green color curves
being superimposed on each other. Of note is a linear relation between L∞ and x0 in
Figure 1. Such linear relation is not seen in other distances. This means that the information
length is a unique measure that manifests a linear geometry around its equilibrium point
in a linear Gaussian process [28,30]. Note that for a nonlinear force f , L∞ has a power-law
relation with x0 for a sufficiently large x0 [18,28]. These contrast with the behaviour in a
chaotic system [16,28] where L∞ depends sensitively on the initial condition and abruptly
changes with x0 . Thus, the information length provides a useful tool to geometrically
understand attractor structures in relaxation problems.

Figure 1. The distance against x0 between p( x, 0) and p( x, t → ∞) for the O-U process. (Figure 1
in [22]).

3.2. Correlation between Two Interacting Components

We next show that the information length is also useful in elucidating the correla-
tion between two interacting species such as two competing components relaxing to the
same equilibrium in the long time limit. Specifically, the two interacting components
with the time-dependent PDFs P1 ( x, t) and P2 ( x, t) are coupled through the Dichoto-
mous noise [110,111] (see Appendix B) and relax into the same equilibrium Gaussian PDF
P1 ( x, t → ∞) = P2 ( x, t → ∞) = 12 P( x, t → ∞) around x = 0 in the long time limit. Here,
P( x, t) = P1 ( x, t) + P2 ( x, t) is the total PDF. For the case considered below, P( x, t) satisfies
the O-U process (see Appendix B for details). We choose the initial conditions where
P1 (t = 0) = P1 (t → ∞) with zero initial mean value while P2 (t = 0) takes an initial mean
value x0 . These are demonstrated in the cartoon figure, Figure 2a,c.
Entropy 2021, 23, 1393 8 of 24

P1 P1 P1

x x x
P2 P2 P2

x0 x x x
(a) t=0 (b) 0 < t < ∞ (c) t=∞

Figure 2. P1 (top) and P2 (bottom) at time t = 0 in panel (a), t = (0, ∞) in panel (b), and t → ∞ in
panel (c). Note that P1 (0 < t < ∞) 6= P1 (t = 0) (= P1 (t → ∞)).

Although P1 (t = 0) = P1 (t → ∞), at the intermediate time t = (0, ∞), P1 ( x, t) evolves

in time due to its coupling to P2 and thus P1 ( x, t) 6= P1 ( x, t = 0), as shown in Figure 2b.
Consequently, L(t) calculated from P1 monotonically increases to its asymptotic value
L∞ until it reaches the equilibrium (see Figure A2 in [22] for time-evolution of L from P1
and P2 ). On the other hand, P2 with an initial mean value x0 undergoes a different time
evolution (unless x0 = 0) until it reaches the equilibrium.
The distances in Equations (4)–(7) and (13) can be calculated from the total P =
P1 + P2 , P1 and P2 for different values of x0 . Results are shown in Figure 3a–c, respec-
tively; (a) P( x, 0) and P( x, t → ∞), (b) P1 ( x, 0) and P1 ( x, t → ∞), and (c) P2 ( x, 0) and
P2 ( x, t → ∞), respectively. Specifically, for each value of x0 , we calculate the distances
in Equations (4)–(7) and (13) by using p1 ( x, t1 ) = P( x, 0) and p2 ( x, t2 ) = P( x, t → ∞) for
Figure 3a; p1 ( x, t1 ) = P1 ( x, 0) and p2 ( x, t2 ) = P1 ( x, t → ∞) for Figure 3b; p1 ( x, t1 ) =
P2 ( x, 0) and p2 ( x, t2 ) = P2 ( x, t → ∞) for Figure 3c. The same procedure above is then
repeated for many other x0 ’s to show how each distance depends on x0 .

(a) (b) (c)

Figure 3. The distance between P( x, 0) and P( x, t → ∞) against in x0 in (a); P1 ( x, 0) and P2 ( x, t → ∞) in (b); P2 ( x, 0) and P2 ( x, t → ∞)
in (c). (Figure 4 in [22]).

For the total P, a linear relation between L∞ and x0 is seen in Figure 3a (like in
Figure 1). This linear relation is not seen in L∞ calculated from either P1 or P2 in Figure3b
or Figure 3c; a non-monotonic dependence of L∞ in Figure 3b,c is due to large-fluctuations
and strong-correlation between P1 and P2 during time-evolution for large x0 . What is
quite remarkable is that in contrast to other distances, L∞ calculated from P1 and P2 in
Figure 3b,c exhibits a very similar dependence on x0 . It means that despite very different
time-evolutions of P1 and P2 (see Figure 2), they undergo similar total change in information.
These results suggest that strong coupling between two components can be inferred from
their similar information length (see also [24,25]).
Entropy 2021, 23, 1393 9 of 24

4. Thermodynamic Relations
To elucidate the utility of information geometric theory in understanding
non-equilibrium thermodynamics, we review some of the important thermodynamic
measures of irreversibility and dissipation [112] and relate them to information geometric
measures Γ and K [29]. For illustration below, we use the model in Equations (14) and
(15) unless stated otherwise. Corresponding to Equations (14) and (15) is the following
Fokker-Planck equation [109]

∂2 p( x, t)
 
∂p( x, t) ∂
= − f ( x, t) p( x, t) + D = −∂ x J ( x, t), (20)
∂t ∂x ∂x2

where J = f p − D∂ x p is the probability current.

4.1. Entropy Production Rate and Flow

For non-equilibrium thermodynamics, we need to consider the entropy in the system
S and the environment Sm , and the total entropy ST = S + Sm . To clarify the difference
among these, we go over some derivation by using ∂t p = −∂ x J and J = f p − D∂ x p
to obtain
dS( x, t) dS ( x, t) dSm ( x, t)
Z
Ṡ = = − dx∂t p ln p = T − , (21)
dt dt dt
where,
!  
dST 1 2 dSm 1
Z Z
ṠT = = dx J , Ṡm = = dx Jf . (22)
dt Dp dt D

Here, we used integration by parts in t and x. ṠT denotes the (total) entropy production rate,
which is non-negative ṠT ≥ 0 by definition, and serves as a measure of irreversibility [112].
The sign of Ṡm in Equation (22) represents the direction in which the entropy flows between
the system and environment. Specifically, Ṡm > 0 (Ṡm < 0) when the entropy flows from the
system (environment) to the environment (system). Ṡm is related to the heat flux Q = DSm
from the system to the environment. The equality ṠT = 0 holds in an equilibrium reversible
process. In this case, Ṡ = −Ṡm = − Q D , which is the usual equilibrium thermodynamic
relation. In comparison, when Ṡ = 0, ṠT = Ṡm ≥ 0.
For the O-U process with V = γ2 ( x − v(t))2 and f = −γ( x − v(t)) in Equations (14),
(17)–(19), (21) and (22) lead to (see [29] for details)
" #
1 π
S(t) = 1 + ln , (23)
2 β
( ∂ t β )2
D S˙T = = + (∂t h x i)2 = (∂t σ)2 + (∂t h x i)2 , (24)
8β3
∂t β ∂t σ
Ṡ = − = , (25)
2β σ
Ṡm = ṠT − Ṡ. (26)
h i
  ∂t β
Here, we used J = f + 2Dβ(δx ) p = − 2β (δx ) + ∂t h x i p, ∂ x p = −2β(δx ) p, f =
−γ( x − v(t)), ∂t h x i = h f i, 2Dβ − γ = − ∂2β
tβ
and ,
∂t β
= −2 ∂σt σ .
β
In order to relate these thermodynamical quantities ṠT and Ṡ above to the information
rate Γ, we recall that for the O-U process [15,17,26–28],

( ∂ t β )2 1h i
E = Γ2 = 2β(∂t h x i)2 + = 2 ( ∂ t σ ) 2
+ ( ∂ t h x i) 2
. (27)
2β2 σ2
Entropy 2021, 23, 1393 10 of 24

Equations (24) and (27) then give us

D
Γ2 = ṠT + Ṡ2 . (28)
σ2
Interestingly, Equation (28) reveals that the entropy production rate needs be normal-
ized by variance σ2 . This is because of the extensive nature of ṠT unlike Γ or Ṡ. That is, ṠT
changes its value when the variable x is rescaled by a scalar factor, say, α (> 0) as y = αx.
Furthermore, Equation (28) shows that the information rate Γ in general does not have a
simple relation to the entropy production rate (c.f., [107]).
One interesting limit of Equation (28) is the case of constant β(t) with Ṡ = 0. In that
case, Equation (24) becomes D ṠT = (∂t h x i)2 while Equations (13), (27) and (28) give us

h x (t)i − h x (t = 0)i
L(t) = , (29)
σ
1 D
Γ2 = (∂t h x i)2 = 2 ṠT . (30)
σ2 σ
Equation (29) simply states that L measures the total change in the mean value
normalized by fluctuation level σ. Equation (30) manifests a linear relation between Γ2
and ṠT when ∂t σ = 0, as invoked in the previous works (e.g., [107]). Furthermore, a
linear relation between Γ2 and ṠT in Equation (30) implies that minimizing the entropy
Rt Rt
production dt1 ṠT along the trajectory corresponds to minimizing 0 dt1 Γ2 (t1 ), which, in
turn, is equivalent to minimizing L(t) (see Section 5 for further discussions).
Finally, to demonstrate how entropy production rate and thermal bath temperature
(D) are linked to the speed of fluctuations c = σΓ [97], we rewrite Equation (28) as
h i1
2
c = σΓ = D ṠT + σ2 Ṡ2 . (31)
p
For constant variance β̇ = 0, Equation (31) gives a simple relation c = σΓ = D ṠT .

4.2. Non-Equilibrium Thermodynamical Laws

To relate the statistical measures in Section 4.1 to thermodynamics, we let U (inter-
nal energy) be the average potential energy U = hV i and obtain (see also [66,113] and
references therein)

dU d
= hV i ≡ Ẇ − Q̇, (32)
dt dt
where
Z
Ẇ = dx (∂t V ) p = h∂t V i, (33)
Z Z
Q̇ = − dxV (∂t p) = dxJ f = h f ẋ i = D Ṡm . (34)

The power Ẇ represents the average rate at which the work is done to the system
because of time-varying potential; the average work during the time interval [t0 , t] is
Rt
calculated by W = t dt0 Ẇ (t0 ). On the other hand, Q̇ represents the rate of dissipated heat.
0
Equation (32) establishes the non-equilibrium thermodynamic relation U̇ = Ẇ − Q̇.
Physically, it simply means that the work done to the system Ẇ increases U while the
dissipated heat to the environment Q̇ decreases it. Equations (21), (32), and (34) permit
us to define a non-equilibrium (information) free energy F (t) = U (t) − DS(t) [92] and
its time-derivative

Ḟ = U̇ − D Ṡ = Ẇ − D ṠT , (35)
Entropy 2021, 23, 1393 11 of 24

dF dU
where Ḟ = dt and U̇ = dt . Since ṠT ≥ 0, Equation (35) leads to the following inequality

D ṠT = Ẇ − Ḟ ≡ ẆD ≥ 0, (36)

where the non-negative dissipated power (lost to the environment) ẆD is defined. Fi-
nally, the time-integral version of Equation (36) provides the bound on the average work
performed on the system as W − ∆F = WD ≥ 0 (e.g., [68]).

4.3. Relative Entropy as a Measure of Irreversibility

The relative entropy has proven to be useful in understanding irreversibilities and
non-equilibrium thermodynamic inequality relations [91–94,114–116]. In particular, the
dissipated work WD = W − ∆F (in Equation (36)) is related to the relative entropy between
the PDFs in the forward and reverse processes

WD = D K[ p F (γF (t))| p R (γR (t))]. (37)

(e.g., see [91–94].) Here, p F (γF (t)) and p R (γR (t)) are the PDFs for the forward and reverse
processes driven by the forward γF (t) and reverse γR (t) protocols, respectively. Using
Equation (36) in Equation (37) immediately gives

d
ṠT = K[ p F (γF (t))| p R (γR (t))] ≥ 0, (38)
dt
which is a proxy for irreversibility (see [115,116] for a slightly different expression of
Equation (38)). It is useful to note that forward and reversal protocols are also used to
establish various fluctuations theorems for different dissipative measures such as entropy
production, dissipated work, etc. (see, e.g., [80] for a nice review and references therein).
However, we cannot consider forward and reversal protocols in the absence of a
model control parameter that can be prescribed as a function of time. Even in this case,
the relative entropy is useful in quantifying irreversibility through inequalities, and this is
what we focus on in the remainder of Section 4.3.
To this end, let us consider a non-equilibrium state p( x, t) which has an instantaneous
non-equilibrium stationary state ps ( x, t) and calculate the relative entropy between the two.
Here, ps ( x, t) is a steady solution of the Fokker-Planck equation ∂t ps = 0 in Equation (20)
V ( x,t)−Fs (t)
(e.g., see [29]). Specifically, one finds ps ( x, t) = e− D by treating the parameters to
be constant (being frozen to their instantaneous values at a given time). Here, V and Fs are
the potential energy and the stationary free energy, respectively. For clarity, an example of
ps ( x, t) is given in Section 4.4.
The average of ln ps (t) in the non-equilibrium state p( x, t) can be expressed as follows:

1 1
Z Z
dxp( x, t) ln ps ( x, t) = − dxp( x, t)(V ( x, t) − Fs (t)) = − (U (t) − Fs (t)). (39)
D D
Equations (35) and (39) then give us
" #
p( x, t)
Z
F (t) − Fs (t) = D dxp( x, t) ln ≡ D K[ p( x, t)| ps ( x, t)] ≥ 0. (40)
ps ( x, t)

Here, we used the fact the relative entropy is non-negative. Equation (40) explicitly
shows that non-equilibrium free energy is bounded below by the stationary one F ≥ Fs
(see also [1,92] and references therein for open Hamiltonian systems).
Equation (40) together with Equation (35) then lead to the following irreversible work
Wirr [29,92]:

Wirr ≡ W − ∆Fs = D∆ST + ∆(F − Fs ) = D∆ST + D∆K[ p( x, t)| ps ( x, t)]. (41)

Entropy 2021, 23, 1393 12 of 24

Here, ∆K[ p( x, t)| ps ( x, t)] = K[ p( x, t)| ps ( x, t)] − K[ p( x, t0 )| ps ( x, t0 )], etc. The deriva-
tion of Equation (41) for open-driven Hamiltonian systems is provided in [92] (see their
Equation (38)).
On the other hand, we directly calculate the time-derivative of K[ p( x, t)| ps ( x, t)] in
V ( x,t)−Fs (t)
(40) by using ps ( x, t) = e−
R
Equation D , ṠT = Ṡ + Ṡm , dxp∂t V = Ẇ and Q̇ =
− dx ∂t pV = D Ṡm , and Wirr = W − ∆Fs :
R

Z   
d 1 d 1 d
K[ p( x, t)| ps ( x, t)] = −Ṡ + dxV p − Fs = −ṠT + Ẇ − Fs . (42)
dt D dt D dt

One can see easily that equating Equation (42) to D1 [Ḟ − Ḟs ] (from Equation (40))
d
simply recovers Ẇ − dt F = D ṠT in Equation (35).
d
Finally, we obtain a differential form of Equation (41) by using Ẇirr = Ẇ − dt Fs in
Equation (42) as follows

d
Ẇirr = D ṠT + D K[ p( x, t)| ps ( x, t)]. (43)
dt

4.4. Example
We consider v(t) = ut with a constant u so that V = − γ2 ( x − ut)2 in Equation (14).
While the discussion below explicitly involves v(t), the results are general and valid for
the limiting case v(t) = 0. The case with v(t) = 0 is an example where the forward and
reversal protocols do not exist while a non-equilibrium stationary state does.
For f = −γ( x − ut), Equation (19) is simplified as follows
u 
h x (t)i = x0 e−γt + ut − 1 − e−γt . (44)
γ
γ
For the non-equilibrium stationary state with fixed γ and D, β s = 2D is also constant
d
( dt Fs = 0). Therefore, we have
r
β s − β s ( x−ut)2
ps ( x, t) = e . (45)
π
Then, we can find (see [29] for details)
!
1  2 1
K[ p( x, t)| ps ( x, t)] = −1 + ln ( β/β s ) + β s (h x i − ut) + , (46)
2 2β
" #
  2 1
D ṠT = −γh x0 ie−γt + u 1 − e−γt + (2βD − γ)2 , (47)
2β
   2

γ
−γh x0 ie−γt + u 1 − e−γt


Q̇ = − 2βD − γ , (48)
2β
Z  
Ẇ = −u dxγ( x − ut) p = −uγh x − uti = u2 1 − e−γt , (49)
d 1h i 1
K[ p( x, t)| ps ( x, t)] = − (∂t h x i)2 + (∂t σ)2 − u∂t h x i = −ṠT + Ẇ. (50)
dt D D
Here, we used Equations (23)–(26), h x i and β in Equations (44) and (18), respectively, and
Ẇ = h−γu( x − ut)i = u∂t h x i.
It is worth looking at the two interesting limits of Equations (46)–(50). First, in the
long time limit as t → ∞, the following simpler relations are found:
γ
h x i → u(t − γ−1 ), 2β → = 2β s ,
D
D ṠT = Q̇ = Ẇ = (σΓ)2 → u2 ,
Entropy 2021, 23, 1393 13 of 24

d
Ṡ → 0, K[ p( x, t)| ps ( x, t)] → 0. (51)
dt
Equation (51) illustrates how the external force v(t) = ut 6= 0 keeps the system out of
equilibrium even in the long time limit, with non-zero entropy production and dissipation.
When there is no external force u = 0, the system reaches equilibrium as t → ∞, and all
quantities in Equation (51) apart from β become zero.
The second is when the system is initially in equilibrium with β(t = 0) = β(t → ∞) =
γ
2D and h x0 i = 0 and evolve in time as it is driven out of equilibrium by u 6= 0. As u does
γ
not affect variance, β(t) = β 0 = 2D (∂t σ = 0) and Ṡ = 0 for all time. In this case, we find

D ṠT = Q̇ = u2 (1 − e−γt )2 = (σΓ)2 , Ẇ = u2 (1 − e−γt ),

u2
D K[ p( x, t)| ps ( x, t)] = (1 − e−γt )2 ,
2γ
d
D K[ p( x, t)| ps ( x, t)] = u2 (1 − e−γt )e−γt . (52)
dt

Equation (52) shows that ṠT , Q̇, Γ2 , Ẇ, and K start with zero values at t = 0 and monotoni-
cally increase to their asymptotic values as t → ∞.
Finally, both cases considered above in Equations (51) and (52) have ∂t σ = 0 and thus
recover Equation (30):
D Ṡ Q̇
Γ2 = 2T = 2 . (53)
σ σ

5. Inequalities
R (e.g., hV i) and the average
Section 4 utilized the average (first moment) of a variable
of its first time derivative (h∂t V i = Ẇ) while the work W = dt Ẇ is defined by the time
integral of Ẇ = h∂t V i in Equation (33). This section aims to show that the rates at which
average quantities vary with time are bounded by fluctuations and Γ. Since the average
and time derivatives do not commute, we pay particular attention to when the average
is taken.
To this end, let us first define the microscopic free energy µ = V + D ln p (called
the chemical potential energy in [113]). In terms of µ, we have J = − p∂ x µ and hµi =
U − DS = F . On the other hand,
ṗ
∂t µ = ∂t V + D , h∂t µi = h∂t V i = Ẇ. (54)
p

h∂t µi = Ẇ means that the average rate of change in the microscopic free is the power.
From Equation (54), it follows

ṗ2
Z
h(∂t µ − ∂t V )2 i = D2 dx = D 2 Γ2 . (55)
p

Equation (55) establishes the relation between the microscopic free energy and Γ.
Next, we calculate the time-derivative of F

d d
Z Z
F = hµi = hµ̇i + dxµ ṗ = Ẇ + dxµ ṗ. (56)
dt dt
d
Using dt F = Ẇ − D ṠT in Equation (56) gives ṠT in terms of µ as

d
Z
Ẇ − F = D S˙T = − dxµ ṗ. (57)
dt

Equation (57) is to be used in Section 5.1 for linking ṠT to Γ through an inequality.
Entropy 2021, 23, 1393 14 of 24

5.1. General Inequality Relations

R R R
We now use dx ṗ = 0, dx ṗh Ai = h Ai dx ṗ = 0 for any A = A( x, t) and apply the
Rt Rt 1 Rt 1
Schwartz inequality | dt1 A(t1 ) B(t1 )| ≤ [ dt1 A2 (t1 )] 2 [ dt1 B2 (t1 )] 2 to Equations (21),
(34) and (57) to obtain
Z Z 1/2
|Ṡ| = | dx ṗ ln p| ≤ Γ dxp(δ ln p)2 , (58)
Z Z 1/2
2
| Q̇| = | dxV ṗ| ≤ Γ dxp(δV ) , (59)
Z Z 1/2
2
D ṠT = dxµ ṗ ≤ Γ dxp(δµ) . (60)

Equation (60) (Equation (59)) establishes the inequality between entropy production
rate (heat flux) and the product of the RMS fluctuations of the microscopic free energy
(potential energy) and Γ. Since δµ = δV + D (δ ln p), we have

h(δµ)2 i = h(δV )2 i + D2 h(δ ln p)2 i + 2D hδVδ ln pi. (61)

These relations are to be used in Section 5.2 below.

5.2. Applications to the Non-Autonomous O-U Process

For a linear O-U process with V = γ2 ( x − v(t))2 and f = −γ( x − v(t)) in Equation (14),
1
we use h(δx )2 i = 2β , h(δx )4 i = 3h(δx )2 i = 4β3 2 and ∂t h x i = −γh x − v(t)i to show
" #
γ 2 1
δV = −∂t h x iδx + (δx ) −
2 2β
σ4
h(δV )2 i = (∂t h x i)2 σ2 + γ2 ,
2
1
δ ln p = − β(δx )2 ,
2
1
h(δ ln p)2 i = ,
2
γ
h(δ ln p)(δV )i = − . (62)
4β

t ∂β
Using Equations (61)–(62) in Equations (58)–(60) together with 2Dβ − γ = − 2β and
∂t β
β = −2 ∂σt σ leads to

1
|Ṡ| ≤ √ Γ,
2
" #1
γ 2 σ2 2
| Q̇| ≤ Γσ (∂t h x i)2 + ,
2
 1
2 1 2
2
D ṠT ≤ Γσ (∂t h x i) + (∂t σ ) . (63)
2

Finally, it is useful to examine the extreme cases of Equation (63). First, when ∂t σ = 0,
Equation (63) holds as an equality as D ṠT = (∂t h x i)2 = σ2 Γ2 (see Equation (27)), recovering
Equation (28) with ∂t σ = 0. Second,q when ∂t h x i = 0, Equation (63) again holds as an
equality since D ṠT = (∂t σ )2 and Γ 1 2
2 σ (∂t σ )
2 = ( ∂ t σ )2 .
Entropy 2021, 23, 1393 15 of 24

6. Geodesics, Control and Hyperbolic Geometry

The section aims to discuss geodesics in information geometry and its implications
for self-organization and control. To illustrate the key concepts, we utilize an analytically
solvable, generalized O-U process given by

dx
= −γ(t)[ x − v(t)] + ξ, (64)
dt
where γ(t) > 0 is a damping constant; v(t) is a deterministic force which determines
the time evolution of the mean value of x; ξ is a short (delta)-correlated noise with the
time-dependent amplitude D (t) in general, satisfying Equation (15).
For the initial condition in Equation (16), the mean value h x i ≡ y(t) and β(t) are given by
Rt Z t Rt
y(t) = h x i = x0 e − 0 dt1 γ(t1 )dt1
+ dt1 e− 0 dt1 [γ(t1 )−γ(t)]dt1
γ ( t1 ) f ( t1 ) , (65)
0
Rt
e −2 dt1 γ(t1 )dt1 Z t
1 0 Rt
= h( x − h x i) i = 2
+ dt1 e−2 0 dt1 [γ(t1 )−γ(t)]dt1
2D (t1 ), (66)
2β(t) 2β 0 0

where x0 = h x (t = 0)i.

6.1. Geodesics–Shortest-Distance Path

A geodesics between the two spatial locations is a unique path with the shortest
distance. A similar concept can be applied to information geometry to define a unique
evolution path between the two given PDFs, say, p( x, t1 ) and p( x, t2 ) in the statistical
space. The Wootters’ distance in quantum mechanics in Equation (4) is such an example.
For time-varying stochastic processes, there is an infinite number of different trajectories
between the two PDFs at different times. The key question that we address in this section
is how to find an exact time evolution of p( x, t) when initial and final PDFs [15] are given.
This is a much more difficult problem than finding a minimum distance between two PDFs
(like the Wootter’s distance). In the following, we sketch some main steps needed for
finding such a unique evolution path (the so-called geodesics) between given initial and
final PDFs by minimizing L (see [15] for detailed steps).
For the O-U process in Equation (64), a geodesic solution does not exist for constant γ,
v(t) and D. Thus, finding a geodesic solution boils down to determing suitable functions
of γ(t), v(t) or D (t) [15]. To be specific, let p( x, t0 ) and p( x, t F ), respectively, be the PDFs
at the time t = t0 and t F (> t0 ) and find a geodesic solution by minimizing L(t) =
R tF 0 0
R tF 0 0
t0 dt Γ ( t ). The latter is equivalent to minimizing t0 dt E ( t ) and to keeping Γ constant.
(This geodesics is also called an optimal path (e.g., see [107]).) We rewrite E in Equation (27)
for the O-U process in terms of y = h x i
 2  2
1 dβ dy
E= 2 + 2β . (67)
2β dt dt

The Euler-Lagrange equation

dE d dE dE d dE
0= − , 0= − (68)
dβ dt d β̇ dy dt dẏ

dβ dy
( β̇ = dt and ẏ = dt ) then gives us

1 dβ 2
 2
d2 β
 
2 dy
− − 2β = 0, (69)
dt2 β dt dt
 
d dy dy
β =0→β = c, (70)
dt dt dt
Entropy 2021, 23, 1393 16 of 24

where c is constant. An alternative method of obtaining Equations (69) and (70) is provided
in Appendix C. The following equations are obtained from Equations (69) and (70) [15]
 2
dβ
= −4c2 β + αβ2 , (71)
dt
1 2 α
Γ2 = β̇ + 2c2 β = , (72)
2β2 2

where α is another (integration) constant. General solutions to Equations (70) and (71) for
c 6= 0 were found in terms of hyperbolic functions as [15]
√  √
4c2 1√ 1√
  
α α
β(t) = cosh2 α(t − A) , y(t) = tanh α(t − A) − + B, (73)
α 2 2c 2 2c

where A and B are constant.

1
Equation (73) can be rewritten using σ = (2β)− 2 and z = √y as follows
2

B Γ
(z − zc )2 + σ2 = R2 , zc = √ − sR, R = , (74)
2 2c

where s denotes the sign of c so that s = 1 when c > 0 while s = −1 when c < 0.
Equation (74) is an equation of a circle for the variables z and σ with the radius R and the
center zc , defined in the upper-half plane where σ ≥ 0. Thus, geodesic motions occur along
the portions of a circle as long as c 6= 0 (as can be seen in Figure 4). A geodesic moves
on a circle with a larger radius for a larger information rate Γ and speed and vice versus.
This manifests the hyperbolic geometry in the upper half Poincaré model [13,117] where
the half-plane represents z and σ 6= 0 (see also Appendix D). The constants c, α, A, and
B determine the coordinate of the center and the radius of the circle R. These constants
should be fixed by the fixed conditions at the initial t = 0 and final time t F .
Having found the most general form of the geodesic solution for y(t) and β, the
next steps require finding the values of constant values c, α, A, B to satisfy the boundary
conditions at t = t0 and t F , and then finding appropriate γ(t), D (t), and v(t) that ensure
the geodesic solutions. This means the O-U process should be controlled by γ(t), D (t) and
v(t) to ensure a geodesic solution.

(a) (b)
2 1.87

1.5 1.86
y
1.85
β -1/2

1 β -1/2
1.84
0.5 1.83
0 1.82
0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8
time (β 0=0.3) y
(c) (d)
0.8 0.7

0.6
0.65
β -1/2

0.4 y
0.6
0.2
β -1/2
0 0.55
0 0.2 0.4 0.6 0.2 0.4 0.6 0.8
time (β 0=3) y

Figure 4. y and β−1/2 against time for β 0 = 0.3 and 3 in (a,c), respectively; the corresponding
geodesic circular segments in the (y, β−1/2 ) upper half-plane in (b,d), respectively. In both cases,
y0 = 56 and y F = 30
1
. (Figure 3 in [15]).
Entropy 2021, 23, 1393 17 of 24

Figure 4 shows an example of a geodesic solution in the upper half-plane y and

β−1/2 when γ(t) = 1 is constant while D (t) and v(t) are time-dependent. The boundary
conditions are chosen as y(t0 ) = y0 = 65 and y(t F ) = y F = 30 1
in all panels (a)–(d).
β(t0 ) = β 0 = β(t F ) = β F = 0.3 in panels (a) and (b) while β 0 = β F = 3 in panels (c) and
(d). Interestingly, circular-shape phase-portraits are seen in panels (b) and (d), reflecting
hyperbolic geometry noted above (see also Appendix D) [13,117]. The speed at which the
geodesic
q motion takes place in the phase portrait is determined by the constant value of
Γ= α
2 (i.e., the larger α, the faster time evolution).
Figure 5a,b are the corresponding PDF snapshots at different times (shown in different
colors), demonstrating how the PDF evolves from the initial PDF in red to the final PDF
in blue. In both cases, it is prominent that the PDF width (∝ β−1/2 ) initially broadens and
then becomes narrower.

Figure 5. Time evolution of PDFs against x: (a) β 0 = 0.3 corresponding to Figure 4a,b; (b) β 0 = 3
corresponding to Figure 4c,d. In both cases, y0 = 56 and y F = 30 1
. The initial and final PDFs are
shown by thick red and blue lines, respectively. (Figure 4a,b in [15]).

6.2. Comments on Self-Organization and Control

Self-organization (also called homeostasis) is the novel phenomena where order
spontaneously emerges out of disorder and is maintained by different feedbacks in complex
systems [45,52,53,118–123]. The extremum principles of thermodynamics such as the
minimum entropy production (e.g., [119,121]) or maximum entropy entropy production
(e.g., [122,123]) have been proposed by considering a steady state or an instant time in
different problems.
However, far from equilibrium, self-organization can be a time-varying non-equilibrium
process involving perpetual or large fluctuations (e.g., see [52–54]). In this case, the extreme
of entropy production should be on accumulative entropy production over time rather
than at one instant time nor in a steady state. That is, we should consider p the time-integral
of the entropy production ṠT , or equivalently, the time-integral of ṠT . As seen from
Equations (24) and (53), for p a linear O-U process with a constant variance, there is an exact
Rt
proportionality between ṠT and Γ. In this case, the extreme of L(t) = dt1 Γ(t1 ) would
Rt p p
be the same as the extreme of dt1 ṠT . However, as noted previously, Γ ∝ ṠT does
not hold in general (e.g., see Equation (28)).
With these comments, we now look at the implications of a geodesic for self-organization,
in particular, in biosystems. For the very existence and optimal functions of a living organiss,
it is critical to minimize the dispersion of its physical states and to maintain its states within
certain bounds upon changing conditions [124]. How R fast its state changes in time can be
quantified by the surprise rate ∂t [ln ( p( x, t)]. Since dxp( x, t)∂t ln ( p( x, t)) = 0, we use
its RMS value h(∂t ln p)2 i = Γ (see Equation (9)) and realize that the total change over
p
Rt
a finite time interval [t0 , t F ] is nothing more than L( t) = t F dt1 Γ(t1 ). Thus, minimizing
0
the accumulative/time-integral of the RMS surprise rate is equivalent to minimizing L.
Envisioning surprise rate as biological cost associated with changes (e.g., needed in updating
Entropy 2021, 23, 1393 18 of 24

the future prediction based on the current state [124,125]), we can then interpret L as an
accumulative biological cost. Thus, geodesic would be an optimal path that minimizes such
an accumulative biological cost.
Ref [15] addressed how to utilize this idea to control populations (tumors). Specifically,
the results in Section 6.1 were applied to a nonlinear stochastic growth model (obtained
by a nonlinear change of variables of the O-U process), and the geodesic solution in
Equation (73) was used to find the optimal protocols v(t) and D (t) in reducing a large-
size tumor to a smaller one. Here, in this problem, D (t) represents the heterogeneity of
tumor cells (e.g., larger D for metastatic tumor) that can be controlled by gene reprogram-
ming while v(t) models the effect of a drug or radiation that reduces the mean tumor
population/size.

7. Discussions and Conclusions

There has been a growing interest in information geometry from theoretical and
practical considerations. This paper discussed some recent developments in information
geometric theory, focusing on time-dependent dynamic aspects of non-equilibrium pro-
cesses (e.g., time-varying mean value, time-varying variance, or temperature) and their
thermodynamic and physical/biological implications.
In Sections 2 and 3, by utilizing a Langevin model of an over-damped stochastic
process x (t), we highlighted the importance of a path-dependent distance L in describing
time-varying processes. In Sections 4 and 5, we elucidated the thermodynamic meanings of
the relative entropy and the information rate Γ by relating them to the entropy production
rate (ṠT ), Ṡ, heat flux (Q = D Ṡm ), dissipated work (ẆD ), etc., and demonstrated the role of
Γ in determining bounds (or speed limit) on thermodynamical quantities.
Specifically,
q in the O-U process, we showed the exact relation
p
Γ = D
σ
2
2 Ṡ T + Ṡ (Equation (28)), which is simplified as σΓ = D ṠT when ∂t σ = 0
p
(σ = h(δx )2 i is the standard deviation of x). Finally, Section 6 discussed geodesic and
its implication for self-organization as well as the underlying hyperbolic geometry. It
remains future works to explore the link between Γ and the entropy production rate in
other (e.g., nonlinear) systems consisting of three or more interacting components or data
from self-organizing systems (e.g., normal brain).

Funding: This research received no funding.

Data Availability Statement: Data are available from the author.
Acknowledgments: Eun-jin Kim acknowledges the Leverhulme Trust Research Fellowship (RF-
2018-142-9) and thanks the collaborators, especially, James Heseltine who contributed to the works in
this paper.
Conflicts of Interest: The author declares no conflicts of interest.

Appendix A
In this Appendix, we show the invariance of Equation (9) when x changes as y = F ( x ).
Using the conservation of the probability, we then have
−1
dx dF ( x )
p(y, t) = p( x, t) = p( x, t) , (A1)
dy dx

dF ( x ) dF ( x ) −1
Since dx is independent of time t, it follows that ∂t p(y, t) = [∂t p( x, t)] dx .
dF ( x )
Using this and dy = dx dx , we have

" #2 Z " #2
1 ∂p(y, t) 1 ∂p( x, t)
Z
dy = dx . (A2)
p(y, t) ∂t p( x, t) ∂t
Entropy 2021, 23, 1393 19 of 24

This shows that p( x, t) and p(y, t) give the same Γ2 (t).

Appendix B. The Coupled O-U Process

The coupled O-U process for Figure 3 in Section 3.1 is governed by the Fokker-Planck
equation [22]
∂P1 ∂ ∂2 P
= [γ1 ( x − µ) P1 ] + D 21 − f 0 P1 + g0 P2 , (A3)
∂t ∂x ∂x
∂P2 ∂ ∂2 P
= [γ2 ( x − µ) P2 ] + D 22 + f 0 P1 − g0 P2 . (A4)
∂t ∂x ∂x
Here, D is the strength of a short-correlated Gaussian noise given by Equation (15).
These equations are the coupled O-U processes with the coupling constants f 0 and g0
through the Dichotomous noise [110,111].
For simplicity, we use γ1 = γ2 = γ and f 0 = g0 = e and the following initial conditions
r
1 β 10
P1 ( x, 0) = exp [− β 10 x2 ], (A5)
2 π
r
1 β 20
P2 ( x, 0) = exp [− β 20 ( x − x0 )2 ]. (A6)
2 π
The solutions are given by
"r r #
1 β1 −2et − β 1 x2 β2 −2et − β 2 ( x −e−γt x0 )2
P1 ( x, t) = (1 + e )e + (1 − e )e , (A7)
4 π π
"r r #
1 β1 −2et − β 1 x2 β2 −2et − β 2 ( x −e−γt x0 )2
P2 ( x, t) = (1 − e )e + (1 + e )e , (A8)
4 π π

where
1 e−2γt D 
= + 1 − e−2γt , (A9)
2β m 2β m0 γ
for m = 1, 2. In the limit of t → ∞, P1 and P2 in Equations (A7) and (A8) approach the same
equilibrium distribution
r
1 β m ( t → ∞ ) − β m (t→∞) x2
Pm ( x, t) = e , (A10)
2 π

where β m (t → ∞) = 2D γ
. We note that the total PDF P = P1 + P2 satisfies the single O-U
process where the initial PDF is given by the sum of Equations (A5) and (A6).
Figure 3 is shown for the fixed parameter values γ = 0.1, D = 1 and e = 0.5,
β 20 = β 10 = 0.05 = β(t → ∞) = 2Dγ
= 0.05. Different values of the initial mean position x0
of P2 are used to examine how metrics depend on x0 . As noted in Section 2.2, P1 at t = 0 is
chosen to be the same as the final equilibrium state which has the zero mean value and
inverse temperature β 10 = 0.05.

Appendix C. Curved Geometry: The Christoffel and Ricci-Curvature Tensors

A geodesic solution in Section 6.1 can also be found by solving the geodesic equation
in general relativity (e.g., [31,107]). To this end, we let the two parameters be λ1 = h x i = y
and λ2 = β and express Equation (A11) in terms of the metric tensor gij as follows (see also
Equation (10))
dλi dλ j
E =∑ g , (A11)
ij
dt ij dt
Entropy 2021, 23, 1393 20 of 24

where ! !
1
2β2
0 i β
gij = , λ = . (A12)
0 2β y

Note that while gij is diagonal, the 1-st diagonal component depends on β (the
second parameter). That is, gii is not independent of j-th parameter for j 6= i in gen-
eral. From
h Equation (A12), i we can find non-zero components of connection tensor
Γijk = 1
2 ∂i g jk + ∂ j gik − ∂k gij (Γijk = gim Γ jkm )

1 1
Γ111 = − , Γ122 = −2β2 , Γ212 = Γ221 = . (A13)
β 2β

d2 λ i m dλk
A geodesic equation dt2
+ Γimk dλdt dt = 0 in terms of the Christoffel tensors becomes

β̈ + Γ111 β̇2 + Γ122 ẏ2 = 0, (A14)

ÿ + Γ212 β̇ẏ + Γ221 β̇ẏ = 0. (A15)

Equations (A13)–(A15) give Equation (70). Note that if gii is independent of the λ j (j 6= i)
for all i and j, the Christoffel tensors have non-zero values only for Γiii , leading to a much
simpler geodesic solution (e.g., see [31]).
Finally, to appreciate the curved geometry associated with this geodesic solution,
p
we proceed to calculate the Riemann curvature tensor Rikmn = ∂m Γink + Γimp Γnk − ∂n Γimk −
p
Γinp Γmk and the Ricci tensor Rij = Rikj
k from Equation (A13) and find the following non-zero

components [15]
1
R1212 = − R1221 = − β, R2112 = − R2121 = . (A16)
4β2
Non-zero curvature tensors represent that the metric space is curved with a finite
k and curvature R:
curvature. Specifically, we find the Ricci tensor Rij = Rikj

1
R11 = − , R22 = − β, R12 = R21 = 0, (A17)
4β2
R = gij Rij = −1 (A18)

The negative curvature is typical of hyperbolic geometry. Finally, using R = −1, we

calculate the Einstein field equation
! !
1 − 4β1 2 0 1 1
2β 2 0
Gij = Rij − R gij = + = 0. (A19)
2 0 −β 2 0 2β

Since Gij = 8πTij where Tij is the stress-energy tensor, we see that Tij = 0 for this problem.

Appendix D. Hyperbolic Geometry

The Hyperbolic geometry in the upper-half plane [13,117] becomes more obvious
when Equation (A12) is expressed in terms of the two parameters h x (t)i and σ (t) where x
and y axes represent h x (t)i and σ (t) with the metric tensor
!
2
0
gijH = σ 2
. (A20)
0 σ12
Entropy 2021, 23, 1393 21 of 24

References
1. Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley: New York, NY, USA, 1991.
2. Parr, T.; Da Costa, L.; Friston, K.J. Markov blankets, information geometry and stochastic thermodynamics. Philos. Trans. R. Soc.
A 2019, 378, 20190159. [CrossRef] [PubMed]
3. Oizumi, M.; Tsuchiya, N.; Amari, S. Unified framework for information integration based on information geometry. Proc. Nat.
Am. Soc. 2016, 113, 14817. [CrossRef] [PubMed]
4. Kowalski, A.M.; Martin, M.T.; Plastino, A.; Rosso, O.A.; Casas, M. Distances in Probability Space and the Statistical Complexity
Setup. Entropy 2011, 13, 1055–1075. [CrossRef]
5. Martin, M.T.; Plastino, A.; Rosso, O.A. Statistical complexity and disequilibrium. Phys. Lett. A 2003, 311, 126–132. [CrossRef]
6. Gibbs, A.L.; Su, F.E. On choosing and bounding probability metrics. Int. Stat. Rev. 2002, 70, 419–435. [CrossRef]
7. Jordan, R.; Kinderlehrer, D.; Otto, F. The variational formulation of the Fokker–Planck equation. SIAM J. Math. Anal. 1998, 29,
1–17. [CrossRef]
8. Takatsu, A. Wasserstein geometry of Gaussian measures Osaka J. Math. 2011, 48 1005–1026.
9. Lott, J. Some geometric calculations on Wasserstein space. Commun. Math. Phys. 2008, 277, 423–437. [CrossRef]
10. Gangbo, W.; McCann, R.J. The geometry of optimal transportation. Acta Math. 1996, 177, 113–161. [CrossRef]
11. Zamir, R. A proof of the Fisher information inequality via a data processing argument. IEEE Trans. Inf. Theory 1998, 44, 1246–1250.
[CrossRef]
12. Otto, F.; Villani, C. Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality. J. Funct.
Anal. 2000, 173, 361–400. [CrossRef]
13. Costa, S.; Santos, S.; Strapasson, J. Fisher information distance. Discrete Appl. Math. 2015, 197, 59–69. [CrossRef]
14. Ferradans, S.; Xia, G.-S.; Peyré, G.; Aujol, J.-F. Static and dynamic texture mixing using optimal transport. Lecture Notes Comp. Sci.
2013, 7893, 137–148.
15. Kim, E.; Lee, U.; Heseltine, J.; Hollerbach, R. Geometric structure and geodesic in a solvable model of nonequilibrium process.
Phys. Rev. E 2016, 93, 062127. [CrossRef] [PubMed]
16. Nicholson, S.B.; Kim, E. Investigation of the statistical distance to reach stationary distributions. Phys. Lett. A 2015, 379, 83–88.
[CrossRef]
17. Heseltine, J.; Kim, E. Novel mapping in non-equilibrium stochastic processes. J. Phys. A 2016, 49, 175002. [CrossRef]
18. Kim, E.; Hollerbach, R. Signature of nonlinear damping in geometric structure of a nonequilibrium process. Phys. Rev. E
2017, 95, 022137. [CrossRef] [PubMed]
19. Kim, E.; Hollerbach, R. Geometric structure and information change in phase transitions. Phys. Rev. E 2017, 95, 062107. [CrossRef]
20. Kim, E.; Jacquet, Q.; Hollerbach, R. Information geometry in a reduced model of self-organised shear flows without the uniform
coloured noise approximation. J. Stat. Mech. 2019, 2019, 023204. [CrossRef]
21. Anderson, J.; Kim, E.; Hnat, B.; Rafiq, T. Elucidating plasma dynamics in Hasegawa-Wakatani turbulence by information
geometry. Phys. Plasmas 2020, 27, 022307. [CrossRef]
22. Heseltine, J.; Kim, E. Comparing information metrics for a coupled Ornstein-Uhlenbeck process. Entropy 2019, 21, 775. [CrossRef]
[PubMed]
23. Kim, E.; Heseltine, J.; Liu, H. Information length as a useful index to understand variability in the global circulation. Mathematics
2020, 8, 299. [CrossRef]
24. Kim, E.; Hollerbach, R. Time-dependent probability density functions and information geometry of the low-to-high confinement
transition in fusion plasma. Phys. Rev. Res. 2020, 2, 023077. [CrossRef]
25. Hollerbach, R; Kim, E.; Schmitz, L. Time-dependent probability density functions and information diagnostics in forward and
backward processes in a stochastic prey-predator model of fusion plasmas. Phys. Plasmas 2020, 27, 102301. [CrossRef]
26. Guel-Cortez, A.J.; Kim, E. Information Length Analysis of Linear Autonomous Stochastic Processes. Entropy 2020, 22, 1265.
[CrossRef] [PubMed]
27. Guel-Cortez, A.J.; Kim, E. Information geometric theory in the prediction of abrupt changes in system dynamics. Entropy 2021,
23, 694. [CrossRef]
28. Kim, E. Investigating Information Geometry in Classical and Quantum Systems through Information Length. Entropy 2018,
20, 574. [CrossRef]
29. Kim, E. Information geometry and non-equilibrium thermodynamic relations in the over-damped stochastic processes. J. Stat.
Mech. Theory Exp. 2021, 2021, 093406. [CrossRef]
30. Parr, T.; Da Costa, L.; Heins, C.; Ramstead, M.J.D.; Friston, K.J. Memory and Markov Blankets. Entropy 2021, 23, 1105. [CrossRef]
31. Da Costa, L. ; Thomas, P.; Biswa, S.; Karl, F.J. Neural Dynamics under Active Inference: Plausibility and Efficiency of Information
Processing. Entropy 2021, 23, 454. [CrossRef]
32. Frieden, B.R. Science from Fisher Information; Cambridge University Press: Cambridge, UK, 2004.
33. Wootters, W. Statistical distance and Hilbert-space. Phys. Rev. D 1981, 23, 357–362. [CrossRef]
34. Ruppeiner, G. Thermodynamics: A Riemannian geometric model. Phys. Rev. A. 1079, 20, 1608. [CrossRef]
35. Salamon, P.; Nulton, J.D.; Berry, R.S. Length in statistical thermodynamics. J. Chem. Phys. 1985, 82, 2433–2436. [CrossRef]
36. Nulton, J.; Salamon, P.; Andresen, B.; Anmin, Q. Quasistatic processes as step equilibrations. J Chem. Phys. 1985, 83, 334.
[CrossRef]
Entropy 2021, 23, 1393 22 of 24

37. Braunstein, S.L.; Caves, C.M. Statistical distance and the geometry of quantum states. Phys. Rev. Lett. 1994, 72, 3439. [CrossRef]
38. Diósi, L.; Kulacsy, K.; Lukács, B.; Rácz, A. Thermodynamic length, time, speed, and optimum path to minimize entropy
production. J. Chem. Phys. 1996, 105, 11220. [CrossRef]
39. Crooks, G E. Measuring thermodynamic length. Phys. Rev. Lett. 2007, 99, 100602. [CrossRef]
40. Salamon, P.; Nulton, J.D.; Siragusa, G.; Limon, A.; Bedeaus, D.; Kjelstrup, D. A Simple Example of Control to Minimize Entropy
Production. J. Non-Equilib. Thermodyn. 2002, 27, 45–55. [CrossRef]
41. Feng, E.H.; Crooks, G.E. Far-from-equilibrium measurements of thermodynamic length. Phys. Rev. E. 2009, 79, 012104. [CrossRef]
42. Sivak, D.A.; Crooks, G.E. Thermodynamic Metrics and Optimal Paths. Phys. Rev. Lett. 2012, 8, 190602. [CrossRef] [PubMed]
43. Matey, A.; Lamberti, P.W.; Martin, M.T.; Plastron, A. Wortters’ distance resisted: A new distinguishability criterium. Eur. Rhys. J.
D 2005, 32, 413–419.
44. d’Onofrio, A. Fractal growth of tumors and other cellular populations: linking the mechanistic to the phenomenological modeling
and vice versa. Chaos Solitons Fractals 2009, 41, 875. [CrossRef]
45. Newton, A.P.L.; Kim, E.; Liu, H.-L. On the self-organizing process of large scale shear flows. Phys. Plasmas 2013, 20, 092306.
[CrossRef]
46. Kim, E.; Liu, H.-L.; Anderson, J. Probability distribution function for self-organization of shear flows. Phys. Plasmas 2009,
16, 0552304. [CrossRef]
47. Kim, E.; Diamond, P.H. Zonal flows and transient dynamics of the L-H transition. Phys. Rev. Lett. 2003, 90, 185006. [CrossRef]
[PubMed]
48. Feinberg, A.P.; Irizarry, R.A. Stochastic epigenetic variation as a driving force of development, evolutionary adaptation, and
disease. Proc. Natl. Acad. Sci. USA 2010, 107, 1757. [CrossRef] [PubMed]
49. Wang, N.X.; Zhang, X.M.; Han, X.B. The effects of environmental disturbances on tumor growth. Braz. J. Phys. 2012, 42, 253.
[CrossRef]
50. Lee, J.; Farquhar, K.S.; Yun, J.; Frankenberger, C.; Bevilacqua, E.; Yeung, E.; Kim, E.; Balázsi, G.; Rosner, M.R. Network of mutually
repressive metastasis regulators can promote cell heterogeneity and metastatic transitions. Proc. Natl. Acad. Sci. USA 2014,
111, E364. [CrossRef] [PubMed]
51. Lee, U.; Skinner, J.J.; Reinitz, J.; Rosner, M.R.; Kim, E. Noise-driven phenotypic heterogeneity with finite correlation time. PLoS
ONE 2015, 10, e0132397.
52. Haken, H. Information and Self-Organization: A Macroscopic Approach to Complex Systems, 3rd ed., Springer: Berlin/Heidelberg,
Germany, 2006; pp. 63–64.
53. Kim, E. Intermittency and self-organisation in turbulence and statistical mechanics. Entropy 2019, 21, 574. [CrossRef]
54. Aschwanden, M.J.; Crosby, N.B.; Dimitropoulou, M.; Georgoulis, M.K.; Hergarten, S.; McAteer, J.; Milovanov, A.V.; Mineshige, S.;
Morales, L.; Nishizuka, N.; et al. 25 Years of Self-Organized Criticality: Solar and Astrophysics. Space Sci. Rev. 2016, 198, 47–166.
[CrossRef]
55. Zweben, S.J.; Boedo, J.A.; Grulke, O.; Hidalgo, C.; LaBombard, B.; Maqueda, R.J.; Scarin, P.; Terry, J.L. Edge turbulence
measurements in toroidal fusion devices. Plasma Phys. Contr. Fusion 2007, 49, S1–S23. [CrossRef]
56. Politzer, P.A. Observation of avalanche-like phenomena in a magnetically confined plasma. Phys. Rev. Lett. 2000, 84, 1192–1195.
[CrossRef] [PubMed]
57. Beyer, P.; Benkadda, S.; Garbet, X.; Diamond, P.H. Nondiffusive transport in tokamaks: Three-dimensional structure of bursts and
the role of zonal flows. Phys. Rev. Lett. 2000, 85, 4892–4895. [CrossRef]
58. Drake, J.F.; Guzdar, P.N.; Hassam, A.B. Streamer formation in plasma with a temperature gradient. Phys. Rev. Lett. 1988, 61,
2205–2208. [CrossRef]
59. Antar, G.Y.; Krasheninnikov, S.I.; Devynck, P.; Doerner, R.P.; Hollmann, E.M.; Boedo, J.A.; Luckhardt, S.C.; Conn, R.W. Ex-
perimental evidence of intermittent convection in the edge of magnetic confinement devices. Phys. Rev. Lett. 2001, 87, 065001.
[CrossRef]
60. Carreras, B.A.; Hidalgo, E.; Sanchez, E.; Pedrosa, M.A.; Balbin, R.; Garcia-Cortes, I. van Milligen, B.; Newman, D.E.; Lynch, V.E.
Fluctuation-induced flux at the plasma edge in toroidal devices. Phys. Plasmas 1996, 3, 2664–2672. [CrossRef]
61. De Vries, P.C.; Johnson, M.F.; Alper, B.; Buratti, P.; Hender, T.C.; Koslowski, H.R.; Riccardo, V. JET-EFDA Contributors, Survey of
disruption causes at JET. Nucl. Fusion 2011, 51, 053018. [CrossRef]
62. Kates-Harbeck, J.; Svyatkovskiy, A.; Tang, W. Predicting disruptive instabilities in controlled fusion plasmas through deep
learning. Nature 2019, 568, 527. [CrossRef]
63. Landau, L.; Lifshitz, E.M. Statistical Physics: Part 1. In Course of Theoretical Physics; Elsevier Ltd.: New York, NY, USA, 1980;
Volume 5.
64. Shannon, C.E. A Mathematical Theory of Communication. Bell Syst. Tech. J. 1948, 27, 623. [CrossRef]
65. Jarzynski, C.R. Equalities and Inequalities: Irreversibility and the Second Law of Thermodynamics at the Nanoscale. Annu. Rev.
Condens. Matter Phys. 2011, 2, 329–351. [CrossRef]
66. Sekimoto, K. Stochastic Energetic; Lecture Notes in Physics 799; Springer: Heidelberg, Germany, 2010. [CrossRef]
67. Jarzynski, C.R. Comparison of far-from-equilibrium work relations. Physique 2007, 8, 495–506. [CrossRef]
68. Jarzynski, C.R. Nonequilibrium Equality for Free Energy Differences. Phys. Rev. Lett. 1997, 78, 2690–2693. [CrossRef]
Entropy 2021, 23, 1393 23 of 24

69. Evans, D.J.; Cohen, G.D.; Morriss, G.P. Probability of Second Law Violations in Shearing Steady States. Phys. Rev. Lett. 1993,
71, 2401. [CrossRef]
70. Evans, D.J.; Searles, D.J. The Fluctuation Theorem. Adv. Phys. 2012, 51, 1529–1585. [CrossRef]
71. Gallavotti, G.; Cohen, E.G.D. Dynamical Ensembles in Nonequilibrium Statistical Mechanics. Phys. Rev. Lett. 1995, 74, 2694.
[CrossRef]
72. Kurchan, J. Fluctuation theorem for stochastic dynamics. J. Phys. A Math. Gen. 1998, 31, 3719. [CrossRef]
73. Searles, D.J.; Evans, D.J. Ensemble dependence of the transient fluctuation theorem. J. Chem. Phys. 2000, 13, 3503. [CrossRef]
74. Seifert, U. Entropy production along a stochastic trajectory and an integral fluctuation theorem. Phys. Rev. Lett. 2005, 95, 040602.
[CrossRef] [PubMed]
75. Abreu, D.; Seifert, U. Extracting work from a single heat bath through feedback. EuroPhys. Lett. 2011, 94 10001. [CrossRef]
76. Seifert, U. Stochastic thermodynamics, fluctuation theorems and molecular machines. Rep. Prog. Phys. 2012, 75, 26001. [CrossRef]
77. Spinney, R.E.; Ford, I.J. Fluctuation relations: A pedagogical overview. arXiv 2012, arXiv:1201.6381S.
78. Haas, K.R.; Yang, H.; Chu, J.-W. Trajectory Entropy of Continuous Stochastic Processes at Equilibrium. J. Phys. Chem. Lett. 2014,
5, 999. [CrossRef]
79. Van den Broeck, C. Stochastic thermodynamics: A brief introduction. Phys. Complex Colloids 2013, 184, 155–193.
80. Murashita, Y. Absolute Irreversibility in Information Thermodynamics. arXiv 2015, arXiv:1506.04470.
81. Tomé, T. Entropy Production in Nonequilibrium Systems Described by a Fokker-Planck Equation. Braz. J. Phys. 2016, 36,
1285–1289. [CrossRef]
82. Salazar, D.S.P. Work distribution in thermal processes. Phys. Rev. E 2020, 101, 030101. [CrossRef] [PubMed]
83. Kullback, S. Letter to the Editor: The Kullback-Leibler distance. Am. Stat. 1951, 41, 340–341.
84. Sagawa, T. Thermodynamics of Information Processing in Small Systems; Springer: Berlin/Heidelberg, Germany, 2012.
85. Nielsen, M.A.; Chuang, I.L. Quantum Computation and Quantum Information; Cambridge University Press: Cambridge, UK, 2000.
86. Landauer, R. Irreversibility and heat generation in the computing process. IBM J. Res. Dev. 1961, 5, 183–191. [CrossRef]
87. Bérut, A.; Arakelyan, A.; Petrosyan, A.; Ciliberto, S.; Dillenschneider, R.; Lutz, E. Experimental verification of Landauer’s
principle linking information and thermodynamics. Nature 2012, 483, 187. [CrossRef]
88. Leff, H.S.; Rex, A.F. Maxwell’s Demon: Entropy, Information, Computing; Princeton University Press: Princeton, NJ, USA, 1990.
89. Bekenstein, J.D. How does the entropy/information bound work? Found. Phys. 2005, 35, 1805. [CrossRef]
90. Capozziello, S.; Luongo, O. Information entropy and dark energy evolution. Int. J. Mod. Phys. D 2018, 27, 1850029. [CrossRef]
91. Kawai, R.; Parrondo, J.M.R.; Van den Broeck, C. Dissipation: The phase-space perspective. Phys. Rev. Lett. 2007, 98, 080602.
[CrossRef] [PubMed]
92. Esposito, M.; Van den Broeck, C. Second law and Landauer principle far from equilibrium. Europhys. Lett. 2011, 95, 40004.
[CrossRef]
93. Horowitz, J.; Jarzynski, C.R. An illustrative example of the relationship between dissipation and relative entropy. Phys. Rev. E
2009, 79, 021106. [CrossRef] [PubMed]
94. Parrondo, J.M.R.; van den Broeck, C.; Kawai, R. Entropy production and the arrow of time. New J. Phys. 2009, 11, 073008.
[CrossRef]
95. Deffner, S.; Lutz, E. Information free energy for nonequilibrium states. arXiv 2012, arXiv:1201.3888.
96. Horowitz, J.M.; Sandberg, H. Second-law-like inequalities with information and their interpretations. New J. Phys. 2014, 16, 125007.
[CrossRef]
97. Nicholson, S.B.; García-Pintos, L.P.; del Campo, A.; Green, J.R. Time-information uncertainty relations in thermodynamics. Nat.
Phys. 2020, 16, 1211–1215. [CrossRef]
98. Flego, S.P.; Frieden, B.R.; Plastino, A.; Plastino, A.R.; Soffer, B.H. Nonequilibrium thermodynamics and Fisher information: Sound
wave propagation in a dilute gas. Phys. Rev. E 2003, 68, 016105. [CrossRef]
99. Carollo, A.; Spagnolo, B.; Dubkov, A.A.; Valenti, D. On quantumness in multi-parameter quantum estimation. J. Stat. Mech.
Theory E 2019, 2019, 094010. [CrossRef]
100. Carollo, A.; Valenti, D.; Spagnolo, B. Geometry of quantum phase transitions. Phys. Rep. 2020, 838, 1–72 . [CrossRef]
101. Davies, P. Does new physics lurk inside living matter?. Phys. Today 2020, 73, 34. [CrossRef]
102. Sjöqvist, E. Geometry along evolution of mixed quantum states. Phys. Rev. Res. 2020, 2, 013344. [CrossRef]
103. Briët, J.; Harremoës, P. Properties of classical and quantum Jensen-Shannon divergence. Phys. Rev. A 2009, 79, 052311. [CrossRef]
104. Casas, M.; Lambertim, P.; Lamberti, P.; Plastino, A.; Plastino, A.R. Jensen-Shannon divergence, Fisher information, and Wootters’
hypothesis. arXiv 2004, arXiv:quant-ph/0407147.
105. Sánchez-Moreno, P.; Zarzo, A.; Dehesa, J.S. Jensen divergence based on Fisher’s information. J. Phys. A Math. Theor. 2012,
45, 125305. [CrossRef]
106. López-Ruiz, L.; Mancini, H.; Calbet, X. A statistical measure of complexity. Phys. Lett. A 1995, 209, 321–326. [CrossRef]
107. Cafaro, C.; Alsing, P.M. Information geometry aspects of minimum entropy production paths from quantum mechanical
evolutions. Phys. Rev. E 2020, 101, 022110. [CrossRef]
108. Ashida, K.; Oka, K. Stochastic thermodynamic limit on E. coli adaptation by information geometric approach. Biochem. Biophys.
Res. Commun. 2019, 508, 690–694. [CrossRef]
109. Risken, H. The Fokker-Planck Equation: Methods of Solution and Applications; Springer: Berlin, Germany, 1996.
Entropy 2021, 23, 1393 24 of 24

110. Van Den Brock, C. On the relation between white shot noise, Gaussian white noise, and the dichotomic Markov process. J. Stat.
Phys. 1983, 31, 467–483. [CrossRef]
111. Bena, I. Dichotomous Markov Noise: Exact results for out-of-equilibrium systems (a brief overview). Int. J. Mod. Phys. B 2006, 20,
2825–2888. [CrossRef]
112. Onsager, L.; Machlup, S. Fluctuations and Irreversible Processes. Phys. Rev. 2951, 91, 1505–1512. [CrossRef]
113. Parrondo, J.M.R.; de Cisneros, B.J.; Brito R. Thermodynamics of Isothermal Brownian Motors. In Stochastic Processes in Physics,
Chemistry, and Biology; Freund, J.A., Pöschel, T., Eds.; Lecture Notes in Physics; Springer: Berlin/Heidelberg, Germany, 2000;
Volume 557. [CrossRef]
114. Gaveau, B.; Granger, L.; Moreau, M.; Schulman, L. Dissipation, interaction, and relative entropy. Phys. Rev. E 2014, 89, 032107.
[CrossRef] [PubMed]
115. Ignacio A.; Martínez, G.B.; Jordan M.H.; Juan M.R.P. Inferring broken detailed balance in the absence of observable currents. Nat.
Commun. 2019, 10, 3542.
116. Roldán, É.; Barral, J.; Martin, P.; Parrondo, J.M.R.; Jülicher, F. Quantifying entropy production in active fluctuations of the hair-cell
bundle from time irreversibility and uncertainty relations. New J. Phys. 2021, 23, 083013. [CrossRef]
117. Chevallier, E.; Kalunga, E.; Angulo, J. Kernel Density Estimation on Spaces of Gaussian Distributions and Symmetric Positive
Definite Matrices. 2015. Aavailable online: hal.archives-ouvertes.fr/hal-01245712 (accessed on 29 September 2021).
118. Nicolis, G.; Prigogine, I. Self-Organization in Nonequilibrium Systems: From Dissipative Structures to Order through Fluctuations; John
Wiley and Son: New York, NY, USA, 1977.
119. Prigogine, I. Time, structure, and fluctuations. Science 1978, 201, 777–785. [CrossRef]
120. Jaynes, E.T. The Minimum Entropy Production Principle. Ann. Rev. Phys. Chem. 1980, 31, 579–601. [CrossRef]
121. Mehdi, N. On the Evidence of Thermodynamic Self-Organization during Fatigue: A Review. Entropy 2020, 22, 372.
122. Dewar, R.C. Information theoretic explanation of maximum entropy production, the fluctuation theorem and self-organized
criticality in non-equilibrium stationary states. J. Phys. A. Math. Gen. 2003, 36, 631–641. [CrossRef]
123. Sekhar, J.A. Self-Organization, Entropy Generation Rate, and Boundary Defects: A Control Volume Approach. Entropy 2021, 23,
1092. [CrossRef] [PubMed]
124. Philipp, S.; Thomas, F.; Ray, D.; Friston, K.J. Exploration, novelty, surprise, and free energy minimization. Front. Psychol. 2013,
4, 1–5.
125. Friston, K.J. The free-energy principle: A unified brain theory? Nat. Rev. Neurosci. 2012, 11, 127–138. [CrossRef] [PubMed]