Time-universal data compression and prediction

In this section we first consider the following problem: There is a set of data compressors $F=\{\varphi_{1},\varphi_{2},...\}$ and let $x_{1}x_{2}...$ be a sequence of letters from a finite alphabet $A$ whose initial part $x_{1}...x_{n}$ should be compressed by some $\varphi\in F$. Let, as before, $v_{i}$ be the time spent on encoding one letter by the data compressor $\varphi_{i}$ and suppose that all $v_{i}$ are upper-bounded by a certain constant $v$, i.e. $\sup_{i=1,2,...,}v_{i}\,\leq\,v\,.$ (Note, that $v_{i}$ can be unknown beforehand, but $v$ should be known.)

The goal is to find a good data compressor from $F$ in order to compress $x_{1}...x_{n}$ in such a way that the total time spent for all calculations and compressions does not exceed $T(1+\delta),\delta>0$, where $T=v\,n$ is the minimum time that must be reserved for compression and $\delta T$ is an additional time that can be used to find the good compressor (among $\varphi_{1},\varphi_{2},...$). In order to accurately describe the problem, we suppose also that there is a probability distribution $\omega=$ $\omega_{1},\omega_{2},...$ such that all $\omega_{i}>0$. The goal is to fined such $\varphi_{i}$ that the value

is close to minimal. (Here the first part $\lceil-\log\omega_{i}\rceil$ is used for encoding number $i$.) The decoder first finds $i$ and then $x_{1}x_{2}...x_{n}$ using the decoder corresponding $\varphi_{i}$.

Comment 1. Here and below we did not take into account the time required for the calculation of $\log\omega_{i}$ and some other auxiliary calculations. If in a certain situation this time is not negligible, it is possible to reduce $\hat{T}$ in advance by the required value.

Suppose that there is a file $x_{1}x_{2}...x_{n}$ and data compressors $\varphi_{1},...,\varphi_{m}$, $n\geq 1,m\geq 1$. Let, as before, $v_{i}$ be the time spent on encoding one letter by the data compressor $\varphi_{i}$,

and let

The goal is to find the data compressor $\varphi_{j}$, $j=1,...,m$, that compresses the file $x_{1}x_{2}...x_{n}$ in the best way in time $\hat{T}$. Seemingly, the simplest method is as follows:

Step 1 Calculate $r=\lfloor\delta T/v\rfloor$.

Step 2 Compress the file $x_{1}x_{2}...x_{r}$ by $\varphi_{1}$ and find the length of compressed file $|\varphi_{1}(x_{1}...x_{r})|$, then , likewise, find $|\varphi_{2}(x_{1}...x_{r})|$, $|\varphi_{3}(x_{1}...x_{r})|$, etc.

Step 3 Calculate $s=\arg\min_{i=1,...,m}$ $|\varphi_{i}(x_{1}...x_{r})|$

Step 4 Compress the whole file $x_{1}x_{2}...x_{n}$ by $\varphi_{s}$ and compose the codeword $\langle s\rangle$ $\varphi_{s}(x_{1}...x_{n})$, where $\langle s\rangle$ is $\lceil\log m\rceil$-bit word with the presentation of $s$.

The decoding is obvious. Denote this method by $\Phi^{\delta}_{1}$.

Comment 2. We considered the case of data compression. It is possible to apply the described method for time-universal prediction. In this case one should calculate $\pi_{i}(x_{1}...x_{r})$ instead of $|\varphi_{i}(x_{1}...x_{r})|$ and the third step should be changed as follows:

Calculate $s=\arg\max_{i=1,...,m}$ $\pi_{i}(x_{1}...x_{r}).$

The asymptotic properties of the method $\Phi_{1}^{\delta}$ are as follows:

Claim 1. Let there be an infinite sequence $x_{1},x_{2},...$ and data compressors $\varphi_{1},...,\varphi_{m}$ such that there exist the following limits

for all $i=1,...,m$. Then, for any $\delta>0$

i.e. $\Phi^{\delta}_{1}$ is time-universal.

Next we describe a more general method, for which Claim 1 is a special case.

Generally speaking, it is possible to offer many reasonable strategies for finding the optimal data compressor (or predictor) for a given time. For the finite set of data-compressors such a strategy can be as follows: try all the compressors on a (very) short sequence $x_{1}x_{2}...x_{k}$ and choose a few of the best ones. Then try those chosen data-compressors on a larger sequence $x_{1}x_{2}...x_{l}$, $k<l$, and choose the best which will be used for compression of the whole sequence $x_{1}x_{2}...x_{n}$. Another reasonable strategy can be based on maximization of the probability to determine the optimal data compressor as a function of the extra time $\delta\,T$ and other parameters.

So, we can see that there are a lot of reasonable strategies and each of them has a lot of parameters. That is why, it could be useful to use multidimensional optimization methods, such as machine learning, so-called deep learning, etc. Since this is the first paper devoted to time-adaptive and time-universal data compression and prediction, we consider only some general conditions needed for time-universality.

For a time-adaptive data-compressor $\hat{\Phi}$ and $x_{1}...x_{t}$ we define for any $\varphi_{i}$

A proof is given in Appendix 1, but here we note that Claim 1 is a particular case of this theorem.

Comment 3. If the sequence $x_{1}x_{2}...$ is generated by a stationary source and all $\varphi_{i}$ are universal codes, the property iii) is valid with probability 1 (See, for example, [19]). Hence, this theorem (and the claim 1) are valid for this case.

In general, the property iii) shows that the sequence under consideration has some stability. In turn, it gives a possibility to estimate characteristics of the whole sequence $x_{1}x_{2}...$ based on its initial part.

In this subsection we describe a time-universal code (and the corresponding predictor) for stationary sources. It is based on optimal universal codes for Markov chains, developed by Krichevsky [20, 21] and the twice-universal code [12]. Denote by $M_{i}$, $i=1,2,...$ the set of Markov chains with memory (connectivity) $i$, and let $M_{0}$ be the set of Bernoulli sources. For stationary ergodic $\mu$ and an integer $r$ we denote by $h_{r}(\mu)$ the $r$-order entropy (per letter) and let $h_{\infty}(\mu)$ be the limit entropy; see for definitions [19].

Krichevsky [20, 21] described the codes $\psi_{0},\psi_{1},...$ which are asymptotically optimal for $M_{0},M_{1},...$, correspondingly. If the sequence $x_{1}x_{2}...x_{n}$ , $x_{i}\in A$, is generated by a source $\mu$ $\in M_{i}$, the following inequalities are valid almost surely (a.s.):

where $t$ grows. (Here $C$ is a constant.) The length of a codeword of the twice-universal code $\rho$ is defined as the following ”mixture”:

(It is well-known in Information Theory [19] that there exists a code with such codeword lengths, because $\sum_{x_{1}...x_{t}\in A^{t}}$ $2^{-|\rho(x_{1}...x_{t})|}$ $=1$.) This code is called twice-universal because for any $M_{i}$, $i=0,1,...$, and $\mu\in M_{i}$ the equality (10) is valid (with different $C$). Besides, for any stationary ergodic source $\mu$ a.s.

Let us estimate the time of calculations necessary when using $\rho$. First, note that it suffices to sum a finite number of terms in (11), because all the terms $2^{-|\psi_{i}(x_{1}...x_{t})|}$ are equal for $i\geq t$. On the other hand, the number of different terms grows, where $t\to\infty$ and, hence, the encoder should calculate $2^{-|\psi_{i}(x_{1}...x_{t})|}$ for growing number $i$’s. It is known [12] that the time spent for encoding one letter is close for different codes $\psi_{i}$. Hence, the time spent for encoding one letter by the code $\rho$ grows to infinity, when $t$ grows. The described below time-universal code $\Psi^{\delta}$ has the same asymptotic performance, but the time spent for encoding one letter is a constant.

In order to describe the time-universal code $\Psi^{\delta}$ we give some definitions. Let, as before, $v$ be an upper-bound of the time spent for encoding one letter by any $\psi_{i}$, $x_{1}...x_{t}$ be the generated word,

Denote by $\Psi^{\delta}$ the following method:

Step 1 Calculate $m(t),s(t)$ and

Step 2 Find such a $j$ that

Step 3 Calculate the codeword $\psi_{j}(x_{1}...x_{t})$ and output

where $<j>$ is the $\lceil-\log\omega_{j+1}\rceil$-bit codeword of $j$.

The decoding is obvious.

A proof is given in Appendix 1.

Here we note some possible generalisations. We consider mainly the case of off-line prediction and data compression, where the whole sequence $x_{1}...x_{n}$ can be investigated in order to find a suitable data-compressor or predictor. There are situations where the forecast should be made step-by-step, i.e. $x_{i+1}$ should be predicted based on $x_{1}...x_{t}$, $x_{i+2}$ should be predicted based on $x_{1}...x_{i+1}$ and so on. The suggested approach and methods can be naturally extended to this case, too, if we take into account the possibility to store results of calculations done on previous steps.

Another generalization is connected with the need to know in advance the speed of computing the forecast (or data compression). In such a case the goal of time-universal method can be the same: it should limit the time of calculation by $T(1+\delta)$, where $T$ is (unknown beforehand) the time of the optimal method (from a given set). In such a case the speeds can be evaluated during the calculation.

Proof of Theorem 1. Define $\lambda_{i}=\lim_{t\to\infty}|\varphi_{i}(x_{1}...x_{t})|/t$, and let

Let $\epsilon$ be any positive number. Having taken into account that the set $F$ is finite, from these definitions we can see that there exists such $t_{1}$ that

Taking into account the property i), we can see that there exists such a number $t_{2}$ for which $\tau_{i}(t)$ is defined for all $\varphi_{i}$ and $t>t_{2}$, and denote $t_{3}$ $=\max\{t_{1},t_{2}\}$. Take any $n>t_{3}$ and suppose that a data-compressor $\varphi_{s}$ was chosen, when $\hat{\Phi}$ was applied to $x_{1}x_{2}...x_{n}$. Hence, from the property ii) we can see that there exists $t_{4}>t_{3}$, such that

From (15) we obtain the following two inequalities

Having taken into account (17) we can see from the two latest inequalities that $\lambda_{s}-\epsilon<\lambda_{o}+\epsilon$ and, hence, $\lambda_{s}<\lambda_{o}+2\epsilon$. Taking into account, that, by definition (15), $\lambda_{o}<\lambda_{s}$, we obtain

Since $n>t_{1}$, we can see from (16) that

Taking into account that $(-\log\omega_{s}+|\varphi_{s}(x_{1}...x_{n})|)/n$ $=\hat{\Phi}(x_{1}...x_{n})/n$ we obtain from (18) that

and, hence,

It is true for any $\epsilon>0$, hence, $\lim_{n\to\infty}\hat{\Phi}(x_{1}...x_{n})/n=\lambda_{o}$. The theorem is proven.

Proof of Theorem 2. It is known in Information Theory [19] that $h_{r}(\mu)$ $\geq h_{r+1}(\mu)$ $\geq h_{\infty}(\mu)$ for any $r$ and (by definition) $\lim_{r\to\infty}h_{r}(\mu)$ $=h_{\infty}(\mu)$. Let $\epsilon>0$ and $r$ be such an integer that $h_{r}-h_{\infty}$ $<\epsilon$. From (V) we can see that there exists such $t_{1}$ that $m(t)\geq r$ if $t\geq t_{1}$. Taking into account (10) and (V), we can see that there exists $t_{2}$ for which a.s. $||\psi_{r}(x_{1}...x_{t})|/t-h_{r}(\mu)|$ $<\epsilon$ if $t>t_{2}$. From the description of $\Psi^{\delta}$ (the step 3) we can see that there exists such $t_{3}>\max\{t_{1},t_{2}\}$ for which a.s.

if $t>t_{3}$. By definition,

Having taken into account that $\epsilon$ is an arbitrary number and two latest inequalities as well as the fact that a.s. $\inf_{i=0,1,...}\lim_{t\to\infty}|\psi_{r}(x_{1}...x_{t})|/t$ $=h_{\infty}(\mu)$, we obtain (14). The theorem is proven.

Let there be a source with unknown statistics which generates sequences $x_{1}x_{2}\cdots$ of letters from some finite alphabet $A$. Let the source generate a message $x_{1}\ldots x_{t-1}x_{t}$, $\,x_{i}\in A$, and the following letter $x_{t+1}$ needs to be predicted. This problem can be traced back to Laplace who considered the problem of estimation of the probability that the sun will rise tomorrow, given that it has risen every day since Creation [22]. In our notation the alphabet $A$ contains two letters, $0\;$ (“the  sun  rises”) and $1\;$ (“the  sun  does  not  rise” ); $t$ is the number of days since Creation, $x_{1}\ldots x_{t-1}x_{t}=00\ldots 0.$

Laplace suggested the following predictor:

where $\nu_{x_{1}\cdots x_{t}}(a)$ denotes the count of letter $a$ occurring in the word $x_{1}\ldots x_{t-1}x_{t}.$ For example, if $A=\{0,1\},$ $\>x_{1}...x_{5}$ $=01010$, then the Laplace prediction is as follows: $L_{0}(x_{6}=0|x_{1}...x_{5}=01010)$ $=(3+1)/(5+2)$ $=4/7,$ $L_{0}(x_{6}=1|x_{1}...x_{5}=$ $(2+1)/(5+2)=3/7.$ In other words, $3/7$ and $4/7$ are estimations of the unknown probabilities $P(x_{t+1}=0|x_{1}\ldots x_{t}=01010)$ and $P(x_{t+1}$ $=$ $1|x_{1}\ldots x_{t}$ $=$ $01010).$ (In what follows we will use the shorter notation: $P(0|01010)$ and $P(1|01010)).$ We can see that Laplace considered prediction as a set of estimations of unknown (conditional) probabilities, because they contain all information about the future behaviour of any stochastic process. In general, we call as a predictor $\pi$ any conditional probabilities $\pi(x_{i+1}=a|$ $x_{1}=a_{1},...,x_{n}=a_{n})$ defined for all integers $n$, $a\in A$, $a_{1},...,a_{n}$ $\in A^{n}$.

Proximity of the theory of universal coding and prediction, as well as asymptotically optimal methods of prediction in a framework of the Laplace approach were found for the cases of a finite alphabet and continues one in [13] and [23], correspondingly.

This work was supported by Russian Foundation for Basic Research (grant 15-07-01851).