Theory of Computation

Dr. Sarmad Abbasi

Dr. Sarmad Abbasi () Theory of Computation 1 / 33

Lecture 20: Overview

Incompressible strings
Minimal Length Descriptions
Descriptive Complexity

Dr. Sarmad Abbasi () Theory of Computation 2 / 33

Incompressible strings

Recall that a string, x, is b-compressible if

K (x) ≤ |x| − b

1 A string is b-incompressible if it is not b-compressible.

2 A string is incompressible if it is not 1-compressible.

Incompressible strings cannot be compressed. Their description is at least

as long as the string itself.

Dr. Sarmad Abbasi () Theory of Computation 3 / 33

Incompressible strings

1 Incompressible strings look like random strings.

2 This can be made precise in many ways.
Many properties of random strings are also true for incompressible strings.
For example

Dr. Sarmad Abbasi () Theory of Computation 4 / 33

Incompressible strings

We proved two theorems:

Theorem
For every n there is a incompressible string of length n.

Theorem
The number of b-incompressible strings of length b is at least

2n − 2n−b+1 + 1

The proof is simple counting!

Dr. Sarmad Abbasi () Theory of Computation 5 / 33

Incompressible strings

1 Incompressible strings have roughly the same number of 0’s and 1’s.
2 The longest run of 0 is of size O(log n).
Let us prove a theorem which is central in this theory.

Dr. Sarmad Abbasi () Theory of Computation 6 / 33

Incompressible strings

Theorem
A computable property that holds for ”almost all strings” also holds for
incompressible strings.

Let us formalize this statement. A property p is a mapping from set of all

strings to {TRUE , FALSE }. Thus

p : {0, 1}∗ → {true, false}

Dr. Sarmad Abbasi () Theory of Computation 7 / 33

Incompressible strings

We say that p holds for almost all strings if

|{x ∈ {0, 1}n : p(x) = false }|

2n
approaches 0 as n approaches infinity.

Dr. Sarmad Abbasi () Theory of Computation 8 / 33

Incompressible strings

Theorem
Let p be a computable property. If p holds for almost all strings the for
any b > 0, the property p is false on only finitely many strings that are
incompressible by b.

Let M be the following algorithm:

1 On input i (in binary)
2 Find the ith string s such that p(s) = False. Considering the strings
lexicographically.
3 Output s.

Dr. Sarmad Abbasi () Theory of Computation 9 / 33

Incompressible strings

We can use M to obtain short descriptions of string that do not have

property p. Let x be a string that does not have property p. Let ix be the
index of x in the list of all strings that do not have property p ordered
lexicographically. Then hM, ix i is a description of x. The length of this
description is
|ix | + c

Dr. Sarmad Abbasi () Theory of Computation 10 / 33

Incompressible strings

Now we count. For any give b > 0 select n such that at most
1
2b+c
fraction of the strings of length ≤ n do not have property p. Note that
since p holds for almost all strings, this is true for all sufficiently large n.

Dr. Sarmad Abbasi () Theory of Computation 11 / 33

Incompressible strings

Then we have
2n
ix ≤
2b+c
Therefore, |ix | ≤ n − b − c. Hence the length of hMiix is at most
n − b − c + c = n − b. Thus

K (x) ≤ n − b

which means x is b-compressible. Hence, only finitely many strings that

fail p can be incompressible by b.

Dr. Sarmad Abbasi () Theory of Computation 12 / 33

Incompressible strings

We can ask a very interesting question now:

Is it possible to compress a string that is already compressed?

The answer to this question seems to be no! Let us prove something that
is quite close to this.

Dr. Sarmad Abbasi () Theory of Computation 13 / 33

Incompressible strings

Theorem
There is a constant b such that for every string x the minimal description
d(x) is incompressible by b.

Roughly the idea of the proof is that

If d(x) were compressible we could use the compressed version of d(x) to

obtain a smaller description of x than d(x).

Dr. Sarmad Abbasi () Theory of Computation 14 / 33

Incompressible strings

Consider the following TM M:

1 On input hR, y i
2 Run R on y and reject if its output is not of the form hS, zi
3 Run S on z and halt with its output on the tape.
This machine is ”decoding twice”.
Let b = 2|hMi| + 3. We now show that b satisfies the statement of the
theorem. Suppose on the contrary that d(x) is b compressible for some
string. The
|d(d(x))| ≤ |d(x)| − b.
But what happens if we run M on d(d(x)). Note that it outputs

Dr. Sarmad Abbasi () Theory of Computation 15 / 33

Incompressible strings

Thus < M > d(d(x)) is a description of x. How long is this description?

2|hMi| + 2 + |d(d(x))| = (b − 1) + (|d(x) − b) = |d(x)| − 1

This contradicts the fact that d(x) is the minimal description of x.

Dr. Sarmad Abbasi () Theory of Computation 16 / 33

Complexity Theory

Now we will look at resource bounded computations.

Two of the most important resources are going to be

1 Time
2 Space

Dr. Sarmad Abbasi () Theory of Computation 17 / 33

Complexity Theory

Let

A = {0k 1k : k ≥ 0}
Consider the following TM that accepts A
1 Scan across the tape to check if all 0’s are before 1’s if not reject.
2 Scan across the tape crossing of a 0 and a 1 each time.
3 If 0’s remain and 1’s finished reject.
4 if 1’s remain and 0’s are finished reject.
5 if both 0’s and 1’s are finished accept.
6 Goto Step 2
We want to analyze this algorithm and talk about how many steps (time)
does it take.

Dr. Sarmad Abbasi () Theory of Computation 18 / 33

Complexity Theory

A step of the Turing machine is assumed to take unit time. Lets start with
a definition. Let M be a TM that halts on all inputs.

The running time or the time complexity of M is a function

f :N→N

such that f (n) is the maximum number of steps that M uses on any input
of length n.

1 We say that M runs in time f (n).

2 M is f (n)-time TM.

Dr. Sarmad Abbasi () Theory of Computation 19 / 33

Big O notation

We want to understand how much time a TM takes in the worst case.

The exact running time t(n) is going to be very complicated expression. In
some cases we will not be able to compute it.
But what we are interested in is getting a rough idea of how t(n) grows for
large inputs. So we will use asymptotic notation.

Dr. Sarmad Abbasi () Theory of Computation 20 / 33

Big O notation

Lets look at an example let

f (n) = 5n3 + 13n2 − 6n + 48

This function has four terms. He most important term is

5n3

which will be the bulk if n is large.

Dr. Sarmad Abbasi () Theory of Computation 21 / 33

Big O notation

In fact
5n3
the constant 5 does not contribute to the growth of this function. Thus
we will say that f (n) is order n3 . We will denote this by

f (n) = O(n3 ).

Let us state this precisely:

Dr. Sarmad Abbasi () Theory of Computation 22 / 33

Big O notation

Let f , g : N → R + . We say that

f (n) = O(g (n))

if there are positive integers c, n0 such that for all n ≥ n0

f (n) ≤ cg (n)

When f (n) = O(g (n)) we say that g is an asymptotic upper-bound on f .

Sometime we will just say that g is an upper bound on f .

Dr. Sarmad Abbasi () Theory of Computation 23 / 33

Big O notation

Intuitively if f (n) = O(g (n)) you can think that f is less than or equal to
some multiple of g . For most functions the highest order term is is an
obvious candidate to be g .

Dr. Sarmad Abbasi () Theory of Computation 24 / 33

Big O notation

Let
f (n) = 13n4 + 7n2 + 13
Then
f (n) = O(n4 )
Furthermore,
f (n) = O(n5 )
But it is not true that
f (n) = O(n3 )

Dr. Sarmad Abbasi () Theory of Computation 25 / 33

Big O notation

When we say that f (n) = O(g (n)) we are only claiming that g (n) is an
asymptotic upper bound on f . It may not be the best one. Lets look at
other examples.

Dr. Sarmad Abbasi () Theory of Computation 26 / 33

Big O notation

Suppose that
f (n) = logb n
where b is a constant. Note that
log2 n
logb n = .
log2 b
Hence we can say
f (n) = O(log n)
Let
f2 (n) = 5n log n + 200n log log n + 17
then
f2 (n) = O(n log n)

Dr. Sarmad Abbasi () Theory of Computation 27 / 33

Big O notation
We can use O notation in exponents also. Thus

2O(n)

stands for a function that is upper bounded by

2cn

for some constant c. We have to be careful though

For example
2O(log n)
Well remember that O(log n) has a constant attached to it. Thus this is
upper bounded by
2c log n
by some constant c. Now, we get

2c log n = (2log n )c = nc

Dr. Sarmad Abbasi () Theory of Computation 28 / 33

Big O notation

Bounds of the form nc are called polynomial bounds. Where as bounds of

c
the form 2n are called exponential bounds.

Dr. Sarmad Abbasi () Theory of Computation 29 / 33

small o notation

The Big O notation roughly says that f is upper bounded by g . Now we

want to say

f grows strictly slower that g .

Let f , g : N → R + . We say that f (n) = o(g (n)) if

f (n)
lim =0
n→∞ g (n)

Thus the ratio of f and g becomes closer and closer to 0.

Dr. Sarmad Abbasi () Theory of Computation 30 / 33

small o notation

√
1 n = o(n)
2 n = o(n log log n)
3 n log log n = o(n log n)
4 n log n = o(n2 )
5 n2 = o(n3 ).

Dr. Sarmad Abbasi () Theory of Computation 31 / 33

small o notation

Lets look at a new TM that recognizes

A = {0k 1k : k ≥ 0}

1 Scan the tape and make sure all the 0s are before 1s.
2 Repeat till there are 0s and 1’s on the tape.
3 Scan the tape and if the total number of 0’s and 1’s is odd reject.
4 Scan and cross of every other 0 and do the same with every other 1.
5 If all 0’s and 1’s are crossed accept.
Thus if we have 13 0’s in the next stage there would be 6.

Dr. Sarmad Abbasi () Theory of Computation 32 / 33

small o notation

How much time does this TM take?

O(n log n)
Thus the time taken by this TM is o(n2 ). Hence, this reflects a real
algorithmic improvement.

Dr. Sarmad Abbasi () Theory of Computation 33 / 33