Universality and uncomputability

“A function of a variable quantity is an analytic expression composed in any way whatsoever of the variable quantity and numbers or constant quantities.”, Leonhard Euler, 1748.

“The importance of the universal machine is clear. We do not need to have an infinity of different machines doing different jobs. … The engineering problem of producing various machines for various jobs is replaced by the office work of ‘programming’ the universal machine”, Alan Turing, 1948

One of the most significant results we showed for Boolean circuits (or equivalently, straight-line programs) is the notion of universality: there is a single circuit that can evaluate all other circuits. However, this result came with a significant caveat. To evaluate a circuit of \(s\) gates, the universal circuit needed to use a number of gates larger than \(s\). It turns out that uniform models such as Turing machines or NAND-TM programs allow us to “break out of this cycle” and obtain a truly universal Turing machine \(U\) that can evaluate all other machines, including machines that are more complex (e.g., more states) than \(U\) itself. (Similarly, there is a Universal NAND-TM program \(U'\) that can evaluate all NAND-TM programs, including programs that have more lines than \(U'\).)

It is no exaggeration to say that the existence of such a universal program/machine underlies the information technology revolution that began in the latter half of the 20th century (and is still ongoing). Up to that point in history, people have produced various special-purpose calculating devices such as the abacus, the slide ruler, and machines that compute various trigonometric series. But as Turing (who was perhaps the one to see most clearly the ramifications of universality) observed, a general purpose computer is much more powerful. Once we build a device that can compute the single universal function, we have the ability, via software, to extend it to do arbitrary computations. For example, if we want to simulate a new Turing machine \(M\), we do not need to build a new physical machine, but rather can represent \(M\) as a string (i.e., using code) and then input \(M\) to the universal machine \(U\).

Beyond the practical applications, the existence of a universal algorithm also has surprising theoretical ramifications, and in particular can be used to show the existence of uncomputable functions, upending the intuitions of mathematicians over the centuries from Euler to Hilbert. In this chapter we will prove the existence of the universal program, and also show its implications for uncomputability, see Figure 9.1

Universality or a meta-circular evaluator

We start by proving the existence of a universal Turing machine. This is a single Turing machine \(U\) that can evaluate arbitrary Turing machines \(M\) on arbitrary inputs \(x\), including machines \(M\) that can have more states and larger alphabet than \(U\) itself. In particular, \(U\) can even be used to evaluate itself! This notion of self reference will appear time and again in this book, and as we will see, leads to several counter-intuitive phenomena in computing.

Proving the existence of a universal Turing Machine

To prove (and even properly state) Theorem 9.1, we need to fix some representation for Turing machines as strings. One potential choice for such a representation is to use the equivalence between Turing machines and NAND-TM programs and hence represent a Turing machine \(M\) using the ASCII encoding of the source code of the corresponding NAND-TM program \(P\). However, we will use a more direct encoding.

Using this representation, we can formally prove Theorem 9.1.

# constants
def EVAL(δ,x):
    '''Evaluate TM given by transition table δ
    on input x'''
    Tape = ["▷"] + [a for a in x]
    i = 0; s = 0 # i = head pos, s = state
    while True:
        s, Tape[i], d = δ[(s,Tape[i])]
        if d == "H": break
        if d == "L": i = max(i-1,0)
        if d == "R": i += 1
        if i>= len(Tape): Tape.append('Φ')

    j = 1; Y = [] # produce output
    while Tape[j] != 'Φ':
        Y.append(Tape[j])
        j += 1
    return Y

The construction above yields a universal Turing machine with a very large number of states. However, since universal Turing machines have such a philosophical and technical importance, researchers have attempted to find the smallest possible universal Turing machines, see Section 9.7.

Implications of universality (discussion)

There is more than one Turing machine \(U\) that satisfies the conditions of Theorem 9.1, but the existence of even a single such machine is already extremely fundamental to both the theory and practice of computer science. Theorem 9.1’s impact reaches beyond the particular model of Turing machines. Because we can simulate every Turing machine by a NAND-TM program and vice versa, Theorem 9.1 immediately implies there exists a universal NAND-TM program \(P_U\) such that \(P_U(P,x)=P(x)\) for every NAND-TM program \(P\). We can also “mix and match” models. For example since we can simulate every NAND-RAM program by a Turing machine, and every Turing machine by the \(\lambda\) calculus, Theorem 9.1 implies that there exists a \(\lambda\) expression \(e\) such that for every NAND-RAM program \(P\) and input \(x\) on which \(P(x)=y\), if we encode \((P,x)\) as a \(\lambda\)-expression \(f\) (using the \(\lambda\)-calculus encoding of strings as lists of \(0\)’s and \(1\)’s) then \((e\; f)\) evaluates to an encoding of \(y\). More generally we can say that for every \(\mathcal{X}\) and \(\mathcal{Y}\) in the set \(\{\) Turing machines, RAM Machines, NAND-TM, NAND-RAM, \(\lambda\)-calculus, JavaScript, Python, \(\ldots\) \(\}\) of Turing equivalent models, there exists a program/machine in \(\mathcal{X}\) that computes the map \((P,x) \mapsto P(x)\) for every program/machine \(P \in \mathcal{Y}\).

The idea of a “universal program” is of course not limited to theory. For example compilers for programming languages are often used to compile themselves, as well as programs more complicated than the compiler. (An extreme example of this is Fabrice Bellard’s Obfuscated Tiny C Compiler which is a C program of 2048 bytes that can compile a large subset of the C programming language, and in particular can compile itself.) This is also related to the fact that it is possible to write a program that can print its own source code, see Figure 9.3. There are universal Turing machines known that require a very small number of states or alphabet symbols, and in particular there is a universal Turing machine (with respect to a particular choice of representing Turing machines as strings) whose tape alphabet is \(\{ \triangleright, \varnothing, 0, 1 \}\) and has fewer than \(25\) states (see Section 9.7).

Is every function computable?

In Theorem 4.12, we saw that NAND-CIRC programs can compute every finite function \(f:\{0,1\}^n \rightarrow \{0,1\}\). Therefore a natural guess is that NAND-TM programs (or equivalently, Turing machines) could compute every infinite function \(F:\{0,1\}^* \rightarrow \{0,1\}\). However, this turns out to be false. That is, there exists a function \(F:\{0,1\}^* \rightarrow \{0,1\}\) that is uncomputable!

The existence of uncomputable functions is quite surprising. Our intuitive notion of a “function” (and the notion most mathematicians had until the 20th century) is that a function \(f\) defines some implicit or explicit way of computing the output \(f(x)\) from the input \(x\). The notion of an “uncomputable function” thus seems to be a contradiction in terms, but yet the following theorem shows that such creatures do exist:

The type of argument used to prove Theorem 9.5 is known as diagonalization since it can be described as defining a function based on the diagonal entries of a table as in Figure 9.4. The proof can be thought of as an infinite version of the counting argument we used for showing lower bound for NAND-CIRC programs in Theorem 5.3. Namely, we show that it’s not possible to compute all functions from \(\{0,1\}^* \rightarrow \{0,1\}\) by Turing machines simply because there are more functions like that than there are Turing machines.

As mentioned in Remark 7.4, many texts use the “language” terminology and so will call a set \(L \subseteq \{0,1\}^*\) an undecidable or non-recursive language if the function \(F:\{0,1\}^* \rightarrow \{0,1\}\) such that \(F(x)=1 \leftrightarrow x\in L\) is uncomputable.

The Halting problem

Theorem 9.5 shows that there is some function that cannot be computed. But is this function the equivalent of the “tree that falls in the forest with no one hearing it”? That is, perhaps it is a function that no one actually wants to compute. It turns out that there are natural uncomputable functions:

Before turning to prove Theorem 9.6, we note that \(\ensuremath{\mathit{HALT}}\) is a very natural function to want to compute. For example, one can think of \(\ensuremath{\mathit{HALT}}\) as a special case of the task of managing an “App store”. That is, given the code of some application, the gatekeeper for the store needs to decide if this code is safe enough to allow in the store or not. At a minimum, it seems that we should verify that the code would not go into an infinite loop.

Is the Halting problem really hard? (discussion)

Many people’s first instinct when they see the proof of Theorem 9.6 is to not believe it. That is, most people do believe the mathematical statement, but intuitively it doesn’t seem that the Halting problem is really that hard. After all, being uncomputable only means that \(\ensuremath{\mathit{HALT}}\) cannot be computed by a Turing machine.

But programmers seem to solve \(\ensuremath{\mathit{HALT}}\) all the time by informally or formally arguing that their programs halt. It’s true that their programs are written in C or Python, as opposed to Turing machines, but that makes no difference: we can easily translate back and forth between this model and any other programming language.

While every programmer encounters at some point an infinite loop, is there really no way to solve the halting problem? Some people argue that they personally can, if they think hard enough, determine whether any concrete program that they are given will halt or not. Some have even argued that humans in general have the ability to do that, and hence humans have inherently superior intelligence to computers or anything else modeled by Turing machines.¹

The best answer we have so far is that there truly is no way to solve \(\ensuremath{\mathit{HALT}}\), whether using Macs, PCs, quantum computers, humans, or any other combination of electronic, mechanical, and biological devices. Indeed this assertion is the content of the Church-Turing Thesis. This of course does not mean that for every possible program \(P\), it is hard to decide if \(P\) enters an infinite loop. Some programs don’t even have loops at all (and hence trivially halt), and there are many other far less trivial examples of programs that we can certify to never enter an infinite loop (or programs that we know for sure that will enter such a loop). However, there is no general procedure that would determine for an arbitrary program \(P\) whether it halts or not. Moreover, there are some very simple programs for which no one knows whether they halt or not. For example, the following Python program will halt if and only if Goldbach’s conjecture is false:

def isprime(p):
    return all(p % i for i in range(2,p-1))

def Goldbach(n):
    return any( (isprime(p) and isprime(n-p))
           for p in range(2,n-1))

n = 4
while True:
    if not Goldbach(n): break
    n+= 2

Given that Goldbach’s Conjecture has been open since 1742, it is unclear that humans have any magical ability to say whether this (or other similar programs) will halt or not.

A direct proof of the uncomputability of \(\ensuremath{\mathit{HALT}}\) (optional)

It turns out that we can combine the ideas of the proofs of Theorem 9.5 and Theorem 9.6 to obtain a short proof of the latter theorem, that does not appeal to the uncomputability of \(F^*\). This short proof appeared in print in a 1965 letter to the editor of Christopher Strachey:

To the Editor, The Computer Journal.

An Impossible Program

Sir,

A well-known piece of folk-lore among programmers holds that it is impossible to write a program which can examine any other program and tell, in every case, if it will terminate or get into a closed loop when it is run. I have never actually seen a proof of this in print, and though Alan Turing once gave me a verbal proof (in a railway carriage on the way to a Conference at the NPL in 1953), I unfortunately and promptly forgot the details. This left me with an uneasy feeling that the proof must be long or complicated, but in fact it is so short and simple that it may be of interest to casual readers. The version below uses CPL, but not in any essential way.

Suppose T[R] is a Boolean function taking a routine (or program) R with no formal or free variables as its arguments and that for all R, T[R] = True if R terminates if run and that T[R] = False if R does not terminate.

Consider the routine P defined as follows

rec routine P
§L: if T[P] go to L
Return §

If T[P] = True the routine P will loop, and it will only terminate if T[P] = False. In each case T[P] has exactly the wrong value, and this contradiction shows that the function T cannot exist.

Yours faithfully,
C. Strachey

Churchill College, Cambridge

Since CPL is not as common today, let us reproduce this proof. The idea is the following: suppose for the sake of contradiction that there exists a program T such that T(f,x) equals True iff f halts on input x. (Strachey’s letter considers the no-input variant of \(\ensuremath{\mathit{HALT}}\), but as we’ll see, this is an immaterial distinction.) Then we can construct a program P and an input x such that T(P,x) gives the wrong answer. The idea is that on input x, the program P will do the following: run T(x,x), and if the answer is True then go into an infinite loop, and otherwise halt. Now you can see that T(P,P) will give the wrong answer: if P halts when it gets its own code as input, then T(P,P) is supposed to be True, but then P(P) will go into an infinite loop. And if P does not halt, then T(P,P) is supposed to be False but then P(P) will halt. We can also code this up in Python:

def CantSolveMe(T):
    """
    Gets function T that claims to solve HALT.
    Returns a pair (P,x) of code and input on which
    T(P,x) ≠ HALT(x)
    """
    def fool(x):
        if T(x,x):
            while True: pass
        return "I halted"

    return (fool,fool)

For example, consider the following Naive Python program T that guesses that a given function does not halt if its input contains while or for

def T(f,x):
    """Crude halting tester - decides it doesn't halt if it contains a loop."""
    import inspect
    source = inspect.getsource(f)
    if source.find("while"): return False
    if source.find("for"): return False
    return True

If we now set (f,x) = CantSolveMe(T), then T(f,x)=False but f(x) does in fact halt. This is of course not specific to this particular T: for every program T, if we run (f,x) = CantSolveMe(T) then we’ll get an input on which T gives the wrong answer to \(\ensuremath{\mathit{HALT}}\).

Reductions

The Halting problem turns out to be a linchpin of uncomputability, in the sense that Theorem 9.6 has been used to show the uncomputability of a great many interesting functions. We will see several examples of such results in this chapter and the exercises, but there are many more such results (see Figure 9.6).

The idea behind such uncomputability results is conceptually simple but can at first be quite confusing. If we know that \(\ensuremath{\mathit{HALT}}\) is uncomputable, and we want to show that some other function \(\ensuremath{\mathit{BLAH}}\) is uncomputable, then we can do so via a contrapositive argument (i.e., proof by contradiction). That is, we show that if there exists a Turing machine that computes \(\ensuremath{\mathit{BLAH}}\) then there exists a Turing machine that computes \(\ensuremath{\mathit{HALT}}\). (Indeed, this is exactly how we showed that \(\ensuremath{\mathit{HALT}}\) itself is uncomputable, by deriving this fact from the uncomputability of the function \(F^*\) of Theorem 9.5.)

For example, to prove that \(\ensuremath{\mathit{BLAH}}\) is uncomputable, we could show that there is a computable function \(R:\{0,1\}^* \rightarrow \{0,1\}^*\) such that for every pair \(M\) and \(x\), \(\ensuremath{\mathit{HALT}}(M,x)=\ensuremath{\mathit{BLAH}}(R(M,x))\). The existence of such a function \(R\) implies that if \(\ensuremath{\mathit{BLAH}}\) was computable then \(\ensuremath{\mathit{HALT}}\) would be computable as well, hence leading to a contradiction! The confusing part about reductions is that we are assuming something we believe is false (that \(\ensuremath{\mathit{BLAH}}\) has an algorithm) to derive something that we know is false (that \(\ensuremath{\mathit{HALT}}\) has an algorithm). Michael Sipser describes such results as having the form “If pigs could whistle then horses could fly”.

A reduction-based proof has two components. For starters, since we need \(R\) to be computable, we should describe the algorithm to compute it. The algorithm to compute \(R\) is known as a reduction since the transformation \(R\) modifies an input to \(\ensuremath{\mathit{HALT}}\) to an input to \(\ensuremath{\mathit{BLAH}}\), and hence reduces the task of computing \(\ensuremath{\mathit{HALT}}\) to the task of computing \(\ensuremath{\mathit{BLAH}}\). The second component of a reduction-based proof is the analysis of the algorithm \(R\): namely a proof that \(R\) does indeed satisfy the desired properties.

Reduction-based proofs are just like other proofs by contradiction, but the fact that they involve hypothetical algorithms that don’t really exist tends to make reductions quite confusing. The one silver lining is that at the end of the day the notion of reductions is mathematically quite simple, and so it’s not that bad even if you have to go back to first principles every time you need to remember what is the direction that a reduction should go in.

Example: Halting on the zero problem

Here is a concrete example for a proof by reduction. We define the function \(\ensuremath{\mathit{HALTONZERO}}:\{0,1\}^* \rightarrow \{0,1\}\) as follows. Given any string \(M\), \(\ensuremath{\mathit{HALTONZERO}}(M)=1\) if and only if \(M\) describes a Turing machine that halts when it is given the string \(0\) as input. A priori \(\ensuremath{\mathit{HALTONZERO}}\) seems like a potentially easier function to compute than the full-fledged \(\ensuremath{\mathit{HALT}}\) function, and so we could perhaps hope that it is not uncomputable. Alas, the following theorem shows that this is not the case:

def N(z):
    M = r'.......'
    # a string constant containing desc. of M
    x = r'.......'
    # a string constant containing x
    return eval(M,x)
    # note that we ignore the input z

Rice’s Theorem and the impossibility of general software verification

The uncomputability of the Halting problem turns out to be a special case of a much more general phenomenon. Namely, that we cannot certify semantic properties of general purpose programs. “Semantic properties” mean properties of the function that the program computes, as opposed to properties that depend on the particular syntax used by the program.

An example for a semantic property of a program \(P\) is the property that whenever \(P\) is given an input string with an even number of \(1\)’s, it outputs \(0\). Another example is the property that \(P\) will always halt whenever the input ends with a \(1\). In contrast, the property that a C program contains a comment before every function declaration is not a semantic property, since it depends on the actual source code as opposed to the input/output relation.

Checking semantic properties of programs is of great interest, as it corresponds to checking whether a program conforms to a specification. Alas it turns out that such properties are in general uncomputable. We have already seen some examples of uncomputable semantic functions, namely \(\ensuremath{\mathit{HALT}}\) and \(\ensuremath{\mathit{HALTONZERO}}\), but these are just the “tip of the iceberg”. We start by observing one more such example:

Another result along similar lines is the following:

Rice’s Theorem

Theorem 9.13 can be generalized far beyond the parity function. In fact, this generalization rules out verifying any type of semantic specification on programs. We define a semantic specification on programs to be some property that does not depend on the code of the program but just on the function that the program computes.

For example, consider the following two C programs

int First(int n) {
    if (n<0) return 0;
    return 2*n;
}

int Second(int n) {
    int i = 0;
    int j = 0
    if (n<0) return 0;
    while (j<n) {
        i = i + 2;
        j = j + 1;
    }
    return i;
}

First and Second are two distinct C programs, but they compute the same function. A semantic property, would be either true for both programs or false for both programs, since it depends on the function the programs compute and not on their code. An example for a semantic property that both First and Second satisfy is the following: “The program \(P\) computes a function \(f\) mapping integers to integers satisfying that \(f(n) \geq n\) for every input \(n\)”.

A property is not semantic if it depends on the source code rather than the input/output behavior. For example, properties such as “the program contains the variable k” or “the program uses the while operation” are not semantic. Such properties can be true for one of the programs and false for others. Formally, we define semantic properties as follows:

There are two trivial examples of semantic functions: the constant one function and the constant zero function. For example, if \(Z\) is the constant zero function (i.e., \(Z(M)=0\) for every \(M\)) then clearly \(F(M)=F(M')\) for every pair of Turing machines \(M\) and \(M'\) that are functionally equivalent \(M\) and \(M'\). Here is a non-trivial example

Often the properties of programs that we are most interested in computing are the semantic ones, since we want to understand the programs’ functionality. Unfortunately, Rice’s Theorem tells us that these properties are all uncomputable:

Yale[0] = NAND(X[0],X[0])
Y[0] = NAND(X[0],Yale[0])

Harvard[0] = NAND(X[0],X[0])
Y[0] = NAND(X[0],Harvard[0])

Halting and Rice’s Theorem for other Turing-complete models

As we saw before, many natural computational models turn out to be equivalent to one another, in the sense that we can transform a “program” of one model (such as a \(\lambda\) expression, or a game-of-life configurations) into another model (such as a NAND-TM program). This equivalence implies that we can translate the uncomputability of the Halting problem for NAND-TM programs into uncomputability for Halting in other models. For example:

The same proof carries over to other computational models such as the \(\lambda\) calculus, two dimensional (or even one-dimensional) automata etc. Hence for example, there is no algorithm to decide if a \(\lambda\) expression evaluates the identity function, and no algorithm to decide whether an initial configuration of the game of life will result in eventually coloring the cell \((0,0)\) black or not.

Indeed, we can generalize Rice’s Theorem to all these models. For example, if \(F:\{0,1\}^* \rightarrow \{0,1\}\) is a non-trivial function such that \(F(P)=F(P')\) for every functionally equivalent NAND-TM programs \(P,P'\) then \(F\) is uncomputable, and the same holds for NAND-RAM programs, \(\lambda\)-expressions, and all other Turing complete models (as defined in Definition 8.5), see also Exercise 9.12.

Is software verification doomed? (discussion)

Programs are increasingly being used for mission critical purposes, whether it’s running our banking system, flying planes, or monitoring nuclear reactors. If we can’t even give a certification algorithm that a program correctly computes the parity function, how can we ever be assured that a program does what it is supposed to do? The key insight is that while it is impossible to certify that a general program conforms with a specification, it is possible to write a program in the first place in a way that will make it easier to certify. As a trivial example, if you write a program without loops, then you can certify that it halts. Also, while it might not be possible to certify that an arbitrary program computes the parity function, it is quite possible to write a particular program \(P\) for which we can mathematically prove that \(P\) computes the parity. In fact, writing programs or algorithms and providing proofs for their correctness is what we do all the time in algorithms research.

The field of software verification is concerned with verifying that given programs satisfy certain conditions. These conditions can be that the program computes a certain function, that it never writes into a dangerous memory location, that is respects certain invariants, and others. While the general tasks of verifying this may be uncomputable, researchers have managed to do so for many interesting cases, especially if the program is written in the first place in a formalism or programming language that makes verification easier. That said, verification, especially of large and complex programs, remains a highly challenging task in practice as well, and the number of programs that have been formally proven correct is still quite small. Moreover, even phrasing the right theorem to prove (i.e., the specification) is often a highly non-trivial endeavor.

Exercises

Bibliographical notes

The cartoon of the Halting problem in Figure 9.1 and taken from Charles Cooper’s website, Copyright 2019 Charles F. Cooper.

Section 7.2 in (Moore, Mertens, 2011) gives a highly recommended overview of uncomputability. Gödel, Escher, Bach (Hofstadter, 1999) is a classic popular science book that touches on uncomputability, and unprovability, and specifically Gödel’s Theorem that we will see in Chapter 11. See also the recent book by Holt (Holt, 2018) .

The history of the definition of a function is intertwined with the development of mathematics as a field. For many years, a function was identified (as per Euler’s quote above) with the means to calculate the output from the input. In the 1800’s, with the invention of the Fourier series and with the systematic study of continuity and differentiability, people have started looking at more general kinds of functions, but the modern definition of a function as an arbitrary mapping was not yet universally accepted. For example, in 1899 Poincare wrote “we have seen a mass of bizarre functions which appear to be forced to resemble as little as possible honest functions which serve some purpose. … they are invented on purpose to show that our ancestor’s reasoning was at fault, and we shall never get anything more than that out of them”. Some of this fascinating history is discussed in (Grabiner, 1983) (Kleiner, 1991) (Lützen, 2002) (Grabiner, 2005) .

The existence of a universal Turing machine, and the uncomputability of \(\ensuremath{\mathit{HALT}}\) was first shown by Turing in his seminal paper (Turing, 1937) , though closely related results were shown by Church a year before. These works built on Gödel’s 1931 incompleteness theorem that we will discuss in Chapter 11.

Some universal Turing machines with a small alphabet and number of states are given in (Rogozhin, 1996) , including a single-tape universal Turing machine with the binary alphabet and with less than \(25\) states; see also the survey (Woods, Neary, 2009) . Adam Yedidia has written software to help in producing Turing machines with a small number of states. This is related to the recreational pastime of “Code Golfing” which is about solving a certain computational task using the as short as possible program. Finding “highly complex” small Turing machine is also related to the “Busy Beaver” problem, see Exercise 9.13 and the survey (Aaronson, 2020) .

The diagonalization argument used to prove uncomputability of \(F^*\) is derived from Cantor’s argument for the uncountability of the reals discussed in Chapter 2.

Christopher Strachey was an English computer scientist and the inventor of the CPL programming language. He was also an early artificial intelligence visionary, programming a computer to play Checkers and even write love letters in the early 1950’s, see this New Yorker article and this website.

Rice’s Theorem was proven in (Rice, 1953) . It is typically stated in a form somewhat different than what we used, see Exercise 9.11.

We do not discuss in the chapter the concept of recursively enumerable languages, but it is covered briefly in Exercise 9.10. As usual, we use function, as opposed to language, notation.

Comments

Comments are posted on the GitHub repository using the utteranc.es app. A GitHub login is required to comment. If you don't want to authorize the app to post on your behalf, you can also comment directly on the GitHub issue for this page.

Compiled on 12/06/2023 00:07:11

Copyright 2023, Boaz Barak.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Produced using pandoc and panflute with templates derived from gitbook and bookdown.