Loops and infinity

“The bounds of arithmetic were however outstepped the moment the idea of applying the [punched] cards had occurred; and the Analytical Engine does not occupy common ground with mere ‘calculating machines.’… In enabling mechanism to combine together general symbols, in successions of unlimited variety and extent, a uniting link is established between the operations of matter and the abstract mental processes of the most abstract branch of mathematical science.” , Ada Augusta, countess of Lovelace, 1843

As the quote of Chapter 6 says, an algorithm is “a finite answer to an infinite number of questions”. To express an algorithm, we need to write down a finite set of instructions that will enable us to compute on arbitrarily long inputs. To describe and execute an algorithm we need the following components (see Figure 7.1):

• The finite set of instructions to be performed.

• Some “local variables” or finite state used in the execution.

• A potentially unbounded working memory to store the input and any other values we may require later.

• While the memory is unbounded, at every single step we can only read and write to a finite part of it, and we need a way to address which are the parts we want to read from and write to.

• If we only have a finite set of instructions but our input can be arbitrarily long, we will need to repeat instructions (i.e., loop back). We need a mechanism to decide when we will loop and when we will halt.

Turing Machines

“Computing is normally done by writing certain symbols on paper. We may suppose that this paper is divided into squares like a child’s arithmetic book.. The behavior of the [human] computer at any moment is determined by the symbols which he is observing, and of his’ state of mind’ at that moment… We may suppose that in a simple operation not more than one symbol is altered.” ,
“We compare a man in the process of computing … to a machine which is only capable of a finite number of configurations… The machine is supplied with a ‘tape’ (the analogue of paper) … divided into sections (called ‘squares’) each capable of bearing a ‘symbol’” , Alan Turing, 1936

“What is the difference between a Turing machine and the modern computer? It’s the same as that between Hillary’s ascent of Everest and the establishment of a Hilton hotel on its peak.” , Alan Perlis, 1982.

The “granddaddy” of all models of computation is the Turing machine. Turing machines were defined in 1936 by Alan Turing in an attempt to formally capture all the functions that can be computed by human “computers” (see Figure 7.4) that follow a well-defined set of rules, such as the standard algorithms for addition or multiplication.

Turing thought of such a person as having access to as much “scratch paper” as they need. For simplicity, we can think of this scratch paper as a one dimensional piece of graph paper (or tape, as it is commonly referred to). The paper is divided into “cells”, where each “cell” can hold a single symbol (e.g., one digit or letter, and more generally, some element of a finite alphabet). At any point in time, the person can read from and write to a single cell of the paper. Based on the contents of this cell, the person can update their finite mental state, and/or move to the cell immediately to the left or right of the current one.

Turing modeled such a computation by a “machine” that maintains one of \(k\) states. At each point in time the machine reads from its “work tape” a single symbol from a finite alphabet \(\Sigma\) and uses that to update its state, write to tape, and possibly move to an adjacent cell (see Figure 7.7). To compute a function \(F\) using this machine, we initialize the tape with the input \(x\in \{0,1\}^*\) and our goal is to ensure that the tape will contain the value \(F(x)\) at the end of the computation. Specifically, a computation of a Turing machine \(M\) with \(k\) states and alphabet \(\Sigma\) on input \(x\in \{0,1\}^*\) proceeds as follows:

• Initially the machine is at state \(0\) (known as the “starting state”) and the tape is initialized to \(\triangleright,x_0,\ldots,x_{n-1},\varnothing,\varnothing,\ldots\). We use the symbol \(\triangleright\) to denote the beginning of the tape, and the symbol \(\varnothing\) to denote an empty cell. We will always assume that the alphabet \(\Sigma\) is a (potentially strict) superset of \(\{ \triangleright, \varnothing , 0 , 1 \}\).

• The location \(i\) to which the machine points to is set to \(0\).

• At each step, the machine reads the symbol \(\sigma = T[i]\) that is in the \(i^{th}\) location of the tape. Based on this symbol and its state \(s\), the machine decides on:

  • What symbol \(\sigma'\) to write on the tape
    
  • Whether to move Left (i.e., \(i \leftarrow i-1\)), Right (i.e., \(i \leftarrow i+1\)), Stay in place, or Halt the computation.
  • What is going to be the new state \(s \in [k]\)

• The set of rules the Turing machine follows is known as its transition function.

• When the machine halts, its output is the binary string obtained by reading the tape from the beginning until the first location in which it contains a \(\varnothing\) symbol, and then outputting all \(0\) and \(1\) symbols in sequence, dropping the initial \(\triangleright\) symbol if it exists, as well as the final \(\varnothing\) symbol.

Extended example: A Turing machine for palindromes

Let \(\ensuremath{\mathit{PAL}}\) (for palindromes) be the function that on input \(x\in \{0,1\}^*\), outputs \(1\) if and only if \(x\) is an (even length) palindrome, in the sense that \(x = w_0 \cdots w_{n-1}w_{n-1}w_{n-2}\cdots w_0\) for some \(n\in \N\) and \(w\in \{0,1\}^n\).

We now show a Turing machine \(M\) that computes \(\ensuremath{\mathit{PAL}}\). To specify \(M\) we need to specify (i) \(M\)’s tape alphabet \(\Sigma\) which should contain at least the symbols \(0\),\(1\), \(\triangleright\) and \(\varnothing\), and (ii) \(M\)’s transition function which determines what action \(M\) takes when it reads a given symbol while it is in a particular state.

In our case, \(M\) will use the alphabet \(\{ 0,1,\triangleright, \varnothing, \times \}\) and will have \(k=11\) states. Though the states are simply numbers between \(0\) and \(k-1\), we will give them the following labels for convenience:

We describe the operation of our Turing machine \(M\) in words:

• \(M\) starts in state START and goes right, looking for the first symbol that is \(0\) or \(1\). If it finds \(\varnothing\) before it hits such a symbol then it moves to the OUTPUT_1 state described below.

• Once \(M\) finds such a symbol \(b \in \{0,1\}\), \(M\) deletes \(b\) from the tape by writing the \(\times\) symbol, it enters either the RIGHT_\(b\) mode and starts moving rightwards until it hits the first \(\varnothing\) or \(\times\) symbol.

• Once \(M\) finds this symbol, it goes into the state LOOK_FOR_0 or LOOK_FOR_1 depending on whether it was in the state RIGHT_0 or RIGHT_1 and makes one left move.

• In the state LOOK_FOR_\(b\), \(M\) checks whether the value on the tape is \(b\). If it is, then \(M\) deletes it by changing its value to \(\times\), and moves to the state RETURN. Otherwise, it changes to the OUTPUT_0 state.

• The RETURN state means that \(M\) goes back to the beginning. Specifically, \(M\) moves leftward until it hits the first symbol that is not \(0\) or \(1\), in which case it changes its state to START.

• The OUTPUT_\(b\) states mean that \(M\) will eventually output the value \(b\). In both the OUTPUT_0 and OUTPUT_1 states, \(M\) goes left until it hits \(\triangleright\). Once it does so, it makes a right step, and changes to the 1_AND_BLANK or 0_AND_BLANK states respectively. In the latter states, \(M\) writes the corresponding value, moves right and changes to the BLANK_AND_STOP state, in which it writes \(\varnothing\) to the tape and halts.

The above description can be turned into a table describing for each one of the \(11\cdot 5\) combination of state and symbol, what the Turing machine will do when it is in that state and it reads that symbol. This table is known as the transition function of the Turing machine.

Turing machines: a formal definition

The formal definition of Turing machines is as follows:

One should not confuse the transition function \(\delta_M\) of a Turing machine \(M\) with the function that the machine computes. The transition function \(\delta_M\) is a finite function, with \(k|\Sigma|\) inputs and \(4k|\Sigma|\) outputs. (Can you see why?) The machine can compute an infinite function \(F\) that takes as input a string \(x\in \{0,1\}^*\) of arbitrary length and might also produce an arbitrary length string as output.

In our formal definition, we identified the machine \(M\) with its transition function \(\delta_M\) since the transition function tells us everything we need to know about the Turing machine. However, this choice of representation is somewhat arbitrary, and is based on our convention that the state space is always the numbers \(\{0,\ldots,k-1\}\) with \(0\) as the starting state. Other texts use different conventions, and so their mathematical definition of a Turing machine might look superficially different. However, these definitions describe the same computational process and have the same computational powers. Hence they are equivalent despite their superficial differences. See Section 7.7 for a comparison between Definition 7.1 and the way Turing Machines are defined in texts such as Sipser (Sipser, 1997) .

Computable functions

We now turn to make one of the most important definitions in this book: computable functions.

Defining a function “computable” if and only if it can be computed by a Turing machine might seem “reckless” but, as we’ll see in Chapter 8, being computable in the sense of Definition 7.2 is equivalent to being computable in virtually any reasonable model of computation. This statement is known as the Church-Turing Thesis. (Unlike the extended Church-Turing Thesis which we discussed in Section 5.6, the Church-Turing thesis itself is widely believed and there are no candidate devices that attack it.)

This is a good point to remind the reader that functions are not the same as programs:

A Turing machine (or program) \(M\) can compute some function \(F\), but it is not the same as \(F\). In particular, there can be more than one program to compute the same function. Being computable is a property of functions, not of machines.

We will often pay special attention to functions \(F:\{0,1\}^* \rightarrow \{0,1\}\) that have a single bit of output. Hence we give a special name for the set of computable functions of this form.

Infinite loops and partial functions

One crucial difference between circuits/straight-line programs and Turing machines is the following. Looking at a NAND-CIRC program \(P\), we can always tell how many inputs and how many outputs \(P\) has by simply looking at the X and Y variables. Furthermore, we are guaranteed that if we invoke \(P\) on any input, then some output will be produced.

In contrast, given a Turing machine \(M\), we cannot determine a priori the length of \(M\)’s output. In fact, we don’t even know if an output would be produced at all! For example, it is straightforward to come up with a Turing machine whose transition function never outputs \(\mathsf{H}\) and hence never halts.

If a machine \(M\) fails to stop and produce an output on some input \(x\), then it cannot compute any total function \(F\), since clearly on input \(x\), \(M\) will fail to output \(F(x)\). However, \(M\) can still compute a partial function.¹

For example, consider the partial function \(\ensuremath{\mathit{DIV}}\) that on input a pair \((a,b)\) of natural numbers, outputs \(\ceil{a/b}\) if \(b > 0\), and is undefined otherwise. We can define a Turing machine \(M\) that computes \(\ensuremath{\mathit{DIV}}\) on input \(a,b\) by outputting the first \(c=0,1,2,\ldots\) such that \(cb \geq a\). If \(a>0\) and \(b=0\) then the machine \(M\) will never halt, but this is OK, since \(\ensuremath{\mathit{DIV}}\) is undefined on such inputs. If \(a=0\) and \(b=0\), the machine \(M\) will output \(0\), which is also OK, since we don’t care about what the program outputs on inputs on which \(\ensuremath{\mathit{DIV}}\) is undefined. Formally, we define computability of partial functions as follows:

Note that if \(F\) is a total function, then it is defined on every \(x\in \{0,1\}^*\) and hence in this case, Definition 7.5 is identical to Definition 7.2.

Turing machines as programming languages

The name “Turing machine”, with its “tape” and “head” evokes a physical object, while in contrast we think of a program as a piece of text. But we can think of a Turing machine as a program as well. For example, consider the Turing machine \(M\) of Section 7.1.1 that computes the function \(\ensuremath{\mathit{PAL}}\) such that \(\ensuremath{\mathit{PAL}}(x)=1\) iff \(x\) is a palindrome. We can also describe this machine as a program using the Python-like pseudocode of the form below

# Gets an array Tape initialized to
# [">", x_0 , x_1 , .... , x_(n-1), "∅", "∅", ...]
# At the end of the execution, Tape[1] is equal to 1
# if x is a palindrome and is equal to 0 otherwise
def PAL(Tape):
    head = 0
    state = 0 # START
    while (state != 12):
        if (state == 0 && Tape[head]=='0'):
            state = 3 # LOOK_FOR_0
            Tape[head] = 'x'
            head += 1 # move right
        if (state==0 && Tape[head]=='1')
            state = 4 # LOOK_FOR_1
            Tape[head] = 'x'
            head += 1 # move right
        ... # more if statements here

The precise details of this program are not important. What matters is that we can describe Turing machines as programs. Moreover, note that when translating a Turing machine into a program, the tape becomes a list or array that can hold values from the finite set \(\Sigma\).² The head position can be thought of as an integer-valued variable that holds integers of unbounded size. The state is a local register that can hold one of a fixed number of values in \([k]\).

More generally we can think of every Turing machine \(M\) as equivalent to a program similar to the following:

# Gets an array Tape initialized to
# [">", x_0 , x_1 , .... , x_(n-1), "∅", "∅", ...]
def M(Tape):
    state = 0
    i     = 0 # holds head location
    while (True):
        # Move head, modify state, write to tape
        # based on current state and cell at head
        # below are just examples for how program looks for a particular transition function
        if Tape[i]=="0" and state==7: # δ_M(7,"0")=(19,"1","R")
            Tape[i]="1"
            i += 1

            state = 19
        elif Tape[i]==">" and state == 13: # δ_M(13,">")=(15,"0","S")
            Tape[i]="0"
            state = 15
        elif ...
        ...
        elif Tape[i]==">" and state == 29: # δ_M(29,">")=(.,.,"H")
            break # Halt

If we wanted to use only Boolean (i.e., \(0\)/\(1\)-valued) variables, then we can encode the state variables using \(\ceil{\log k}\) bits. Similarly, we can represent each element of the alphabet \(\Sigma\) using \(\ell=\ceil{\log |\Sigma|}\) bits and hence we can replace the \(\Sigma\)-valued array Tape[] with \(\ell\) Boolean-valued arrays Tape0[],\(\ldots\), Tape\((\ell - 1)\)[].

The NAND-TM Programming language

We now introduce the NAND-TM programming language, which captures the power of a Turing machine with a programming-language formalism. Like the difference between Boolean circuits and Turing machines, the main difference between NAND-TM and NAND-CIRC is that NAND-TM models a single uniform algorithm that can compute a function that takes inputs of arbitrary lengths. To do so, we extend the NAND-CIRC programming language with two constructs:

• Loops: NAND-CIRC is a straight-line programming language- a NAND-CIRC program of \(s\) lines takes exactly \(s\) steps of computation and hence in particular, cannot even touch more than \(3s\) variables. Loops allow us to use a fixed-length program to encode the instructions for a computation that can take an arbitrary amount of time.

• Arrays: A NAND-CIRC program of \(s\) lines touches at most \(3s\) variables. While we can use variables with names such as Foo_17 or Bar[22] in NAND-CIRC, they are not true arrays, since the number in the identifier is a constant that is “hardwired” into the program. NAND-TM contains actual arrays that can have a length that is not a priori bounded.

Thus a good way to remember NAND-TM is using the following informal equation:

Concretely, the NAND-TM programming language adds the following features on top of NAND-CIRC (see Figure 7.8):

• We add a special integer valued variable i. All other variables in NAND-TM are Boolean valued (as in NAND-CIRC).

• Apart from i NAND-TM has two kinds of variables: scalars and arrays. Scalar variables hold one bit (just as in NAND-CIRC). Array variables hold an unbounded number of bits. At any point in the computation we can access the array variables at the location indexed by i using Foo[i]. We cannot access the arrays at locations other than the one pointed to by i.

• We use the convention that arrays always start with a capital letter, and scalar variables (which are never indexed with i) start with lowercase letters. Hence Foo is an array and bar is a scalar variable.

• The input and output X and Y are now considered arrays with values of zeroes and ones. (There are also two other special arrays X_nonblank and Y_nonblank, see below.)

• We add a special MODANDJUMP instruction that takes two Boolean variables \(a,b\) as input and does the following:

  • If \(a=1\) and \(b=1\) then MODANDJUMP(\(a,b\)) increments i by one and jumps to the first line of the program.
  • If \(a=0\) and \(b=1\) then MODANDJUMP(\(a,b\)) decrements i by one and jumps to the first line of the program. (If i is already equal to \(0\) then it stays at \(0\).)
  • If \(a=1\) and \(b=0\) then MODANDJUMP(\(a,b\)) jumps to the first line of the program without modifying i.
  • If \(a=b=0\) then MODANDJUMP(\(a,b\)) halts execution of the program.

• The MODANDJUMP instruction always appears in the last line of a NAND-TM program and nowhere else.

Default values. We need one more convention to handle “default values”. Turing machines have the special symbol \(\varnothing\) to indicate that tape location is “blank” or “uninitialized”. In NAND-TM there is no such symbol, and all variables are Boolean, containing either \(0\) or \(1\). All variables and locations of arrays default to \(0\) if they have not been initialized to another value. To keep track of whether a \(0\) in an array corresponds to a true zero or to an uninitialized cell, a programmer can always add to an array Foo a “companion array” Foo_nonblank and set Foo_nonblank[i] to \(1\) whenever the i th location is initialized. In particular, we will use this convention for the input and output arrays X and Y. A NAND-TM program has four special arrays X, X_nonblank, Y, and Y_nonblank. When a NAND-TM program is executed on input \(x\in \{0,1\}^*\) of length \(n\), the first \(n\) cells of the array X are initialized to \(x_0,\ldots,x_{n-1}\) and the first \(n\) cells of the array X_nonblank are initialized to \(1\). (All uninitialized cells default to \(0\).) The output of a NAND-TM program is the string Y[\(0\)], \(\ldots\), Y[\(m-1\)] where \(m\) is the smallest integer such that Y_nonblank[\(m\)]\(=0\). A NAND-TM program gets called with X and X_nonblank initialized to contain the input, and writes to Y and Y_nonblank to produce the output.

Formally, NAND-TM programs are defined as follows:

Sneak peak: NAND-TM vs Turing machines

As the name implies, NAND-TM programs are a direct implementation of Turing machines in programming language form. We will show the equivalence below, but you can already see how the components of Turing machines and NAND-TM programs correspond to one another:

Examples

We now present some examples of NAND-TM programs.

carry = IF(started,carry,one(started))
started = one(started)
Y[i] = XOR(X[i],carry)
carry = AND(X[i],carry)
Y_nonblank[i] = one(started)
MODANDJUMP(X_nonblank[i],X_nonblank[i])

temp_0 = NAND(started,started)
temp_1 = NAND(started,temp_0)
temp_2 = NAND(started,started)
temp_3 = NAND(temp_1,temp_2)
temp_4 = NAND(carry,started)
carry = NAND(temp_3,temp_4)
temp_6 = NAND(started,started)
started = NAND(started,temp_6)
temp_8 = NAND(X[i],carry)
temp_9 = NAND(X[i],temp_8)
temp_10 = NAND(carry,temp_8)
Y[i] = NAND(temp_9,temp_10)
temp_12 = NAND(X[i],carry)
carry = NAND(temp_12,temp_12)
temp_14 = NAND(started,started)
Y_nonblank[i] = NAND(started,temp_14)
MODANDJUMP(X_nonblank[i],X_nonblank[i])

temp_0 = NAND(X[0],X[0])
Y_nonblank[0] = NAND(X[0],temp_0)
temp_2 = NAND(X[i],Y[0])
temp_3 = NAND(X[i],temp_2)
temp_4 = NAND(Y[0],temp_2)
Y[0] = NAND(temp_3,temp_4)
MODANDJUMP(X_nonblank[i],X_nonblank[i])

Equivalence of Turing machines and NAND-TM programs

Given the above discussion, it might not be surprising that Turing machines turn out to be equivalent to NAND-TM programs. Indeed, we designed the NAND-TM language to have this property. Nevertheless, this is a significant result, and the first of many other such equivalence results we will see in this book.

Specification vs implementation (again)

Once you understand the definitions of both NAND-TM programs and Turing machines, Theorem 7.11 is straightforward. Indeed, NAND-TM programs are not as much a different model from Turing machines as they are simply a reformulation of the same model using programming language notation. You can think of the difference between a Turing machine and a NAND-TM program as the difference between representing a number using decimal or binary notation. In contrast, the difference between a function \(F\) and a Turing machine that computes \(F\) is much more profound: it is like the difference between the equation \(x^2 + x = 12\), and the number \(3\) that is a solution for this equation. For this reason, while we take special care in distinguishing functions from programs or machines, we will often identify the two latter concepts. We will move freely between describing an algorithm as a Turing machine or as a NAND-TM program (as well as some of the other equivalent computational models we will see in Chapter 8 and beyond).

NAND-TM syntactic sugar

Just like we did with NAND-CIRC in Chapter 4, we can use “syntactic sugar” to make NAND-TM programs easier to write. For starters, we can use all of the syntactic sugar of NAND-CIRC, such as macro definitions and conditionals (i.e., if/then). However, we can go beyond this and achieve (for example):

• Inner loops such as the while and for operations common to many programming languages.

• Multiple index variables (e.g., not just i but we can add j, k, etc.).

• Arrays with more than one dimension (e.g., Foo[i][j], Bar[i][j][k] etc.)

In all of these cases (and many others) we can implement the new feature as mere “syntactic sugar” on top of standard NAND-TM. This means that the set of functions computable by NAND-TM with this feature is the same as the set of functions computable by standard NAND-TM. Similarly, we can show that the set of functions computable by Turing machines that have more than one tape, or tapes of more dimensions than one, is the same as the set of functions computable by standard Turing machines.

“GOTO” and inner loops

We can implement more advanced looping constructs than the simple MODANDJUMP. For example, we can implement GOTO. A GOTO statement corresponds to jumping to a specific line in the execution. For example, if we have code of the form

"start":  do foo
   GOTO("end")
"skip": do bar
"end": do blah

then the program will only do foo and blah as when it reaches the line GOTO("end") it will jump to the line labeled with "end". We can achieve the effect of GOTO in NAND-TM using conditionals. In the code below, we assume that we have a variable pc that can take strings of some constant length. This can be encoded using a finite number of Boolean variables pc_0, pc_1, \(\ldots\), pc_\(k-1\), and so when we write below pc = "label" what we mean is something like pc_0 = 0,pc_1 = 1, \(\ldots\) (where the bits \(0,1,\ldots\) correspond to the encoding of the finite string "label" as a string of length \(k\)). We also assume that we have access to conditional (i.e., if statements), which we can emulate using syntactic sugar in the same way as we did in NAND-CIRC.

To emulate a GOTO statement, we will first modify a program P of the form

do foo
do bar
do blah

to have the following form (using syntactic sugar for if):

pc = "line1"
if (pc=="line1"):
    do foo
    pc = "line2"
if (pc=="line2"):
    do bar
    pc = "line3"
if (pc=="line3"):
    do blah

These two programs do the same thing. The variable pc corresponds to the “program counter” and tells the program which line to execute next. We can see that if we wanted to emulate a GOTO("line3") then we could simply modify the instruction pc = "line2" to be pc = "line3".

In NAND-CIRC we could only have GOTOs that go forward in the code, but since in NAND-TM everything is encompassed within a large outer loop, we can use the same ideas to implement GOTOs that can go backward, as well as conditional loops.

Other loops. Once we have GOTO, we can emulate all the standard loop constructs such as while, do .. until or for in NAND-TM as well. For example, we can replace the code

while foo:
    do blah
do bar

with

"loop":
    if NOT(foo): GOTO("next")
    do blah
    GOTO("loop")
"next":
    do bar

for j in range(100):
    do something

do blah

Uniformity, and NAND vs NAND-TM (discussion)

While NAND-TM adds extra operations over NAND-CIRC, it is not exactly accurate to say that NAND-TM programs or Turing machines are “more powerful” than NAND-CIRC programs or Boolean circuits. NAND-CIRC programs, having no loops, are simply not applicable for computing functions with an unbounded number of inputs. Thus, to compute a function \(F:\{0,1\}^* :\rightarrow \{0,1\}^*\) using NAND-CIRC (or equivalently, Boolean circuits) we need a collection of programs/circuits: one for every input length.

The key difference between NAND-CIRC and NAND-TM is that NAND-TM allows us to express the fact that the algorithm for computing parities of length-\(100\) strings is really the same one as the algorithm for computing parities of length-\(5\) strings (or similarly the fact that the algorithm for adding \(n\)-bit numbers is the same for every \(n\), etc.). That is, one can think of the NAND-TM program for general parity as the “seed” out of which we can grow NAND-CIRC programs for length \(10\), length \(100\), or length \(1000\) parities as needed.

This notion of a single algorithm that can compute functions of all input lengths is known as uniformity of computation. Hence we think of Turing machines / NAND-TM as uniform models of computation, as opposed to Boolean circuits or NAND-CIRC, which are non-uniform models, in which we have to specify a different program for every input length.

Looking ahead, we will see that this uniformity leads to another crucial difference between Turing machines and circuits. Turing machines can have inputs and outputs that are longer than the description of the machine as a string, and in particular there exists a Turing machine that can “self replicate” in the sense that it can print its own code. The notion of “self replication”, and the related notion of “self reference” are crucial to many aspects of computation, and beyond that to life itself, whether in the form of digital or biological programs.

For now, what you ought to remember is the following differences between uniform and non-uniform computational models:

• Non-uniform computational models: Examples are NAND-CIRC programs and Boolean circuits. These are models where each individual program/circuit can compute a finite function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\). We have seen that every finite function can be computed by some program/circuit. To discuss computation of an infinite function \(F:\{0,1\}^* \rightarrow \{0,1\}^*\) we need to allow a sequence \(\{ P_n \}_{n\in \N}\) of programs/circuits (one for every input length), but this does not capture the notion of a single algorithm to compute the function \(F\).

• Uniform computational models: Examples are Turing machines and NAND-TM programs. These are models where a single program/machine can take inputs of arbitrary length and hence compute an infinite function \(F:\{0,1\}^* \rightarrow \{0,1\}^*\). The number of steps that a program/machine takes on some input is not a priori bounded in advance and in particular there is a chance that it will enter into an infinite loop. Unlike the non-uniform case, we have not shown that every infinite function can be computed by some NAND-TM program/Turing machine. We will come back to this point in Chapter 9.

Exercises

Bibliographical notes

Augusta Ada Byron, countess of Lovelace (1815-1852) lived a short but turbulent life, though is today most well known for her collaboration with Charles Babbage (see (Stein, 1987) for a biography). Ada took an immense interest in Babbage’s analytical engine, which we mentioned in Chapter 3. In 1842-3, she translated from Italian a paper of Menabrea on the engine, adding copious notes (longer than the paper itself). The quote in the chapter’s beginning is taken from Nota A in this text. Lovelace’s notes contain several examples of programs for the analytical engine, and because of this she has been called “the world’s first computer programmer” though it is not clear whether they were written by Lovelace or Babbage himself (Holt, 2001) . Regardless, Ada was clearly one of very few people (perhaps the only one outside of Babbage himself) to fully appreciate how important and revolutionary the idea of mechanizing computation truly is.

The books of Shetterly (Shetterly, 2016) and Sobel (Sobel, 2017) discuss the history of human computers (who were female, more often than not) and their important contributions to scientific discoveries in astronomy and space exploration.

Alan Turing was one of the intellectual giants of the 20th century. He was not only the first person to define the notion of computation, but also invented and used some of the world’s earliest computational devices as part of the effort to break the Enigma cipher during World War II, saving millions of lives. Tragically, Turing committed suicide in 1954, following his conviction in 1952 for homosexual acts and a court-mandated hormonal treatment. In 2009, British prime minister Gordon Brown made an official public apology to Turing, and in 2013 Queen Elizabeth II granted Turing a posthumous pardon. Turing’s life is the subject of a great book and a mediocre movie.

Sipser’s text (Sipser, 1997) defines a Turing machine as a seven tuple consisting of the state space, input alphabet, tape alphabet, transition function, starting state, accepting state, and rejecting state. Superficially this looks like a very different definition than Definition 7.1 but it is simply a different representation of the same concept, just as a graph can be represented in either adjacency list or adjacency matrix form.

One difference is that Sipser considers a general set of states \(Q\) that is not necessarily of the form \(Q=\{0,1,2,\ldots, k-1\}\) for some natural number \(k>0\). Sipser also restricts his attention to Turing machines that output only a single bit and therefore designates two special halting states: the “\(0\) halting state” (often known as the rejecting state) and the other as the “\(1\) halting state” (often known as the accepting state). Thus instead of writing \(0\) or \(1\) on an output tape, the machine will enter into one of these states and halt. This again makes no difference to the computational power, though we prefer to consider the more general model of multi-bit outputs. (Sipser presents the basic task of a Turing machine as that of deciding a language as opposed to computing a function, but these are equivalent, see Remark 7.4.)

Sipser considers also functions with input in \(\Sigma^*\) for an arbitrary alphabet \(\Sigma\) (and hence distinguishes between the input alphabet which he denotes as \(\Sigma\) and the tape alphabet which he denotes as \(\Gamma\)), while we restrict attention to functions with binary strings as input. Again this is not a major issue, since we can always encode an element of \(\Sigma\) using a binary string of length \(\log \ceil{|\Sigma|}\). Finally (and this is a very minor point) Sipser requires the machine to either move left or right in every step, without the \(\mathsf{S}\)tay operation, though staying in place is very easy to emulate by simply moving right and then back left.

Another definition used in the literature is that a Turing machine \(M\) recognizes a language \(L\) if for every \(x\in L\), \(M(x)=1\) and for every \(x\not\in L\), \(M(x) \in \{0,\bot \}\). A language \(L\) is recursively enumerable if there exists a Turing machine \(M\) that recognizes it, and the set of all recursively enumerable languages is often denoted by \(\mathbf{RE}\). We will not use this terminology in this book.

One of the first programming-language formulations of Turing machines was given by Wang (Wang, 1957) . Our formulation of NAND-TM is aimed at making the connection with circuits more direct, with the eventual goal of using it for the Cook-Levin Theorem, as well as results such as \(\mathbf{P} \subseteq \mathbf{P_{/poly}}\) and \(\mathbf{BPP} \subseteq \mathbf{P_{/poly}}\). The website esolangs.org features a large variety of esoteric Turing-complete programming languages. One of the most famous of them is Brainf*ck.

Comments

Comments are posted on the GitHub repository using the utteranc.es app. A GitHub login is required to comment. If you don't want to authorize the app to post on your behalf, you can also comment directly on the GitHub issue for this page.

Compiled on 12/06/2023 00:07:28

Copyright 2023, Boaz Barak.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Produced using pandoc and panflute with templates derived from gitbook and bookdown.