See any bugs/typos/confusing explanations? Open a GitHub issue. You can also comment below

★ See also the PDF version of this chapter (better formatting/references) ★

Syntactic sugar, and computing every function

  • Get comfort with syntactic sugar or automatic translation of higher level logic to low level gates.
  • Learn proof of major result: every finite function can be computed by a Boolean circuit.
  • Start thinking quantitatively about number of lines required for computation.

“[In 1951] I had a running compiler and nobody would touch it because, they carefully told me, computers could only do arithmetic; they could not do programs.”, Grace Murray Hopper, 1986.

“Syntactic sugar causes cancer of the semicolon.”, Alan Perlis, 1982.

The computational models we considered thus far are as “bare bones” as they come. For example, our NAND-CIRC “programming language” has only the single operation foo = NAND(bar,blah). In this chapter we will see that these simple models are actually equivalent to more powerful ones. The key observation is that we can implement more complex features using our basic building blocks, and then use these new features themselves as building blocks for even more sophisticated features. This is known as “syntactic sugar” in the field of programming language design, since we are not modifying the underlying programming model itself but rather we merely implement new features by syntactically transforming a program that uses such features into one that doesn’t.

This chapter provides a “toolkit” that can be used to show that many functions can be computed by NAND-CIRC programs, and hence also by Boolean circuits. We will also use this toolkit to prove a fundamental theorem: every finite function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a Boolean circuit, see Theorem 4.12 below. While the syntactic sugar toolkit is important in its own right, Theorem 4.12 can also be proven directly without using this toolkit. We present this alternative proof in Section 4.5. See Figure 4.1 for an outline of the results of this chapter.

4.1: An outline of the results of this chapter. In  we give a toolkit of “syntactic sugar” transformations showing how to implement features such as programmer-defined functions and conditional statements in NAND-CIRC. We use these tools in  to give a NAND-CIRC program (or alternatively a Boolean circuit) to compute the \ensuremath{\mathit{LOOKUP}} function. We then build on this result to show in  that NAND-CIRC programs (or equivalently, Boolean circuits) can compute every finite function. An alternative direct proof of the same result is given in .
4.1: An outline of the results of this chapter. In Section 4.1 we give a toolkit of “syntactic sugar” transformations showing how to implement features such as programmer-defined functions and conditional statements in NAND-CIRC. We use these tools in Section 4.3 to give a NAND-CIRC program (or alternatively a Boolean circuit) to compute the \(\ensuremath{\mathit{LOOKUP}}\) function. We then build on this result to show in Section 4.4 that NAND-CIRC programs (or equivalently, Boolean circuits) can compute every finite function. An alternative direct proof of the same result is given in Section 4.5.

Some examples of syntactic sugar

We now present some examples of “syntactic sugar” transformations that we can use in constructing straightline programs or circuits. We focus on the straight-line programming language view of our computational models, and specifically(for the sake of concreteness) on the NAND-CIRC programming language. This is convenient because many of the syntactic sugar transformations we present are easiest to think about in terms of applying “search and replace” operations to the source code of a program. However, by Theorem 3.18, all of our results hold equally well for circuits, whether ones using NAND gates or Boolean circuits that use the AND, OR, and NOT operations. Enumerating the examples of such syntactic sugar transformations can be a little tedious, but we do it for two reasons:

  1. To convince you that despite their seeming simplicity and limitations, simple models such as Boolean circuits or the NAND-CIRC programming language are actually quite powerful.

  2. So you can realize how lucky you are to be taking a theory of computation course and not a compilers course… :)

User-defined procedures

One staple of almost any programming language is the ability to define and then execute procedures or subroutines. (These are often known as functions in some programming languages, but we prefer the name procedures to avoid confusion with the function that a program computes.) The NAND-CIRC programming language does not have this mechanism built in. However, we can achieve the same effect using the time honored technique of “copy and paste”. Specifically, we can replace code which defines a procedure such as

with the following code where we “paste” the code of Proc

and where proc_code' is obtained by replacing all occurrences of a with d, b with e, and c with f. When doing that we will need to ensure that all other variables appearing in proc_code' don’t interfere with other variables. We can always do so by renaming variables to new names that were not used before. The above reasoning leads to the proof of the following theorem:

Let NAND-CIRC-PROC be the programming language NAND-CIRC augmented with the syntax above for defining procedures. Then for every NAND-CIRC-PROC program \(P\), there exists a standard (i.e., “sugar free”) NAND-CIRC program \(P'\) that computes the same function as \(P\).

Theorem 4.1 can be proven using the transformation above, but since the formal proof is somewhat long and tedious, we omit it here.

Procedures allow us to express NAND-CIRC programs much more cleanly and succinctly. For example, because we can compute AND, OR, and NOT using NANDs, we can compute the Majority function as follows:

Figure 4.2 presents the “sugar free” NAND-CIRC program (and the corresponding circuit) that is obtained by “expanding out” this program, replacing the calls to procedures with their definitions.

Once we show that a computational model \(X\) is equivalent in power to the model with an additional feature \(Y\), we can use this feature whenever we need to show that some function \(f\) is computable by \(X\).

4.2: A standard (i.e., “sugar free”) NAND-CIRC program that is obtained by expanding out the procedure definitions in the program for Majority of . The corresponding circuit is on the right. Note that this is not the most efficient NAND circuit/program for majority: we can save on some gates by “short cutting” steps where a gate u computes \ensuremath{\mathit{NAND}}(v,v) and then a gate w computes \ensuremath{\mathit{NAND}}(u,u) (as indicated by the dashed green arrows in the above figure).
4.2: A standard (i.e., “sugar free”) NAND-CIRC program that is obtained by expanding out the procedure definitions in the program for Majority of Example 4.2. The corresponding circuit is on the right. Note that this is not the most efficient NAND circuit/program for majority: we can save on some gates by “short cutting” steps where a gate \(u\) computes \(\ensuremath{\mathit{NAND}}(v,v)\) and then a gate \(w\) computes \(\ensuremath{\mathit{NAND}}(u,u)\) (as indicated by the dashed green arrows in the above figure).

While we can use syntactic sugar to present NAND-CIRC programs in more readable ways, we did not change the definition of the language itself. Therefore, whenever we say that some function \(f\) has an \(s\)-line NAND-CIRC program we mean a standard “sugar free” NAND-CIRC program, where all syntactic sugar has been expanded out. For example, the program of Example 4.2 is a \(12\)-line program for computing the \(\ensuremath{\mathit{MAJ}}\) function, even though it can be written in fewer lines using NAND-CIRC-PROC.

Proof by Python (optional)

We can write a Python program that implements the proof of Theorem 4.1. This is a Python program that takes a NAND-CIRC-PROC program \(P\) that includes procedure definitions and uses simple “search and replace” to transform \(P\) into a standard (i.e., “sugar free”) NAND-CIRC program \(P'\) that computes the same function as \(P\) without using any procedures. The idea is simple: if the program \(P\) contains a definition of a procedure Proc of two arguments x and y, then whenever we see a line of the form foo = Proc(bar,blah), we can replace this line by:

  1. The body of the procedure Proc (replacing all occurrences of x and y with bar and blah respectively).

  2. A line foo = exp, where exp is the expression following the return statement in the definition of the procedure Proc.

To make this more robust we a prefix to the internal variables used by Proc to ensure they don’t conflict with the variables of \(P\); for simplicity we ignore this issue in the code below though it can be easily added.

The code in Figure 4.3 achieves such a transformation.1

Figure 4.3: Python code for transforming NAND-CIRC-PROC programs into standard sugar free NAND-CIRC programs.

Figure 4.2 shows the result of applying the code of Figure 4.3 to the program of Example 4.2 that uses syntactic sugar to compute the Majority function. Specifically, we first apply desugar to remove usage of the OR function, then apply it to remove usage of the AND function, and finally apply it a third time to remove usage of the NOT function.

The function desugar in Figure 4.3 assumes that it is given the procedure already split up into its name, arguments, and body. It is not crucial for our purposes to describe precisely to scan a definition and splitting it up to these components, but in case you are curious, it can be achieved in Python via the following code:

Conditional statements

Another sorely missing feature in NAND-CIRC is a conditional statement such as the if/then constructs that are found in many programming languages. However, using procedures, we can obtain an ersatz if/then construct. First we can compute the function \(\ensuremath{\mathit{IF}}:\{0,1\}^3 \rightarrow \{0,1\}\) such that \(\ensuremath{\mathit{IF}}(a,b,c)\) equals \(b\) if \(a=1\) and \(c\) if \(a=0\).

Before reading onward, try to see how you could compute the \(\ensuremath{\mathit{IF}}\) function using \(\ensuremath{\mathit{NAND}}\)’s. Once you you do that, see how you can use that to emulate if/then types of constructs.

The \(\ensuremath{\mathit{IF}}\) function can be implemented from NANDs as follows (see Exercise 4.2):

The \(\ensuremath{\mathit{IF}}\) function is also known as a multiplexing function, since \(cond\) can be thought of as a switch that controls whether the output is connected to \(a\) or \(b\). Once we have a procedure for computing the \(\ensuremath{\mathit{IF}}\) function, we can implement conditionals in NAND. The idea is that we replace code of the form

with code of the form

that assigns to foo its old value when condition equals \(0\), and assign to foo the value of blah otherwise. More generally we can replace code of the form

with code of the form

Using such transformations, we can prove the following theorem. Once again we omit the (not too insightful) full formal proof, though see Subsection 4.1.2 for some hints on how to obtain it.

Let NAND-CIRC-IF be the programming language NAND-CIRC augmented with if/then/else statements for allowing code to be conditionally executed based on whether a variable is equal to \(0\) or \(1\).
Then for every NAND-CIRC-IF program \(P\), there exists a standard (i.e., “sugar free”) NAND-CIRC program \(P'\) that computes the same function as \(P\).

Extended example: Addition and Multiplication (optional)

Using “syntactic sugar”, we can write the integer addition function as follows:

where zero is the constant zero function, and MAJ and XOR correspond to the majority and XOR functions respectively. In the above we used the loop for i in range(n) but we can expand this out by simply repeating the code \(n\) times, replacing the value of i with \(0,1,2,\ldots,n-1\). By expanding out all the features, for every value of \(n\) we can translate the above program into a standard (“sugar free”) NAND-CIRC program. Figure 4.4 depicts what we get for \(n=2\).

4.4: The NAND-CIRC program and corresponding NAND circuit for adding two-digit binary numbers that are obtained by “expanding out” all the syntactic sugar. The program/circuit has 43 lines/gates which is by no means necessary. It is possible to add n bit numbers using 9n NAND gates, see .
4.4: The NAND-CIRC program and corresponding NAND circuit for adding two-digit binary numbers that are obtained by “expanding out” all the syntactic sugar. The program/circuit has 43 lines/gates which is by no means necessary. It is possible to add \(n\) bit numbers using \(9n\) NAND gates, see Exercise 4.5.

By going through the above program carefully and accounting for the number of gates, we can see that it yields a proof of the following theorem (see also Figure 4.5):

For every \(n\in \N\), let \(\ensuremath{\mathit{ADD}}_n:\{0,1\}^{2n}\rightarrow \{0,1\}^{n+1}\) be the function that, given \(x,x'\in \{0,1\}^n\) computes the representation of the sum of the numbers that \(x\) and \(x'\) represent. Then there is a constant \(c \leq 30\) such that for every \(n\) there is a NAND-CIRC program of at most \(c\) lines computing \(\ensuremath{\mathit{ADD}}_n\).2

4.5: The number of lines in our NAND-CIRC program to add two n bit numbers, as a function of n, for n’s between 1 and 100. This is not the most efficient program for this task, but the important point is that it has the form O(n).
4.5: The number of lines in our NAND-CIRC program to add two \(n\) bit numbers, as a function of \(n\), for \(n\)’s between \(1\) and \(100\). This is not the most efficient program for this task, but the important point is that it has the form \(O(n)\).

Once we have addition, we can use the grade-school algorithm to obtain multiplication as well, thus obtaining the following theorem:

For every \(n\), let \(\ensuremath{\mathit{MULT}}_n:\{0,1\}^{2n}\rightarrow \{0,1\}^{2n}\) be the function that, given \(x,x'\in \{0,1\}^n\) computes the representation of the product of the numbers that \(x\) and \(x'\) represent. Then there is a constant \(c\) such that for every \(n\), there is a NAND-CIRC program of at most \(cn^2\) that computes the function \(\ensuremath{\mathit{MULT}}_n\).

We omit the proof, though in Exercise 4.7 we ask you to supply a “constructive proof” in the form of a program (in your favorite programming language) that on input a number \(n\), outputs the code of a NAND-CIRC program of at most \(1000n^2\) lines that computes the \(\ensuremath{\mathit{MULT}}_n\) function. In fact, we can use Karatsuba’s algorithm to show that there is a NAND-CIRC program of \(O(n^{\log_2 3})\) lines to compute \(\ensuremath{\mathit{MULT}}_n\) (and can get even further asymptotic improvements using better algorithms).

The LOOKUP function

The \(\ensuremath{\mathit{LOOKUP}}\) function will play an important role in this chapter and later. It is defined as follows:

For every \(k\), the lookup function of order \(k\), \(\ensuremath{\mathit{LOOKUP}}_k: \{0,1\}^{2^k+k}\rightarrow \{0,1\}\) is defined as follows. For every \(x\in\{0,1\}^{2^k}\) and \(i\in \{0,1\}^k\),

\[ \ensuremath{\mathit{LOOKUP}}_k(x,i)=x_i \]
where \(x_i\) denotes the \(i^{th}\) entry of \(x\), using the binary representation to identify \(i\) with a number in \(\{0,\ldots,2^k - 1 \}\).

4.6: The \ensuremath{\mathit{LOOKUP}}_k function takes an input in \{0,1\}^{2^k+k}, which we denote by x,i (with x\in \{0,1\}^{2^k} and i \in \{0,1\}^k). The output is x_i: the i-th coordinate of x, where we identify i as a number in [k] using the binary representation. In the above example x\in \{0,1\}^{16} and i\in \{0,1\}^4. Since i=0110 is the binary representation of the number 6, the output of \ensuremath{\mathit{LOOKUP}}_4(x,i) in this case is x_6 = 1.
4.6: The \(\ensuremath{\mathit{LOOKUP}}_k\) function takes an input in \(\{0,1\}^{2^k+k}\), which we denote by \(x,i\) (with \(x\in \{0,1\}^{2^k}\) and \(i \in \{0,1\}^k\)). The output is \(x_i\): the \(i\)-th coordinate of \(x\), where we identify \(i\) as a number in \([k]\) using the binary representation. In the above example \(x\in \{0,1\}^{16}\) and \(i\in \{0,1\}^4\). Since \(i=0110\) is the binary representation of the number \(6\), the output of \(\ensuremath{\mathit{LOOKUP}}_4(x,i)\) in this case is \(x_6 = 1\).

See Figure 4.6 for an illustration of the LOOKUP function. It turns out that for every \(k\), we can compute \(\ensuremath{\mathit{LOOKUP}}_k\) using a NAND-CIRC program:

For every \(k>0\), there is a NAND-CIRC program that computes the function \(\ensuremath{\mathit{LOOKUP}}_k: \{0,1\}^{2^k+k}\rightarrow \{0,1\}\). Moreover, the number of lines in this program is at most \(4\cdot 2^k\).

An immediate corollary of Theorem 4.9 is that for every \(k>0\), \(\ensuremath{\mathit{LOOKUP}}_k\) can be computed by a Boolean circuit (with AND, OR and NOT gates) of at most \(8 \cdot 2^k\) gates.

Constructing a NAND-CIRC program for \(\ensuremath{\mathit{LOOKUP}}\)

We prove Theorem 4.9 by induction. For the case \(k=1\), \(\ensuremath{\mathit{LOOKUP}}_1\) maps \((x_0,x_1,i) \in \{0,1\}^3\) to \(x_i\). In other words, if \(i=0\) then it outputs \(x_0\) and otherwise it outputs \(x_1\), which (up to reordering variables) is the same as the \(\ensuremath{\mathit{IF}}\) function presented in Subsection 4.1.3, which can be computed by a 4-line NAND-CIRC program.

As a warm-up for the case of general \(k\), let us consider the case of \(k=2\). Given input \(x=(x_0,x_1,x_2,x_3)\) for \(\ensuremath{\mathit{LOOKUP}}_2\) and an index \(i=(i_0,i_1)\), if the most significant bit \(i_0\) of the index is \(0\) then \(\ensuremath{\mathit{LOOKUP}}_2(x,i)\) will equal \(x_0\) if \(i_1=0\) and equal \(x_1\) if \(i_1=1\). Similarly, if the most significant bit \(i_0\) is \(1\) then \(\ensuremath{\mathit{LOOKUP}}_2(x,i)\) will equal \(x_2\) if \(i_1=0\) and will equal \(x_3\) if \(i_1=1\). Another way to say this is that we can write \(\ensuremath{\mathit{LOOKUP}}_2\) as follows:

or in other words,

More generally, as shown in the following lemma, we can compute \(\ensuremath{\mathit{LOOKUP}}_k\) using two invocations of \(\ensuremath{\mathit{LOOKUP}}_{k-1}\) and one invocation of \(\ensuremath{\mathit{IF}}\):

For every \(k \geq 2\), \(\ensuremath{\mathit{LOOKUP}}_k(x_0,\ldots,x_{2^k-1},i_0,\ldots,i_{k-1})\) is equal to

\[ \ensuremath{\mathit{IF}}(i_0,\ensuremath{\mathit{LOOKUP}}_{k-1}(x_0,\ldots,x_{2^{k-1}-1},i_1,\ldots,i_{k-1}), \ensuremath{\mathit{LOOKUP}}_{k-1}(x_{2^{k-1}},\ldots,x_{2^k-1},i_1,\ldots,i_{k-1})) \]

If the most significant bit \(i_{0}\) of \(i\) is zero, then the index \(i\) is in \(\{0,\ldots,2^{k-1}-1\}\) and hence we can perform the lookup on the “first half” of \(x\) and the result of \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) will be the same as \(a=\ensuremath{\mathit{LOOKUP}}_{k-1}(x_0,\ldots,x_{2^{k-1}-1},i_1,\ldots,i_{k-1})\). On the other hand, if this most significant bit \(i_{0}\) is equal to \(1\), then the index is in \(\{2^{k-1},\ldots,2^k-1\}\), in which case the result of \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) is the same as \(b=\ensuremath{\mathit{LOOKUP}}_{k-1}(x_{2^{k-1}},\ldots,x_{2^k-1},i_1,\ldots,i_{k-1})\). Thus we can compute \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) by first computing \(a\) and \(b\) and then outputting \(\ensuremath{\mathit{IF}}(i_{k-1},a,b)\).

Proof of Theorem 4.9 from Lemma 4.10. Now that we have Lemma 4.10, we can complete the proof of Theorem 4.9. We will prove by induction on \(k\) that there is a NAND-CIRC program of at most \(4\cdot 2^k\) lines for \(\ensuremath{\mathit{LOOKUP}}_k\). For \(k=1\) this follows by the four line program for \(\ensuremath{\mathit{IF}}\) we’ve seen before. For \(k>1\), we use the following pseudocode

If we let \(L(k)\) be the number of lines required for \(\ensuremath{\mathit{LOOKUP}}_k\), then the above pseudo-code shows that

\[ L(k) \leq 2L(k-1)+4 \;. \;\;(4.3) \]
which solves for \(L(k) \leq 4(2^k-1)\). See Figure 4.7 for a plot of the actual number of lines in our implementation of \(\ensuremath{\mathit{LOOKUP}}_k\).

4.7: The number of lines in our implementation of the LOOKUP_k function as a function of k (i.e., the length of the index). The number of lines in our implementation is roughly 3 \cdot 2^k.
4.7: The number of lines in our implementation of the LOOKUP_k function as a function of \(k\) (i.e., the length of the index). The number of lines in our implementation is roughly \(3 \cdot 2^k\).

Computing every function

At this point we know the following facts about NAND-CIRC programs (and so equivalently about Boolean circuits and our other equivalent models):

  1. They can compute at least some non trivial functions.

  2. Coming up with NAND-CIRC programs for various functions is a very tedious task.

Thus I would not blame the reader if they were not particularly looking forward to a long sequence of examples of functions that can be computed by NAND-CIRC programs. However, it turns out we are not going to need this, as we can show in one fell swoop that NAND-CIRC programs can compute every finite function:

There exists some constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a NAND-CIRC program with at most \(c \cdot m 2^n\) lines that computes the function \(f\) .

By Theorem 3.18, the models of NAND circuits, NAND-CIRC programs, AON-CIRC programs, and Boolean circuits, are all equivalent to one another, and hence Theorem 4.11 holds for all these models. In particular, the following theorem is equivalent to Theorem 4.11:

There exists some constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a Boolean circuit with at most \(c \cdot m 2^n\) gates that computes the function \(f\) .

Every finite function can be computed by a large enough Boolean circuit.

Improved bounds. Though it will not be of great importance to us, it is possible to improve on the proof of Theorem 4.11 and shave an extra factor of \(n\), as well as optimize the constant \(c\), and so prove that for every \(\epsilon>0\), \(m\in \N\) and sufficiently large \(n\), if \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) then \(f\) can be computed by a NAND circuit of at most \((1+\epsilon)\tfrac{m\cdot 2^n}{n}\) gates. The proof of this result is beyond the scope of this book, but we do discuss how to obtain a bound of the form \(O(\tfrac{m \cdot 2^n}{n})\) in Subsection 4.4.2; see also the biographical notes.

Proof of NAND’s Universality

To prove Theorem 4.11, we need to give a NAND circuit, or equivalently a NAND-CIRC program, for every possible function. We will restrict our attention to the case of Boolean functions (i.e., \(m=1\)). Exercise 4.9 asks you to extend the proof for all values of \(m\). A function \(F: \{0,1\}^n\rightarrow \{0,1\}\) can be specified by a table of its values for each one of the \(2^n\) inputs. For example, the table below describes one particular function \(G: \{0,1\}^4 \rightarrow \{0,1\}\):3

An example of a function \(G:\{0,1\}^4 \rightarrow \{0,1\}\). { .table #tablefunctiong }
Input (\(x\)) Output (\(G(x)\))
\(0000\) 1
\(0001\) 1
\(0010\) 0
\(0011\) 0
\(0100\) 1
\(0101\) 0
\(0110\) 0
\(0111\) 1
\(1000\) 0
\(1001\) 0
\(1010\) 0
\(1011\) 0
\(1100\) 1
\(1101\) 1
\(1110\) 1
\(1111\) 1

For every \(x\in \{0,1\}^4\), \(G(x)=\ensuremath{\mathit{LOOKUP}}_4(1100100100001111,x)\), and so the following is NAND-CIRC “pseudocode” to compute \(G\) using syntactic sugar for the LOOKUP_4 procedure.

We can translate this pseudocode into an actual NAND-CIRC program by adding three lines to define variables zero and one that are initialized to \(0\) and \(1\) respectively, and then replacing a statement such as Gxxx = 0 with Gxxx = NAND(one,one) and a statement such as Gxxx = 1 with Gxxx = NAND(zero,zero). The call to LOOKUP_4 will be replaced by the NAND-CIRC program that computes \(\ensuremath{\mathit{LOOKUP}}_4\), plugging in the appropriate inputs.

There was nothing about the above reasoning that was particular to the function \(G\) of ?? ??. Given every function \(F: \{0,1\}^n \rightarrow \{0,1\}\), we can write a NAND-CIRC program that does the following:

  1. Initialize \(2^n\) variables of the form F00...0 till F11...1 so that for every \(z\in\{0,1\}^n\), the variable corresponding to \(z\) is assigned the value \(F(z)\).

  2. Compute \(\ensuremath{\mathit{LOOKUP}}_n\) on the \(2^n\) variables initialized in the previous step, with the index variable being the input variables X[\(0\) ],…,X[\(2^n-1\) ]. That is, just like in the pseudocode for G above, we use Y[0] = LOOKUP(F00..00,...,F11..1,X[0],..,x[\(n-1\)])

The total number of lines in the resulting program is \(3+2^n\) lines for initializing the variables plus the \(4\cdot 2^n\) lines that we pay for computing \(\ensuremath{\mathit{LOOKUP}}_n\). This completes the proof of Theorem 4.11.

While Theorem 4.11 seems striking at first, in retrospect, it is perhaps not that surprising that every finite function can be computed with a NAND-CIRC program. After all, a finite function \(F: \{0,1\}^n \rightarrow \{0,1\}^m\) can be represented by simply the list of its outputs for each one of the \(2^n\) input values. So it makes sense that we could write a NAND-CIRC program of similar size to compute it. What is more interesting is that some functions, such as addition and multiplication, have a much more efficient representation: one that only requires \(O(n^2)\) or even fewer lines.

Improving by a factor of \(n\) (optional)

By being a little more careful, we can improve the bound of Theorem 4.11 and show that every function \(F:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a NAND-CIRC program of at most \(O(m 2^n/n)\) lines. In other words, we can prove the following improved version:

There exists a constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a NAND-CIRC program with at most \(c \cdot m 2^n / n\) lines that computes the function \(f\) .

As before, it is enough to prove the case that \(m=1\).

The idea is to use the technique known as memoization. Let \(k= \log(n-2\log n)\) (the reasoning behind this choice will become clear later on). For every \(a \in \{0,1\}^{n-k}\) we define \(F_a:\{0,1\}^k \rightarrow \{0,1\}\) to be the function that maps \(w_0,\ldots,w_{k-1}\) to \(F(a_0,\ldots,a_{n-k-1},w_0,\ldots,w_{k-1})\).

On input \(x=x_0,\ldots,x_{n-1}\), we can compute \(F(x)\) as follows: First we compute a \(2^{n-k}\) long string \(P\) whose \(a^{th}\) entry (identifying \(\{0,1\}^{n-k}\) with \([2^{n-k}]\)) equals \(F_a(x_{n-k},\ldots,x_{n-1})\). One can verify that \(F(x)=\ensuremath{\mathit{LOOKUP}}_{n-k}(P,x_0,\ldots,x_{n-k-1})\). Since we can compute \(\ensuremath{\mathit{LOOKUP}}_{n-k}\) using \(O(2^{n-k})\) lines, if we can compute the string \(P\) (i.e., compute variables P_\(0\), …, P_\(2^{n-k}-1\)) using \(T\) lines, then we can compute \(F\) in \(O(2^{n-k})+T\) lines.

The trivial way to compute the string \(P\) would be to use \(O(2^k)\) lines to compute for every \(a\) the map \(x_0,\ldots,x_{k-1} \mapsto F_a(x_0,\ldots,x_{k-1})\) as in the proof of Theorem 4.11. Since there are \(2^{n-k}\) \(a\)’s, that would be a total cost of \(O(2^{n-k} \cdot 2^k) = O(2^n)\) which would not improve at all on the bound of Theorem 4.11. However, a more careful observation shows that we are making some redundant computations. After all, there are only \(2^{2^k}\) distinct functions mapping \(k\) bits to one bit. If \(a\) and \(a'\) satisfy that \(F_a = F_{a'}\) then we don’t need to spend \(2^k\) lines computing both \(F_a(x)\) and \(F_{a'}(x)\) but rather can only compute the variable P_\(a\) and then copy P_\(a\) to P_\(a'\) using \(O(1)\) lines. Since we have \(2^{2^k}\) unique functions, we can bound the total cost to compute \(P\) by \(O(2^{2^k}2^k)+O(2^{n-k})\).

Now it just becomes a matter of calculation. By our choice of \(k\), \(2^k = n-2\log n\) and hence \(2^{2^k}=\tfrac{2^n}{n^2}\). Since \(n/2 \leq 2^k \leq n\), we can bound the total cost of computing \(F(x)\) (including also the additional \(O(2^{n-k})\) cost of computing \(\ensuremath{\mathit{LOOKUP}}_{n-k}\)) by \(O(\tfrac{2^n}{n^2}\cdot n)+O(2^n/n)\), which is what we wanted to prove.

Using the connection between NAND-CIRC programs and Boolean circuits, an immediate corollary of Theorem 4.14 is the following improvement to Theorem 4.12:

There exists some constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a Boolean circuit with at most \(c \cdot m 2^n / n\) gates that computes the function \(f\) .

Computing every function: An alternative proof

Theorem 4.12 is a fundamental result in the theory (and practice!) of computation. In this section we present an alternative proof of this basic fact that Boolean circuits can compute every finite function. This alternative proof gives somewhat worse quantitative bound on the number of gates but it has the advantage of being simpler, working directly with circuits and avoiding the usage of all the syntactic sugar machinery. (However, that machinery is useful in its own right, and will find other applications later on.)

There exists some constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a Boolean circuit with at most \(c \cdot m\cdot n 2^n\) gates that computes the function \(f\) .

4.8: Given a function f:\{0,1\}^n \rightarrow \{0,1\}, we let \{ x_0, x_1, \ldots, x_{N-1} \} \subseteq \{0,1\}^n be the set of inputs such that f(x_i)=1, and note that N \leq 2^n. We can express f as the OR of \delta_{x_i} for i\in [N] where the function \delta_\alpha:\{0,1\}^n \rightarrow \{0,1\} (for \alpha \in \{0,1\}^n) is defined as follows: \delta_\alpha(x)=1 iff x=\alpha. We can compute the OR of N values using N two-input OR gates. Therefore if we have a circuit of size O(n) to compute \delta_\alpha for every \alpha \in \{0,1\}^n, we can compute f using a circuit of size O(n \cdot N) = O(n \cdot 2^n).
4.8: Given a function \(f:\{0,1\}^n \rightarrow \{0,1\}\), we let \(\{ x_0, x_1, \ldots, x_{N-1} \} \subseteq \{0,1\}^n\) be the set of inputs such that \(f(x_i)=1\), and note that \(N \leq 2^n\). We can express \(f\) as the OR of \(\delta_{x_i}\) for \(i\in [N]\) where the function \(\delta_\alpha:\{0,1\}^n \rightarrow \{0,1\}\) (for \(\alpha \in \{0,1\}^n\)) is defined as follows: \(\delta_\alpha(x)=1\) iff \(x=\alpha\). We can compute the OR of \(N\) values using \(N\) two-input OR gates. Therefore if we have a circuit of size \(O(n)\) to compute \(\delta_\alpha\) for every \(\alpha \in \{0,1\}^n\), we can compute \(f\) using a circuit of size \(O(n \cdot N) = O(n \cdot 2^n)\).

The idea of the proof is illustrated in Figure 4.8. As before, it is enough to focus on the case that \(m=1\) (the function \(f\) has a single output), since we can always extend this to the case of \(m>1\) by looking at the composition of \(m\) circuits each computing a different output bit of the function \(f\). We start by showing that for every \(\alpha \in \{0,1\}^n\), there is an \(O(n)\) sized circuit that computes the function \(\delta_\alpha:\{0,1\}^n \rightarrow \{0,1\}\) defined as follows: \(\delta_\alpha(x)=1\) iff \(x=\alpha\) (that is, \(\delta_\alpha\) outputs \(0\) on all inputs except the input \(\alpha\)). We can then write any function \(f:\{0,1\}^n \rightarrow \{0,1\}\) as the OR of at most \(2^n\) functions \(\delta_\alpha\) for the \(\alpha\)’s on which \(f(\alpha)=1\).

We prove the theorem for the case \(m=1\). The result can be extended for \(m>1\) as before (see also Exercise 4.9). Let \(f:\{0,1\}^n \rightarrow \{0,1\}\). We will prove that there is an \(O(n\cdot 2^n)\)-sized Boolean circuit to compute \(f\) in the following steps:

  1. We show that for every \(\alpha\in \{0,1\}^n\), there is an \(O(n)\) sized circuit that computes the function \(\delta_\alpha:\{0,1\}^n \rightarrow \{0,1\}\), where \(\delta_\alpha(x)=1\) iff \(x=\alpha\).

  2. We then show that this implies the existence of an \(O(n\cdot 2^n)\)-sized circuit that computes \(f\), by writing \(f(x)\) as the OR of \(\delta_\alpha(x)\) for all \(\alpha\in \{0,1\}^n\) such that \(f(\alpha)=1\).

We start with Step 1:

CLAIM: For \(\alpha \in \{0,1\}^n\), define \(\delta_\alpha:\{0,1\}^n\) as follows:

\[ \delta_\alpha(x) = \begin{cases}1 & x=\alpha \\ 0 & \text{otherwise} \end{cases} \;. \]
then there is a Boolean circuit using at most \(2n\) gates that computes \(\delta_\alpha\).

PROOF OF CLAIM: The proof is illustrated in Figure 4.9. As an example, consider the function \(\delta_{011}:\{0,1\}^3 \rightarrow \{0,1\}\). This function outputs \(1\) on \(x\) if and only if \(x_0=0\), \(x_1=1\) and \(x_2=1\), and so we can write \(\delta_{011}(x) = \overline{x_0} \wedge x_1 \wedge x_2\), which translates into a Boolean circuit with one NOT gate and two AND gates. More generally, for every \(\alpha \in \{0,1\}^n\), we can express \(\delta_{\alpha}(x)\) as \((x_0 = \alpha_0) \wedge (x_1 = \alpha_1) \wedge \cdots \wedge (x_{n-1} = \alpha_{n-1})\), where if \(\alpha_i=0\) we replace \(x_i = \alpha_i\) with \(\overline{x_i}\) and if \(\alpha_i=1\) we replace \(x_i=\alpha_i\) by simply \(x_i\). This yields a circuit that computes \(\delta_\alpha\) using \(n\) AND gates and at most \(n\) NOT gates, so a total of at most \(2n\) gates.

Now for every function \(f:\{0,1\}^n \rightarrow \{0,1\}\), we can write

\[ f(x) = \delta_{x_0}(x) \vee \delta_{x_1}(x) \vee \cdots \vee \delta_{x_{N-1}}(x) \;\;(4.5) \]

where \(S=\{ x_0 ,\ldots, x_{N-1}\}\) is the set of inputs on which \(f\) outputs \(1\). (To see this, you can verify that the right-hand side of Equation 4.5 evaluates to \(1\) on \(x\in \{0,1\}^n\) if and only if \(x\) is in the set \(S\).)

Therefore we can compute \(f\) using a Boolean circuit of at most \(2n\) gates for each of the \(N\) functions \(\delta_{x_i}\) and combine that with at most \(N\) OR gates, thus obtaining a circuit of at most \(2n\cdot N + N\) gates. Since \(S \subseteq \{0,1\}^n\), its size \(N\) is at most \(2^n\) and hence the total number of gates in this circuit is \(O(n\cdot 2^n)\).

4.9: For every string \alpha\in \{0,1\}^n, there is a Boolean circuit of O(n) gates to compute the function \delta_\alpha:\{0,1\}^n \rightarrow \{0,1\} such that \delta_\alpha(x)=1 if and only if x=\alpha. The circuit is very simple. Given input x_0,\ldots,x_{n-1} we compute the AND of z_0,\ldots,z_{n-1} where z_i=x_i if \alpha_i=1 and z_i = \ensuremath{\mathit{NOT}}(x_i) if \alpha_i=0. While formally Boolean circuits only have a gate for computing the AND of two inputs, we can implement an AND of n inputs by composing n two-input ANDs.
4.9: For every string \(\alpha\in \{0,1\}^n\), there is a Boolean circuit of \(O(n)\) gates to compute the function \(\delta_\alpha:\{0,1\}^n \rightarrow \{0,1\}\) such that \(\delta_\alpha(x)=1\) if and only if \(x=\alpha\). The circuit is very simple. Given input \(x_0,\ldots,x_{n-1}\) we compute the AND of \(z_0,\ldots,z_{n-1}\) where \(z_i=x_i\) if \(\alpha_i=1\) and \(z_i = \ensuremath{\mathit{NOT}}(x_i)\) if \(\alpha_i=0\). While formally Boolean circuits only have a gate for computing the AND of two inputs, we can implement an AND of \(n\) inputs by composing \(n\) two-input ANDs.

The class \(\ensuremath{\mathit{SIZE}}_{n,m}(T)\)

We have seen that every function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a circuit of size \(O(m\cdot 2^n)\), and some functions (such as addition and multiplication) can be computed by much smaller circuits. This motivates the following definition: we define \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) to be the set of all functions from \(\{0,1\}^n\) to \(\{0,1\}^m\) that can be computed by NAND circuits of at most \(s\) gates (or equivalently, by NAND-CIRC programs of at most \(s\) lines). In other words, \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) is defined as follows:

Let \(n,m,s \in \N\) be numbers with \(s \geq m\). The set \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) denotes the set of all functions \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) such that there exists a NAND circuit of at most \(s\) gates that computes \(f\). We denote by \(\ensuremath{\mathit{SIZE}}_n(s)\) the set \(\ensuremath{\mathit{SIZE}}_{n,1}(s)\).

Figure 4.10 depicts the sets \(\ensuremath{\mathit{SIZE}}_{n,1}(s)\), note that \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) is a set of functions, not of programs! (asking if a program or a circuit is a member of \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) is a category error as in the sense of Figure 4.11).

4.10: There are 2^{2^n} functions mapping \{0,1\}^n to \{0,1\}, and an infinite number of circuits with n bit inputs and a single bit of output. Every circuit computes one function, but every function can be computed by many circuits. We say that f \in \ensuremath{\mathit{SIZE}}_{n,1}(s) if the smallest circuit that computes f has s or fewer gates. For example \ensuremath{\mathit{XOR}}_n \in \ensuremath{\mathit{SIZE}}_{n,1}(4n).  shows that every function g is computable by some circuit of at most c\cdot 2^n/n gates, and hence \ensuremath{\mathit{SIZE}}_{n,1}(c\cdot 2^n/n) corresponds to the set of all functions from \{0,1\}^n to \{0,1\}.
4.10: There are \(2^{2^n}\) functions mapping \(\{0,1\}^n\) to \(\{0,1\}\), and an infinite number of circuits with \(n\) bit inputs and a single bit of output. Every circuit computes one function, but every function can be computed by many circuits. We say that \(f \in \ensuremath{\mathit{SIZE}}_{n,1}(s)\) if the smallest circuit that computes \(f\) has \(s\) or fewer gates. For example \(\ensuremath{\mathit{XOR}}_n \in \ensuremath{\mathit{SIZE}}_{n,1}(4n)\). Theorem 4.11 shows that every function \(g\) is computable by some circuit of at most \(c\cdot 2^n/n\) gates, and hence \(\ensuremath{\mathit{SIZE}}_{n,1}(c\cdot 2^n/n)\) corresponds to the set of all functions from \(\{0,1\}^n\) to \(\{0,1\}\).

While we defined \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) with respect to NAND gates, we would get essentially the same class if we defined it with respect to AND/OR/NOT gates:

Let \(\ensuremath{\mathit{SIZE}}^{AON}_{n,m,s}\) denote the set of all functions \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) that can be computed by an AND/OR/NOT Boolean circuit of at most \(s\) gates. Then,

\[ \ensuremath{\mathit{SIZE}}_{n,m}(s/2) \subseteq \ensuremath{\mathit{SIZE}}^{AON}_{n,m}(s) \subseteq \ensuremath{\mathit{SIZE}}_{n,m}(3s) \]

If \(f\) can be computed by a NAND circuit of at most \(s/2\) gates, then by replacing each NAND with the two gates NOT and AND, we can obtain an AND/OR/NOT Boolean circuit of at most \(s\) gates that computes \(f\). On the other hand, if \(f\) can be computed by a Boolean AND/OR/NOT circuit of at most \(s\) gates, then by Theorem 3.11 it can be computed by a NAND circuit of at most \(3s\) gates.

4.11: A “category error” is a question such as “is a cucumber even or odd?” which does not even make sense. In this book one type of category errors you should watch out for is confusing functions and programs (i.e., confusing specifications and implementations). If C is a circuit or program, then asking if C \in \ensuremath{\mathit{SIZE}}_{n,1}(s) is a category error, since \ensuremath{\mathit{SIZE}}_{n,1}(s) is a set of functions and not programs or circuits.
4.11: A “category error” is a question such as “is a cucumber even or odd?” which does not even make sense. In this book one type of category errors you should watch out for is confusing functions and programs (i.e., confusing specifications and implementations). If \(C\) is a circuit or program, then asking if \(C \in \ensuremath{\mathit{SIZE}}_{n,1}(s)\) is a category error, since \(\ensuremath{\mathit{SIZE}}_{n,1}(s)\) is a set of functions and not programs or circuits.

The results we have seen in this chapter can be phrased as showing that \(\ensuremath{\mathit{ADD}}_n \in \ensuremath{\mathit{SIZE}}_{2n,n+1}(100 n)\) and \(\ensuremath{\mathit{MULT}}_n \in \ensuremath{\mathit{SIZE}}_{2n,2n}(10000 n^{\log_2 3})\). Theorem 4.11 shows that for some constant \(c\), \(\ensuremath{\mathit{SIZE}}_{n,m}(c m 2^n)\) is equal the set of all functions from \(\{0,1\}^n\) to \(\{0,1\}^m\).

Note that \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) does not correspond to a set of programs! Rather, it is a set of functions (see Figure 4.11). As we discussed in Subsection 3.6.2 (and Subsection 2.5.1), the distinction between programs and functions is absolutely crucial. You should always remember that while a program computes a function, it is not equal to a function. In particular, as we’ve seen, there can be more than one program to compute the same function.

A NAND-CIRC program \(P\) can only compute a function with a certain number \(n\) of inputs and a certain number \(m\) of outputs. Hence for example there is no single NAND-CIRC program that can compute the increment function \(\ensuremath{\mathit{INC}}:\{0,1\}^* \rightarrow \{0,1\}^*\) that maps a string \(x\) (which we identify with a number via the binary representation) to the string that represents \(x+1\). Rather for every \(n>0\), there is a NAND-CIRC program \(P_n\) that computes the restriction \(\ensuremath{\mathit{INC}}_n\) of the function \(\ensuremath{\mathit{INC}}\) to inputs of length \(n\). Since it can be shown that for every \(n>0\) such a program \(P_n\) exists of length at most \(10n\), \(\ensuremath{\mathit{INC}}_n \in \ensuremath{\mathit{SIZE}}(10n)\) for every \(n>0\).

If \(T:\N \rightarrow \N\) and \(F:\{0,1\}^* \rightarrow \{0,1\}^*\), we will sometimes slightly abuse notation and write \(F \in \ensuremath{\mathit{SIZE}}(T(n))\) to indicate that for every \(n\) the restriction \(F_{\upharpoonright n}\) of \(F\) to inputs in \(\{0,1\}^n\) is in \(\ensuremath{\mathit{SIZE}}(T(n))\). Hence we can write \(\ensuremath{\mathit{INC}} \in \ensuremath{\mathit{SIZE}}(10n)\). We will come back to this issue of finite vs infinite functions later in this course.

In this exercise we prove a certain “closure property” of the class \(\ensuremath{\mathit{SIZE}}_n(s)\). That is, we show that if \(f\) is in this class then (up to some small additive term) so is the complement of \(f\), which is the function \(g(x)=1-f(x)\).

Prove that there is a constant \(c\) such that for every \(f:\{0,1\}^n \rightarrow \{0,1\}\) and \(s\in \N\), if \(f \in \ensuremath{\mathit{SIZE}}_n(s)\) then \(1-f \in \ensuremath{\mathit{SIZE}}_n(s+c)\).

If \(f\in \ensuremath{\mathit{SIZE}}(s)\) then there is an \(s\)-line NAND-CIRC program \(P\) that computes \(f\). We can rename the variable Y[0] in \(P\) to a variable temp and add the line

at the very end to obtain a program \(P'\) that computes \(1-f\).

  • We can define the notion of computing a function via a simplified “programming language”, where computing a function \(F\) in \(T\) steps would correspond to having a \(T\)-line NAND-CIRC program that computes \(F\).
  • While the NAND-CIRC programming only has one operation, other operations such as functions and conditional execution can be implemented using it.
  • Every function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a circuit of at most \(O(m 2^n)\) gates (and in fact at most \(O(m 2^n/n)\) gates).
  • Sometimes (or maybe always?) we can translate an efficient algorithm to compute \(f\) into a circuit that computes \(f\) with a number of gates comparable to the number of steps in this algorithm.

Exercises

This exercise asks you to give a one-to-one map from \(\N^2\) to \(\N\). This can be useful to implement two-dimensional arrays as “syntactic sugar” in programming languages that only have one-dimensional array.

  1. Prove that the map \(F(x,y)=2^x3^y\) is a one-to-one map from \(\N^2\) to \(\N\).

  2. Show that there is a one-to-one map \(F:\N^2 \rightarrow \N\) such that for every \(x,y\), \(F(x,y) \leq 100\cdot \max\{x,y\}^2+100\).

  3. For every \(k\), show that there is a one-to-one map \(F:\N^k \rightarrow \N\) such that for every \(x_0,\ldots,x_{k-1} \in \N\), \(F(x_0,\ldots,x_{k-1}) \leq 100 \cdot (x_0+x_1+\ldots+x_{k-1}+100k)^k\).

Prove that the NAND-CIRC program below computes the function \(\ensuremath{\mathit{MUX}}\) (or \(\ensuremath{\mathit{LOOKUP}}_1\)) where \(\ensuremath{\mathit{MUX}}(a,b,c)\) equals \(a\) if \(c=0\) and equals \(b\) if \(c=1\):

Give a NAND-CIRC program of at most 6 lines to compute the function \(\ensuremath{\mathit{MAJ}}:\{0,1\}^3 \rightarrow \{0,1\}\) where \(\ensuremath{\mathit{MAJ}}(a,b,c) = 1\) iff \(a+b+c \geq 2\).

In this exercise we will explore Theorem 4.5: transforming NAND-CIRC-IF programs that use code such as if .. then .. else .. to standard NAND-CIRC programs.

  1. Give a “proof by code” of Theorem 4.5: a program in a programming language of your choice that transforms a NAND-CIRC-IF program \(P\) into a “sugar free” NAND-CIRC program \(P'\) that computes the same function. See footnote for hint.4

  2. Prove the following statement, which is the heart of Theorem 4.5: suppose that there exists an \(s\)-line NAND-CIRC program to compute \(f:\{0,1\}^n \rightarrow \{0,1\}\) and an \(s'\)-line NAND-CIRC program to compute \(g:\{0,1\}^n \rightarrow \{0,1\}\). Prove that there exist a NAND-CIRC program of at most \(s+s'+10\) lines to compute the function \(h:\{0,1\}^{n+1} \rightarrow \{0,1\}\) where \(h(x_0,\ldots,x_{n-1},x_n)\) equals \(f(x_0,\ldots,x_{n-1})\) if \(x_n=0\) and equals \(g(x_0,\ldots,x_{n-1})\) otherwise. (All programs in this item are standard “sugar-free” NAND-CIRC programs.)

  1. A half adder is the function \(\ensuremath{\mathit{HA}}:\{0,1\}^2 :\rightarrow \{0,1\}^2\) that corresponds to adding two binary bits. That is, for every \(a,b \in \{0,1\}\), \(\ensuremath{\mathit{HA}}(a,b)= (e,f)\) where \(2e+f = a +b\). Prove that there is a NAND circuit of at most five NAND gates that computes \(\ensuremath{\mathit{HA}}\).

  2. A full adder is the function \(\ensuremath{\mathit{FA}}:\{0,1\}^3 \rightarrow \{0,1\}\) that takes in two bits and a “carry” bit and outputs their sum. That is, for every \(a,b,c \in \{0,1\}\), FA(a,b,c) = (e,f)$ such that \(2e+f = a+b+c\). Prove that there is a NAND circuit of at most nine NAND gates that computes \(\ensuremath{\mathit{FA}}\).

  3. Prove that if there is a NAND circuit of \(c\) gates that computes \(\ensuremath{\mathit{FA}}\), then there is a circuit of \(cn\) gates that computes \(\ensuremath{\mathit{ADD}}_n\) where (as in Theorem 4.6) \(\ensuremath{\mathit{ADD}}_n:\{0,1\}^{2n} \rightarrow \{0,1\}^n\) is the function that outputs the addition of two input \(n\)-bit numbers. See footnote for hint.5

  4. Show that for every \(n\) there is a NAND-CIRC program to compute \(\ensuremath{\mathit{ADD}}_n\) with at most \(9n\) lines.

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{ADD}}_n\). Can you ensure that the program it outputs for \(\ensuremath{\mathit{ADD}}_n\) has fewer than \(10n\) lines?

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{MULT}}_n\). Can you ensure that the program it outputs for \(\ensuremath{\mathit{MULT}}_n\) has fewer than \(1000\cdot n^2\) lines?

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{MULT}}_n\) and has at most \(10000 n^{1.9}\) lines.6 What is the smallest number of lines you can use to multiply two 2048 bit numbers?

In the text Theorem 4.11 is only proven for the case \(m=1\). In this exercise you will extend the proof for every \(m\).

Prove that

  1. If there is an \(s\)-line NAND-CIRC program to compute \(f:\{0,1\}^n \rightarrow \{0,1\}\) and an \(s'\)-line NAND-CIRC program to compute \(f':\{0,1\}^n \rightarrow \{0,1\}\) then there is an \(s+s'\)-line program to compute the function \(g:\{0,1\}^n \rightarrow \{0,1\}^2\) such that \(g(x)=(f(x),f'(x))\).

  2. For every function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\), there is a NAND-CIRC program of at most \(10m\cdot 2^n\) lines that computes \(f\). (You can use the \(m=1\) case of Theorem 4.11, as well as Item 1.)

Let \(P\) be the following NAND-CIRC program:

  1. Write a program \(P'\) with at most three lines of code that uses both NAND as well as the syntactic sugar OR that computes the same function as \(P\).

  2. Draw a circuit that computes the same function as \(P\) and uses only \(\ensuremath{\mathit{AND}}\) and \(\ensuremath{\mathit{NOT}}\) gates.

In the following exercises you are asked to compare the power of pairs programming languages. By “comparing the power” of two programming languages \(X\) and \(Y\) we mean determining the relation between the set of functions that are computable using programs in \(X\) and \(Y\) respectively. That is, to answer such a question you need to do both of:

  1. Either prove that for every program \(P\) in \(X\) there is a program \(P'\) in \(Y\) that computes the same function as \(P\), or give an example for a function that is computable by an \(X\)-program but not computable by a \(Y\)-program.

and

  1. Either prove that for every program \(P\) in \(Y\) there is a program \(P'\) in \(X\) that computes the same function as \(P\), or give an example for a function that is computable by a \(Y\)-program but not computable by an \(X\)-program.

When you give an example as above of a function that is computable in one programming language but not the other, you need to prove that the function you showed is (1) computable in the first programming language, (2) not computable in the second programming programming language.

Let IF-CIRC be the programming language where we have the following operations foo = 0, foo = 1, foo = IF(cond,yes,no) (that is, we can use the constants \(0\) and \(1\), and the \(\ensuremath{\mathit{IF}}:\{0,1\}^3 \rightarrow \{0,1\}\) function such that \(\ensuremath{\mathit{IF}}(a,b,c)\) equals \(b\) if \(a=1\) and equals \(c\) if \(a=0\)). Compare the power of the NAND-CIRC programming language and the IF-CIRC programming language.

Let XOR-CIRC be the programming language where we have the following operations foo = XOR(bar,blah), foo = 1 and bar = 0 (that is, we can use the constants \(0\), \(1\) and the \(\ensuremath{\mathit{XOR}}\) function that maps \(a,b \in \{0,1\}^2\) to \(a+b \mod 2\)). Compare the power of the NAND-CIRC programming language and the XOR-CIRC programming language. See footnote for hint.7

Prove that there is some constant \(c\) such that for every \(n>1\), \(\ensuremath{\mathit{MAJ}}_n \in Size(cn)\) where \(\ensuremath{\mathit{MAJ}}_n:\{0,1\}^n \rightarrow \{0,1\}\) is the majority function on \(n\) input bits. That is \(\ensuremath{\mathit{MAJ}}_n(x)=1\) iff \(\sum_{i=0}^{n-1}x_i > n/2\). NOTE: You can get 16 points by proving the weaker statement \(\ensuremath{\mathit{MAJ}}_n \in Size(c \cdot n \log n)\) for some constant \(c\). See footnote for hint.8

Prove that there is some constant \(c\) such that for every \(n>1\), and integers \(a_0,\ldots,a_{n-1},b \in \{-2^n,-2^n+1,\ldots,-1,0,+1,\ldots,2^n\}\), there is a NAND circuit with at most \(n^c\) gates that computes the threshold function \(f_{a_0,\ldots,a_{n-1},b}:\{0,1\}^n \rightarrow \{0,1\}\) that on input \(x\in \{0,1\}^n\) outputs \(1\) if and only if \(\sum_{i=0}^{n-1} a_i x_i > b\).

Bibliographical notes

See Jukna’s and Wegener’s books (Jukna, 2012) (Wegener, 1987) for much more extensive discussion on circuits. Shannon showed that every Boolean function can be computed by a circuit of exponential size (Shannon, 1938) . The improved bound of \(c \cdot 2^n/n\) (with the optimal value of \(c\) for many bases) is due to Lupanov (Lupanov, 1958) . An exposition of this for the case of NAND is given in Chapter 4 of his book (Lupanov, 1984) . (Thanks to Sasha Golovnev for tracking down this reference!)

The concept of “syntactic sugar” is also known as “macros” or “meta-programming” and is sometimes implemented via a preprocessor or macro language in a programming language or a text editor. One modern example is the Babel JavaScript syntax transformer, that converts JavaScript programs written using the latest features into a format that older Browsers can accept. It even has a plug-in architecture, that allows users to add their own syntactic sugar to the language. We mentioned that almost all programming language support user-defined functions, but one notable exception is the original version of the FORTRAN programming language, developed in the early 1950’s. This was however quickly added in FORTRAN II, released in 1958.

  1. This code uses regular expressions to make the search and replace parts a little easier. We will see the theoretical basis for regular expressions in Chapter 9.

  2. The value of \(c\) can be improved to \(9\), see Exercise 4.5.

  3. In case you are curious, this is the function on input \(i\in \{0,1\}^4\) (which we interpret as a number in \([16]\)), that outputs the \(i\)-th digit of \(\pi\) in the binary basis.

  4. You can start by transforming \(P\) into a NAND-CIRC-PROC program that uses procedure statements, and then use the code of Figure 4.3 to transform the latter into a “sugar free” NAND-CIRC program.

  5. Use a “cascade” of adding the bits one after the other, starting with the least significant digit, just like in the elementary-school algorithm.

  6. Hint: Use Karatsuba’s algorithm.

  7. You can use the fact that \((a+b)+c \mod 2 = a+b+c \mod 2\). In particular it means that if you have the lines d = XOR(a,b) and e = XOR(d,c) then e gets the sum modulo \(2\) of the variable a, b and c.

  8. One approach to solve this is using recursion and the so-called Master Theorem.

Comments

Comments are posted on the GitHub repository using the utteranc.es app. A GitHub login is required to comment. If you don't want to authorize the app to post on your behalf, you can also comment directly on the GitHub issue for this page.

Compiled on 09/12/2019 09:42:10

Copyright 2019, Boaz Barak. Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Produced using pandoc and panflute with templates derived from gitbook and bookdown.