See any bugs/typos/confusing explanations? Open a GitHub issue. You can also comment below

★ See also the **PDF version of this chapter** (better formatting/references) ★

# Syntactic sugar, and computing every function

- Get comfort with syntactic sugar or automatic translation of higher level logic to low level gates.

- Learn proof of major result: every finite function can be computed by a Boolean circuit.

- Start thinking
*quantitatively*about number of lines required for computation.

“[In 1951] I had a running compiler and nobody would touch it because, they carefully told me, computers could only do arithmetic; they could not do programs.”, Grace Murray Hopper, 1986.

“Syntactic sugar causes cancer of the semicolon.”, Alan Perlis, 1982.

The NAND-CIRC programing language is pretty much as “bare bones” as programming languages come. After all, it only has a single operation. But, it turns out we can implement some “added features” on top of it. That is, we can show how we can implement those features using the underlying mechanisms of the language.

For example, some calculations can more convenient to express using AND, OR and NOT than with NAND, but we can always translate code such as

into the valid NAND-CIRC code:

Thus in describing NAND-CIRC programs we can (and will) allow ourselves to use operations such as OR, with the understanding that in actual programs we will replace every line of the first form with the three lines of the second form. In programming language parlance this is known as *“syntactic sugar”*, since we are not changing the definition of the language, but merely introducing some convenient notational shortcuts.^{1} We will use several such “syntactic sugar” constructs to make our descriptions of NAND-CIRC programs shorter and simpler. However, these descriptions are merely shorthand for the equivalent standard or “sugar free” NAND-CIRC program that is obtained after removing the use of all these constructs. In particular, when we say that a function \(f\) has an \(s\)-line NAND-CIRC program, we mean a *standard* NAND-CIRC program, that does not use any syntactic sugar.

## Some examples syntactic sugar

Here are some examples of “syntactic sugar” that we can use in constructing NAND-CIRC programs. This is not an exhaustive list - if you find yourself needing to use an extra feature in your NAND-CIRC program then you can just show how to implement it based on the existing ones. Going over examples for syntatic sugar can be a little tedious, but we do it for two reasons:

To convince you that despite its seeming simplicity and limitations, the NAND-CIRC programming language is actually quite powerful and can capture many of the fancy programming constructs such as

`if`

statements and function definitions that exists in more fashionable languages.So you can realize how lucky you are to be taking a theory of computation course and not a compilers course…

`:)`

### Constants

We can create variables `zero`

and `one`

that have the values \(0\) and \(1\) respectively by adding the lines

Note that since for every \(x\in \{0,1\}\), \(\ensuremath{\mathit{NAND}}(x,\overline{x})=1\), the variable `one`

will get the value \(1\) regardless of the value of \(x_0\), and the variable `zero`

will get the value \(\ensuremath{\mathit{NAND}}(1,1)=0\).

### Functions / Macros

Another staple of almost any programming language is the ability to execute *functions*. However, we can achieve the same effect as (non recursive) functions using the time honored technique of “copy and paste”. That is, we can replace code which defines a macro

with the following code where we “paste” the code of `Func`

and where `function_code'`

is obtained by replacing all occurrences of `a`

with `d`

,`b`

with `e`

, `c`

with `f`

. When doing that we will need to ensure that all other variables appearing in `function_code'`

don’t interfere with other variables. We can always do so by renaming variables to new names that were not used before.

### Example: Computing Majority via NAND’s

Function definitions allow us to express NAND-CIRC programs much more cleanly and succinctly. For example, because we can compute AND,OR, NOT using NANDs, we can compute the *Majority* function as well.

```
def NOT(a): return NAND(a,a)
def AND(a,b): return NOT(NAND(a,b))
def OR(a,b): return NAND(NOT(a),NOT(b))
def MAJ(a,b,c):
return OR(OR(AND(a,b),AND(b,c)),AND(a,c))
print(MAJ(0,1,1))
# 1
```

This is certainly much more pleasant than the full NAND alternative:

```
Temp[0] = NAND(X[0],X[1])
Temp[1] = NAND(Temp[0],Temp[0])
Temp[2] = NAND(X[1],X[2])
Temp[3] = NAND(Temp[2],Temp[2])
Temp[4] = NAND(Temp[1],Temp[1])
Temp[5] = NAND(Temp[3],Temp[3])
Temp[6] = NAND(Temp[4],Temp[5])
Temp[7] = NAND(X[0],X[2])
Temp[8] = NAND(Temp[7],Temp[7])
Temp[9] = NAND(Temp[6],Temp[6])
Temp[10] = NAND(Temp[8],Temp[8])
Y[0] = NAND(Temp[9],Temp[10])
```

### Conditional statements

Another sorely missing feature in NAND is a conditional statement such as the `if`

/`then`

constructs that are found in many programming languages. However, using functions, we can obtain an ersatz if/then construct. First we can compute the function \(\ensuremath{\mathit{IF}}:\{0,1\}^3 \rightarrow \{0,1\}\) such that \(\ensuremath{\mathit{IF}}(a,b,c)\) equals \(b\) if \(a=1\) and \(c\) if \(a=0\).

Before reading onwards, try to see how you could compute the \(\ensuremath{\mathit{IF}}\) function using \(\ensuremath{\mathit{NAND}}\)’s. Once you you do that, see how you can use that to emulate `if`

/`then`

types of constructs.

The \(\ensuremath{\mathit{IF}}\) function can be implemented from NANDs as follows (see Exercise 4.2):

```
def IF(cond,a,b):
notcond = NAND(cond,cond)
temp = NAND(b,notcond)
temp1 = NAND(a,cond)
return NAND(temp,temp1)
print(IF(0,1,0))
# 0
print(IF(1,1,0))
# 1
```

The \(\ensuremath{\mathit{IF}}\) function is also known as the *multiplexing* function, since \(cond\) can be thought of as a switch that controls whether the output is connected to \(a\) or \(b\).

Using the \(\ensuremath{\mathit{IF}}\) function, we can implement conditionals in NAND. The idea is that we replace code of the form

with code of the form

that assigns to `foo`

its old value when `condition`

equals \(0\), and assign to `foo`

the value of `blah`

otherwise. More generally we can replace code of the form

with code of the form

```
temp_a = ...
temp_b = ...
temp_c = ...
a = IF(cond,temp_a,a)
b = IF(cond,temp_b,b)
c = IF(cond,temp_c,c)
```

## Extended example: Addition and Multiplicatoin (optional)

Using “syntactic sugar”, we can write the integer addition function as follows:^{2}

```
# Add two n-bit integers
def ADD(A,B):
Result = [0]*(n+1)
Carry = [0]*(n+1)
Carry[0] = zero(A[0])
for i in range(n):
Result[i] = XOR(Carry[i],XOR(A[i],B[i]))
Carry[i+1] = MAJ(Carry[i],A[i],B[i])
Result[n] = Carry[n]
return Result
ADD([1,1,1,0,0],[1,0,0,0,0]);;
# [0, 0, 0, 1, 0, 0]
```

where `zero`

is the constant zero function, and `MAJ`

and `XOR`

correspond to the majority and XOR functions respectively.

In the above we used the *loop* `for i in range(n)`

but we can expand this out by simply repeating the code \(n\) times, replacing the value of `i`

with \(0,1,2,\ldots,n-1\). The crucial point is that (unlike most programming languages) we do not allow the number of times the loop is executed to depend on the input, and so it is always possible to “expand out” the loop by simply copying the code the requisite number of times.

By expanding out all the features, for every value of \(n\) we can translate the above program into a standard (“sugar free”) NAND-CIRC program. Figure 4.1 depicts what we get for \(n=2\).

By going through the above program carefully and accounting for the number of gates, we can see that it yields a proof of the following theorem (see also Figure 4.2):

For every \(n\in \N\), let \(\ensuremath{\mathit{ADD}}_n:\{0,1\}^{2n}\rightarrow \{0,1\}^{n+1}\) be the function that, given \(x,x'\in \{0,1\}^n\) computes the representation of the sum of the numbers that \(x\) and \(x'\) represent. Then there is a constant \(c \leq 30\) such that for every \(n\) there is a NAND-CIRC program of at most \(c\) lines computing \(\ensuremath{\mathit{ADD}}_n\).^{3}

Once we have addition, we can use the grade-school algorithm to obtain multiplication as well, thus obtaining the following theorem:

For every \(n\), let \(\ensuremath{\mathit{MULT}}_n:\{0,1\}^{2n}\rightarrow \{0,1\}^{2n}\) be the function that, given \(x,x'\in \{0,1\}^n\) computes the representation of the product of the numbers that \(x\) and \(x'\) represent. Then there is a constant \(c\) such that for every \(n\), there is a NAND-CIRC program of at most \(cn^2\) that computes the function \(\ensuremath{\mathit{MULT}}_n\).

We omit the proof, though in Exercise 4.7 we ask you to supply a “constructive proof” in the form of a program (in your favorite programming language) that on input a number \(n\), outputs the code of a NAND-CIRC program of at most \(1000n^2\) lines that computes the \(\ensuremath{\mathit{MULT}}_n\) function. In fact, we can use Karatsuba’s algorithm to show that there is a NAND-CIRC program of \(O(n^{\log_2 3})\) lines to compute \(\ensuremath{\mathit{MULT}}_n\) (and one can even get further asymptotic improvements using the newer algorithms).

## The LOOKUP function

We have seen that NAND-CIRC programs can add and multiply numbers. But can they compute other type of functions, that have nothing to do with arithmetic? Here is one example:

For every \(k\), the *lookup* function \(\ensuremath{\mathit{LOOKUP}}_k: \{0,1\}^{2^k+k}\rightarrow \{0,1\}\) is defined as follows: For every \(x\in\{0,1\}^{2^k}\) and \(i\in \{0,1\}^k\), \[
\ensuremath{\mathit{LOOKUP}}_k(x,i)=x_i
\] where \(x_i\) denotes the \(i^{th}\) entry of \(x\), using the binary representation to identify \(i\) with a number in \(\{0,\ldots,2^k - 1 \}\).

The function \(\ensuremath{\mathit{LOOKUP}}_1: \{0,1\}^3 \rightarrow \{0,1\}\) maps \((x_0,x_1,i) \in \{0,1\}^3\) to \(x_i\). It is actually the same as the \(\ensuremath{\mathit{IF}}\)/\(\ensuremath{\mathit{MUX}}\) function we have seen above, that has a 4 line NAND-CIRC program. However, can we compute higher levels of \(\ensuremath{\mathit{LOOKUP}}\)? This turns out to be the case:

For every \(k\), there is a NAND-CIRC program that computes the function \(\ensuremath{\mathit{LOOKUP}}_k: \{0,1\}^{2^k+k}\rightarrow \{0,1\}\). Moreover, the number of lines in this program is at most \(4\cdot 2^k\).

As a corollary, for every \(k>0\), \(\ensuremath{\mathit{LOOKUP}}_k\) can be computed by a Boolean circuit (with AND, OR and NOT gates) of at most \(8 \cdot 2^k\).

### Constructing a NAND-CIRC program for \(\ensuremath{\mathit{LOOKUP}}\)

We now prove Theorem 4.4. The idea is actually quite simple. Consider the function \(\ensuremath{\mathit{LOOKUP}}_3 : \{0,1\}^{2^3+3} \rightarrow \{0,1\}\) that takes a an input of \(8+3=11\) bits and output a single bit. We can write this function in pseudocode as follows:

```
def LOOKUP_3(X[0],X[1],X[2],X[3],X[4],X[5],X[6],X[7],i[0],i[1],i[8]):
if i == (0,0,0): return X[0]
if i == (0,0,1): return X[1]
if i == (0,1,0): return X[2]
...
if i == (1,1,1): return X[7]
```

A condition such as `i==(0,1,0)`

can be expanded out to the AND of `NOT(i[0])`

, `i[1]`

and `NOT(i[2])`

and each one of these `AND`

and `NOT`

gates can be then translated into `NAND`

. The above can yield a proof of a version of Theorem 4.4 with a slightly larger number of gates, but if we are a little more careful we can prove the theorem with the number of gates as stated.

Specifically, we will prove Theorem 4.4 by induction. We will do so by induction. That is, we show how to use a NAND-CIRC program for computing \(\ensuremath{\mathit{LOOKUP}}_k\) to compute \(\ensuremath{\mathit{LOOKUP}}_{k+1}\). For the case \(k=1\), \(\ensuremath{\mathit{LOOKUP}}_1\) is the same as `IF`

for which we given a NAND-CIRC program with four line.

Now let us consider the case of \(k=2\). Given input \(x=(x_0,x_1,x_2,x_3)\) for \(\ensuremath{\mathit{LOOKUP}}_2\) and an index \(i=(i_0,i_1)\), if the most significant bit \(i_0\) of the index is \(0\) then \(\ensuremath{\mathit{LOOKUP}}_2(x,i)\) will equal \(x_0\) if \(i_1=0\) and equal \(x_1\) if \(i_1=1\). Similarly, if the most significant bit \(i_0\) is \(1\) then \(\ensuremath{\mathit{LOOKUP}}_2(x,i)\) will equal \(x_2\) if \(i_1=0\) and will equal \(x_3\) if \(i_1=1\). Another way to say this is that we can write \(\ensuremath{\mathit{LOOKUP}}_2\) as follows:

```
def LOOKUP2(X[0],X[1],X[2],X[3],i[0],i[1]):
if i[0]==1:
return LOOKUP1(X[2],X[3],i[1])
else:
return LOOKUP1(X[0],X[1],i[1])
```

or in other words,

```
def LOOKUP2(X[0],X[1],X[2],X[3],i[0],i[1]):
a = LOOKUP1(X[2],X[3],i[1])
b = LOOKUP1(X[0],X[1],i[1])
return IF( i[0],a,b)
```

Similarly, we can write

```
def LOOKUP3(X[0],X[1],X[2],X[3],X[4],X[5],X[6],X[7],i[0],i[1],i[2]):
a = LOOKUP2(X[3],X[4],X[5],X[6],i[1],i[2])
a = LOOKUP2(X[0],X[1],X[2],X[3],i[1],i[2])
return IF( i[0],a,b)
```

and so on and so forth. Generally, we can compute \(\ensuremath{\mathit{LOOKUP}}_k\) using two invocations of \(\ensuremath{\mathit{LOOKUP}}_{k-1}\) and one invocation of \(\ensuremath{\mathit{IF}}\), which yields the following lemma:

For every \(k \geq 2\), \(\ensuremath{\mathit{LOOKUP}}_k(x_0,\ldots,x_{2^k-1},i_0,\ldots,i_{k-1})\) is equal to \[ \ensuremath{\mathit{IF}}(i_0,\ensuremath{\mathit{LOOKUP}}_{k-1}(x_0,\ldots,x_{2^{k-1}-1},i_1,\ldots,i_{k-1}), \ensuremath{\mathit{LOOKUP}}_{k-1}(x_{2^{k-1}},\ldots,x_{2^k-1},i_1,\ldots,i_{k-1})) \]

If the most significant bit \(i_{0}\) of \(i\) is zero, then the index \(i\) is in \(\{0,\ldots,2^{k-1}-1\}\) and hence we can perform the lookup on the “first half” of \(x\) and the result of \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) will be the same as \(a=\ensuremath{\mathit{LOOKUP}}_{k-1}(x_0,\ldots,x_{2^{k-1}-1},i_1,\ldots,i_{k-1})\). On the other hand, if this most significant bit \(i_{0}\) is equal to \(1\), then the index is in \(\{2^{k-1},\ldots,2^k-1\}\), in which case the result of \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) is the same as \(b=\ensuremath{\mathit{LOOKUP}}_{k-1}(x_{2^{k-1}},\ldots,x_{2^k-1},i_1,\ldots,i_{k-1})\). Thus we can compute \(\ensuremath{\mathit{LOOKUP}}_k(x,i)\) by first computing \(a\) and \(b\) and then outputting \(\ensuremath{\mathit{IF}}(i_{k-1},a,b)\).

Lemma 4.5 directly implies Theorem 4.4. We prove by induction on \(k\) that there is a NAND-CIRC program of at most \(4\cdot 2^k\) lines for \(\ensuremath{\mathit{LOOKUP}}_k\). For \(k=1\) this follows by the four line program for \(\ensuremath{\mathit{IF}}\) we’ve seen before. For \(k>1\), we use the following pseudocode

```
a = LOOKUP_(k-1)(X[0],...,X[2^(k-1)-1],i[1],...,i[k-1])
b = LOOKUP_(k-1)(X[2^(k-1)],...,Z[2^(k-1)],i[1],...,i[k-1])
return IF(i[k-1],a,b)
```

If we let \(L(k)\) be the number of lines required for \(\ensuremath{\mathit{LOOKUP}}_k\), then the above shows that \[ L(k) \leq 2L(k-1)+4 \;. \;\;(4.3) \] which solves for \(L(k) \leq 4(2^k-1)\). (See Figure 4.3 for a plot of the actual number of lines in our implementation of \(\ensuremath{\mathit{LOOKUP}}_k\).)

## Computing *every* function

At this point we know the following facts about NAND-CIRC programs:

They can compute at least some non trivial functions.

Coming up with NAND-CIRC programs for various functions is a very tedious task.

Thus I would not blame the reader if they were not particularly looking forward to a long sequence of examples of functions that can be computed by NAND-CIRC programs. However, it turns out we are not going to need this, as we can show in one fell swoop that NAND-CIRC programs can compute *every* finite function:

There exists some constant \(c>0\) such that for every \(n,m>0\) and function \(f: \{0,1\}^n\rightarrow \{0,1\}^m\), there is a NAND circuit with at most \(c \cdot m 2^n\) gates that computes the function \(f\) .

Note that up to constants, the models of NAND circuits, NAND-CIRC programs, AON-CIRC programs, and Boolean circuits, are all equivalent to one another, and hence ?? holds for all these models.

As we’ll see in the proof, the constant \(c\) will be smaller than \(10\). In fact, with a tighter proof, we can even shave an extra factor of \(n\), as well as optimize the constant, to obtain the following stronger result:

For every \(\epsilon>0\), \(m\in \N\) and sufficiently large \(n\), if \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) then \(f\) can be computed by a NAND circuit of at most \[ (1+\epsilon)\tfrac{m\cdot 2^n}{n} \] gates.

We will not prove Lemma 4.8 in this book, but discuss how to obtain a bound of the form \(O(\tfrac{m \cdot 2^n}{n})\) in Subsection 4.4.2. See also the biographical notes.

*Every* finite function can be computed by a large enough Boolean circuit.

### Proof of NAND’s Universality

To prove Theorem 4.6, we need to give a NAND circuit, or equivalently a NAND-CIRC program, for *every* possible function. We will restrict our attention to the case of Boolean functions (i.e., \(m=1\)). In Exercise 4.9 you will show how to extend the proof for all values of \(m\). A function \(F: \{0,1\}^n\rightarrow \{0,1\}\) can be specified by a table of its values for each one of the \(2^n\) inputs. For example, the table below describes one particular function \(G: \{0,1\}^4 \rightarrow \{0,1\}\):^{4}

Input (\(x\)) | Output (\(G(x)\)) |
---|---|

\(0000\) | 1 |

\(1000\) | 1 |

\(0100\) | 0 |

\(1100\) | 0 |

\(0010\) | 1 |

\(1010\) | 0 |

\(0110\) | 0 |

\(1110\) | 1 |

\(0001\) | 0 |

\(1001\) | 0 |

\(0101\) | 0 |

\(1101\) | 0 |

\(0011\) | 1 |

\(1011\) | 1 |

\(0111\) | 1 |

\(1111\) | 1 |

We can see that for every \(x\in \{0,1\}^4\), \(G(x)=\ensuremath{\mathit{LOOKUP}}_4(1100100100001111,x)\). Therefore the following is NAND “pseudocode” to compute \(G\):

```
G0000 = 1
G1000 = 1
G0100 = 0
...
G0111 = 1
G1111 = 1
Y[0] = LOOKUP_4(G0000,G1000,...,G1111,
X[0],X[1],X[2],X[3])
```

We can translate this pseudocode into an actual NAND-CIRC program by adding three lines to define variables `zero`

and `one`

that are initialized to \(0\) and \(1\) repsectively, and then replacing a statement such as `Gxxx = 0`

with `Gxxx = NAND(one,one)`

and a statement such as `Gxxx = 1`

with `Gxxx = NAND(zero,zero)`

. The call to `LOOKUP_4`

will be replaced by the NAND-CIRC program that computes \(\ensuremath{\mathit{LOOKUP}}_4\), plugging in the appropriate inputs.

There was nothing about the above reasoning that was particular to the function \(G\) of ??. Given *every* function \(F: \{0,1\}^n \rightarrow \{0,1\}\), we can write a NAND-CIRC program that does the following:

Initialize \(2^n\) variables of the form

`F00...0`

till`F11...1`

so that for every \(z\in\{0,1\}^n\), the variable corresponding to \(z\) is assigned the value \(F(z)\).Compute \(\ensuremath{\mathit{LOOKUP}}_n\) on the \(2^n\) variables initialized in the previous step, with the index variable being the input variables

`X[`

\(0\)`]`

,…,`X[`

\(2^n-1\)`]`

. That is, just like in the pseudocode for`G`

above, we use`Y[0] = LOOKUP(F00..00,...,F11..1,X[0],..,x[`

\(n-1\)`])`

The total number of lines in the program will be \(2^n\) plus the \(4\cdot 2^n\) lines that we pay for computing \(\ensuremath{\mathit{LOOKUP}}_n\). This completes the proof of Theorem 4.6.

While Theorem 4.6 seems striking at first, in retrospect, it is perhaps not that surprising that every finite function can be computed with a NAND-CIRC program. After all, a finite function \(F: \{0,1\}^n \rightarrow \{0,1\}^m\) can be represented by simply the list of its outputs for each one of the \(2^n\) input values. So it makes sense that we could write a NAND-CIRC program of similar size to compute it. What is more interesting is that *some* functions, such as addition and multiplication, have a much more efficient representation: one that only requires \(O(n^2)\) or even smaller number of lines.

### Improving by a factor of \(n\) (optional)

As discussed in Remark 4.7, by being a little more careful, we can improve the bound of Theorem 4.6 and show that every function \(F:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a NAND-CIRC program of at most \(O(m 2^n/n)\) lines. As before, it is enough to prove the case that \(m=1\).

The idea is to use the technique known as *memoization*. Let \(k= \log(n-2\log n)\) (the reasoning behind this choice will become clear later on). For every \(a \in \{0,1\}^{n-k}\) we define \(F_a:\{0,1\}^k \rightarrow \{0,1\}\) to be the function that maps \(w_0,\ldots,w_{k-1}\) to \(F(a_0,\ldots,a_{n-k-1},w_0,\ldots,w_{k-1})\).

On input \(x=x_0,\ldots,x_{n-1}\), we can compute \(F(x)\) as follows: First we compute a \(2^{n-k}\) long string \(P\) whose \(a^{th}\) entry (identifying \(\{0,1\}^{n-k}\) with \([2^{n-k}]\)) equals \(F_a(x_{n-k},\ldots,x_{n-1})\). One can verify that \(F(x)=\ensuremath{\mathit{LOOKUP}}_{n-k}(P,x_0,\ldots,x_{n-k-1})\). Since we can compute \(\ensuremath{\mathit{LOOKUP}}_{n-k}\) using \(O(2^{n-k})\) lines, if we can compute the string \(P\) (i.e., compute variables `P_`

\(0\), …, `P_`

\(2^{n-k}-1\)) using \(T\) lines, then we can compute \(F\) in \(O(2^{n-k})+T\) lines.

The trivial way to compute the string \(P\) would be to use \(O(2^k)\) lines to compute for every \(a\) the map \(x_0,\ldots,x_{k-1} \mapsto F_a(x_0,\ldots,x_{k-1})\) as in the proof of Theorem 4.6. Since there are \(2^{n-k}\) \(a\)’s, that would be a total cost of \(O(2^{n-k} \cdot 2^k) = O(2^n)\) which would not improve at all on the bound of Theorem 4.6. However, a more careful observation shows that we are making some *redundant* computations. After all, there are only \(2^{2^k}\) distinct functions mapping \(k\) bits to one bit. If \(a\) and \(a'\) satisfy that \(F_a = F_{a'}\) then we don’t need to spend \(2^k\) lines computing both \(F_a(x)\) and \(F_{a'}(x)\) but rather can only compute the variable `P_`

\(a\) and then copy `P_`

\(a\) to `P_`

\(a'\) using \(O(1)\) lines. Since we have \(2^{2^k}\) unique functions, we can bound the total cost to compute \(P\) by \(O(2^{2^k}2^k)+O(2^{n-k})\).

Now it just becomes a matter of calculation. By our choice of \(k\), \(2^k = n-2\log n\) and hence \(2^{2^k}=\tfrac{2^n}{n^2}\). Since \(n/2 \leq 2^k \leq n\), we can bound the total cost of computing \(F(x)\) (including also the additional \(O(2^{n-k})\) cost of computing \(\ensuremath{\mathit{LOOKUP}}_{n-k}\)) by \(O(\tfrac{2^n}{n^2}\cdot n)+O(2^n/n)\), which is what we wanted to prove.

## The class \(\ensuremath{\mathit{SIZE}}_{n,m}(T)\)

We now make a fundamental definition, we let \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) denote the set of all functions from \(\{0,1\}^n\) to \(\{0,1\}^m\) that can be computed by NAND circuits of at most \(s\) gates (or equivalently, by NAND-CIRC programs of at most \(s\) lines):

Let \(n,m,s \in \N\) be numbers with \(s \geq m\). The set \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) denotes the set of all functions \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) such that there exists a NAND circuit of at most \(s\) gates that computes \(f\). We denote by \(\ensuremath{\mathit{SIZE}}_n(s)\) the set \(\ensuremath{\mathit{SIZE}}_{n,1}(s)\).

Figure 4.4 depicts the sets \(\ensuremath{\mathit{SIZE}}_{n,1}(s)\), note that \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) is a set of *functions*, not of *programs!* (asking if a program or a circuit is a member of \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) is a *category error* as in the sense of Figure 4.5).

While we defined \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) with respect to NAND gates, we would get essentially the same class if we defined it with respect to AND/OR/NOT gates:

Let \(\ensuremath{\mathit{SIZE}}^{AON}_{n,m,s}\) denote the set of all functions \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) that can be computed by an AND/OR/NOT Boolean circuit of at most \(s\) gates. Then, \[ \ensuremath{\mathit{SIZE}}_{n,m}(s/2) \subseteq \ensuremath{\mathit{SIZE}}^{AON}_{n,m}(s) \subseteq \ensuremath{\mathit{SIZE}}_{n,m}(3s) \]

If \(f\) can be computed by a NAND circuit of at most \(s/2\) gates, then by replacing each NAND with the two gates NOT and AND, we can obtain an AND/OR/NOT Boolean circuit of at most \(s\) gates that computes \(f\). On the other hand, if \(f\) can be computed by a Boolean AND/OR/NOT circuit of at most \(s\) gates, then by Theorem 3.8 it can be computed by a NAND circuit of at most \(3s\) gates.

The results we’ve seen before can be phrased as showing that \(\ensuremath{\mathit{ADD}}_n \in \ensuremath{\mathit{SIZE}}_{2n,n+1}(100 n)\) and \(\ensuremath{\mathit{MULT}}_n \in \ensuremath{\mathit{SIZE}}_{2n,2n}(10000 n^{\log_2 3})\). Theorem 4.6 shows that \(\ensuremath{\mathit{SIZE}}_{n,m}(4 m 2^n)\) is equal the set of all functions from \(\{0,1\}^n\) to \(\{0,1\}^m\). See Figure 5.4.

Note that \(\ensuremath{\mathit{SIZE}}_{n,m}(s)\) does **not** correspond to a set of programs! Rather, it is a set of *functions* (see Figure 4.5). This distinction between *programs* and *functions* will be crucial for us in this course. You should always remember that while a program *computes* a function, it is not *equal* to a function. In particular, as we’ve seen, there can be more than one program to compute the same function.

A NAND-CIRC program \(P\) can only compute a function with a certain number \(n\) of inputs and a certain number \(m\) of outputs. Hence for example there is no single NAND-CIRC program that can compute the increment function \(\ensuremath{\mathit{INC}}:\{0,1\}^* \rightarrow \{0,1\}^*\) that maps a string \(x\) (which we identify with a number via the binary representation) to the string that represents \(x+1\). Rather for every \(n>0\), there is a NAND-CIRC program \(P_n\) that computes the restriction \(\ensuremath{\mathit{INC}}_n\) of the function \(\ensuremath{\mathit{INC}}\) to inputs of length \(n\). Since it can be shown that for every \(n>0\) such a program \(P_n\) exists of length at most \(10n\), \(\ensuremath{\mathit{INC}}_n \in \ensuremath{\mathit{SIZE}}(10n)\) for every \(n>0\).

If \(T:\N \rightarrow \N\) and \(F:\{0,1\}^* \rightarrow \{0,1\}^*\), we will sometimes slightly abuse notation and write \(F \in \ensuremath{\mathit{SIZE}}(T(n))\) to indicate that for every \(n\) the restriction \(F_{\upharpoonright n}\) of \(F\) to inputs in \(\{0,1\}^n\) is in \(\ensuremath{\mathit{SIZE}}(T(n))\). Hence we can write \(\ensuremath{\mathit{INC}} \in \ensuremath{\mathit{SIZE}}(10n)\). We will come back to this issue of finite vs infinite functions later in this course.

In this exercise we prove a certain “closure property” of the class \(\ensuremath{\mathit{SIZE}}_n(s)\). That is, we show that if \(f\) is in this class then (up to some small additive term) so is the complement of \(f\), which is the function \(g(x)=1-f(x)\).

Prove that there is a constant \(c\) such that for every \(f:\{0,1\}^n \rightarrow \{0,1\}\) and \(s\in \N\), if \(f \in \ensuremath{\mathit{SIZE}}_n(s)\) then \(1-f \in \ensuremath{\mathit{SIZE}}_n(s+c)\).

If \(f\in \ensuremath{\mathit{SIZE}}(s)\) then there is an \(s\)-line NAND-CIRC program \(P\) that computes \(f\). We can rename the variable `Y[0]`

in \(P\) to a variable `temp`

and add the line

at the very end to obtain a program \(P'\) that computes \(1-f\).

- We can define the notion of computing a function via a simplified “programming language”, where computing a function \(F\) in \(T\) steps would correspond to having a \(T\)-line NAND-CIRC program that computes \(F\).
- While the NAND-CIRC programming only has one operation, other operations such as functions and conditional execution can be implemented using it.
- Every function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\) can be computed by a circuit of at most \(O(m 2^n)\) gates (and in fact at most \(O(m 2^n/n)\) gates).
- Sometimes (or maybe always?) we can translate an
*efficient*algorithm to compute \(f\) into a circuit that computes \(f\) with a number of gates comparable to the number of steps in this algorithm.

## Exercises

This exercise asks you to give a one-to-one map from \(\N^2\) to \(\N\). This can be useful to implement two-dimensional arrays as “syntacic sugar” in programming languages that only have one-dimensional array.

Prove that the map \(F(x,y)=2^x3^y\) is a one-to-one map from \(\N^2\) to \(\N\).

Show that there is a one-to-one map \(F:\N^2 \rightarrow \N\) such that for every \(x,y\), \(F(x,y) \leq 100\cdot \max\{x,y\}^2+100\).

- For every \(k\), show that there is a one-to-one map \(F:\N^k \rightarrow \N\) such that for every \(x_0,\ldots,x_{k-1} \in \N\), \(F(x_0,\ldots,x_{k-1}) \leq 100 \cdot (x_0+x_1+\ldots+x_{k-1}+100k)^k\).

Prove that the NAND-CIRC program below computes the function \(\ensuremath{\mathit{MUX}}\) (or \(\ensuremath{\mathit{LOOKUP}}_1\)) where \(\ensuremath{\mathit{MUX}}(a,b,c)\) equals \(a\) if \(c=0\) and equals \(b\) if \(c=1\):

Give a NAND-CIRC program of at most 6 lines to compute \(\ensuremath{\mathit{MAJ}}:\{0,1\}^3 \rightarrow \{0,1\}\) where \(\ensuremath{\mathit{MAJ}}(a,b,c) = 1\) iff \(a+b+c \geq 2\).

In this exercise we will show that even though the NAND-CIRC programming language does not have an `if .. then .. else ..`

statement, we can still implement it. Suppose that there is an \(s\)-line NAND-CIRC program to compute \(f:\{0,1\}^n \rightarrow \{0,1\}\) and an \(s'\)-line NAND-CIRC program to compute \(f':\{0,1\}^n \rightarrow \{0,1\}\). Prove that there is a program of at most \(s+s'+10\) lines to compute the function \(g:\{0,1\}^{n+1} \rightarrow \{0,1\}\) where \(g(x_0,\ldots,x_{n-1},x_n)\) equals \(f(x_0,\ldots,x_{n-1})\) if \(x_n=0\) and equals \(f'(x_0,\ldots,x_{n-1})\) otherwise.

A

*half adder*is the function \(\ensuremath{\mathit{HA}}:\{0,1\}^2 :\rightarrow \{0,1\}^2\) that corresponds to adding two binary bits. That is, for every \(a,b \in \{0,1\}\), \(\ensuremath{\mathit{HA}}(a,b)= (e,f)\) where \(2e+f = a +b\). Prove that there is a NAND circuit of at most five NAND gates that computes \(\ensuremath{\mathit{HA}}\).A

*full adder*is the function \(\ensuremath{\mathit{FA}}:\{0,1\}^3 \rightarrow \{0,1\}\) that takes in two bits and a “carry” bit and outputs their sum. That is, for every \(a,b,c \in \{0,1\}\), FA(a,b,c) = (e,f)$ such that \(2e+f = a+b+c\). Prove that there is a NAND circuit of at most nine NAND gates that computes \(\ensuremath{\mathit{FA}}\).Prove that if there is a NAND circuit of \(c\) gates that computes \(\ensuremath{\mathit{FA}}\), then there is a circuit of \(cn\) gates that computes \(\ensuremath{\mathit{ADD}}_n\) where (as in Theorem 4.1) \(\ensuremath{\mathit{ADD}}_n:\{0,1\}^{2n} \rightarrow \{0,1\}^n\) is the function that outputs the addition of two input \(n\)-bit numbers. See footnote for hint.

^{5}- Show that for every \(n\) there is a NAND-CIRC program to compute \(\ensuremath{\mathit{ADD}}_n\) with at most \(9n\) lines.

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{ADD}}_n\). Can you ensure that the program it outputs for \(\ensuremath{\mathit{ADD}}_n\) has fewer than \(10n\) lines?

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{MULT}}_n\). Can you ensure that the program it outputs for \(\ensuremath{\mathit{MULT}}_n\) has fewer than \(1000\cdot n^2\) lines?

Write a program using your favorite programming language that on input an integer \(n\), outputs a NAND-CIRC program that computes \(\ensuremath{\mathit{MULT}}_n\) and has at most \(10000 n^{1.9}\) lines.^{6} What is the smallest number of lines you can use to multiply two 2048 bit numbers?

Prove that

If there is an \(s\)-line NAND-CIRC program to compute \(f:\{0,1\}^n \rightarrow \{0,1\}\) and an \(s'\)-line NAND-CIRC program to compute \(f':\{0,1\}^n \rightarrow \{0,1\}\) then there is an \(s+s'\)-line program to compute the function \(g:\{0,1\}^n \rightarrow \{0,1\}^2\) such that \(g(x)=(f(x),f'(x))\).

- For every function \(f:\{0,1\}^n \rightarrow \{0,1\}^m\), there is a NAND-CIRC program of at most \(10m\cdot 2^n\) lines that computes \(f\).

## Bibliographical notes

See Jukna’s and Wegener’s books (Jukna, 2012) (Wegener, 1987) for much more extensive discussion on circuits. Shannon showed that every Boolean function can be computed by a circuit of exponential size (Shannon, 1938) . The improved bound of \(c \cdot 2^n/n\) (with the optimal value of \(c\) for many bases) is due to Lupanov (Lupanov, 1958) . An exposition of this for the case of NAND is given in Chapter 4 of his book (Lupanov, 1984) . (Thanks to Sasha Golovnev for tracking down this reference!)

This concept is also known as “macros” or “meta-programming” and is sometimes implemented via a preprocessor or macro language in a programming language or a text editor. One modern example is the Babel JavaScript syntax transformer, that converts JavaScript programs written using the latest features into a format that older Browsers can accept. It even has a plug-in architecture, that allows users to add their own syntactic sugar to the language.

We use here least-significant-digit first convention for simplicity of notation.

The value of \(c\) can be improved to \(9\), see Exercise 4.5.

In case you are curious, this is the function that computes the digits of \(\pi\) in the binary basis. Note that as per the convention of this course, if we think of strings as numbers then we right them with the least significant digit first.

Use a “cascade” of adding the bits one after the other, starting with the least significant digit, just like in the elementary-school algorithm.

**Hint:**Use Karatsuba’s algorithm

## Comments

Comments are posted on the GitHub repository using the utteranc.es app. A GitHub login is required to comment. If you don't want to authorize the app to post on your behalf, you can also comment directly on the GitHub issue for this page.

Compiled on 02/15/2019 10:37:12

Copyright 2019, Boaz Barak.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Produced using pandoc and panflute with templates derived from gitbook and bookdown.