Functions with Infinite domains, Automata, and Regular expressions

“An algorithm is a finite answer to an infinite number of questions.”, Attributed to Stephen Kleene.

The model of Boolean circuits (or equivalently, the NAND-CIRC programming language) has one very significant drawback: a Boolean circuit can only compute a finite function $f$. In particular, since every gate has two inputs, a size $s$ circuit can compute on an input of length at most $2s$. Thus this model does not capture our intuitive notion of an algorithm as a single recipe to compute a potentially infinite function. For example, the standard elementary school multiplication algorithm is a single algorithm that multiplies numbers of all lengths. However, we cannot express this algorithm as a single circuit, but rather need a different circuit (or equivalently, a NAND-CIRC program) for every input length (see Figure 6.1).

In this chapter, we extend our definition of computational tasks to consider functions with the unbounded domain of $\{0,1\}^*$. We focus on the question of defining what tasks to compute, mostly leaving the question of how to compute them to later chapters, where we will see Turing machines and other computational models for computing on unbounded inputs. However, we will see one example of a simple restricted model of computation - deterministic finite automata (DFAs).

Functions with inputs of unbounded length

Up until now, we considered the computational task of mapping some string of length $n$ into a string of length $m$. However, in general, computational tasks can involve inputs of unbounded length. For example, the following Python function computes the function $\ensuremath{\mathit{XOR}}:\{0,1\}^* \rightarrow \{0,1\}$, where $\ensuremath{\mathit{XOR}}(x)$ equals $1$ iff the number of $1$’s in $x$ is odd. (In other words, $\ensuremath{\mathit{XOR}}(x) = \sum_{i=0}^{|x|-1} x_i \mod 2$ for every $x\in \{0,1\}^*$.) As simple as it is, the $\ensuremath{\mathit{XOR}}$ function cannot be computed by a Boolean circuit. Rather, for every $n$, we can compute $\ensuremath{\mathit{XOR}}_n$ (the restriction of $\ensuremath{\mathit{XOR}}$ to $\{0,1\}^n$) using a different circuit (e.g., see Figure 6.2).

def XOR(X):
    '''Takes list X of 0's and 1's
       Outputs 1 if the number of 1's is odd and outputs 0 otherwise'''
    result = 0
    for i in range(len(X)):
        result = (result + X[i]) % 2
    return result

Previously in this book, we studied the computation of finite functions $f:\{0,1\}^n \rightarrow \{0,1\}^m$. Such a function $f$ can always be described by listing all the $2^n$ values it takes on inputs $x\in \{0,1\}^n$. In this chapter, we consider functions such as $\ensuremath{\mathit{XOR}}$ that take inputs of unbounded size. While we can describe $\ensuremath{\mathit{XOR}}$ using a finite number of symbols (in fact, we just did so above), it takes infinitely many possible inputs, and so we cannot just write down all of its values. The same is true for many other functions capturing important computational tasks, including addition, multiplication, sorting, finding paths in graphs, fitting curves to points, and so on. To contrast with the finite case, we will sometimes call a function $F:\{0,1\}^* \rightarrow \{0,1\}$ (or $F:\{0,1\}^* \rightarrow \{0,1\}^*$) infinite. However, this does not mean that $F$ takes as input strings of infinite length! It just means that $F$ can take as input a string that can be arbitrarily long, and so we cannot simply write down a table of all the outputs of $F$ on different inputs.

As we have seen before, restricting attention to functions that use binary strings as inputs and outputs does not detract from our generality, since other objects, including numbers, lists, matrices, images, videos, and more, can be encoded as binary strings.

As before, it is essential to differentiate between specification and implementation. For example, consider the following function:

This is a mathematically well-defined function. For every $x$, $\ensuremath{\mathit{TWINP}}(x)$ has a unique value which is either $0$ or $1$. However, at the moment, no one knows of a Python program that computes this function. The Twin prime conjecture posits that for every $n$ there exists $p>n$ such that both $p$ and $p+2$ are primes. If this conjecture is true, then $T$ is easy to compute indeed - the program def T(x): return 1 will do the trick. However, mathematicians have tried unsuccessfully to prove this conjecture since 1849. That said, whether or not we know how to implement the function $\ensuremath{\mathit{TWINP}}$, the definition above provides its specification.

Varying inputs and outputs

Many of the functions that interest us take more than one input. For example, the function

takes the binary representation of a pair of integers $x,y \in \N$, and outputs the binary representation of their product $x \cdot y$. However, since we can represent a pair of strings as a single string, we will consider functions such as MULT as mapping $\{0,1\}^*$ to $\{0,1\}^*$. We will typically not be concerned with low-level details such as the precise way to represent a pair of integers as a string, since virtually all choices will be equivalent for our purposes.

Another example of a function we want to compute is

$\ensuremath{\mathit{PALINDROME}}$ has a single bit as output. Functions with a single bit of output are known as Boolean functions. Boolean functions are central to the theory of computation, and we will discuss them often in this book. Note that even though Boolean functions have a single bit of output, their input can be of arbitrary length. Thus they are still infinite functions that cannot be described via a finite table of values.

“Booleanizing” functions. Sometimes it might be convenient to obtain a Boolean variant for a non-Boolean function. For example, the following is a Boolean variant of $\ensuremath{\mathit{MULT}}$.

If we can compute $\ensuremath{\mathit{BMULT}}$ via any programming language such as Python, C, Java, etc., we can compute $\ensuremath{\mathit{MULT}}$ as well, and vice versa.

def F(x):
    res = []
    i = 0
    while BF(x,i,1):
        res.append(BF(x,i,0))
        i += 1
    return res

Formal Languages

For every Boolean function $F:\{0,1\}^* \rightarrow \{0,1\}$, we can define the set $L_F = \{ x | F(x) = 1 \}$ of strings on which $F$ outputs $1$. Such sets are known as languages. This name is rooted in formal language theory as pursued by linguists such as Noam Chomsky. A formal language is a subset $L \subseteq \{0,1\}^*$ (or more generally $L \subseteq \Sigma^*$ for some finite alphabet $\Sigma$). The membership or decision problem for a language $L$, is the task of determining, given $x\in \{0,1\}^*$, whether or not $x\in L$. If we can compute the function $F$, then we can decide membership in the language $L_F$ and vice versa. Hence, many texts such as (Sipser, 1997) refer to the task of computing a Boolean function as “deciding a language”. In this book, we mostly describe computational tasks using the function notation, which is easier to generalize to computation with more than one bit of output. However, since the language terminology is so popular in the literature, we will sometimes mention it.

Restrictions of functions

If $F:\{0,1\}^* \rightarrow \{0,1\}$ is a Boolean function and $n\in \N$ then the restriction of $F$ to inputs of length $n$, denoted as $F_n$, is the finite function $f:\{0,1\}^n \rightarrow \{0,1\}$ such that $f(x) = F(x)$ for every $x\in \{0,1\}^n$. That is, $F_n$ is the finite function that is only defined on inputs in $\{0,1\}^n$, but agrees with $F$ on those inputs. Since $F_n$ is a finite function, it can be computed by a Boolean circuit, implying the following theorem:

In particular, Theorem 6.1 implies that there exists such a circuit collection $\{ C_n \}$ even for the $\ensuremath{\mathit{TWINP}}$ function we described before, even though we do not know of any program to compute it. Indeed, this is not that surprising: for every particular $n\in \N$, $\ensuremath{\mathit{TWINP}}_n$ is either the constant zero function or the constant one function, both of which can be computed by very simple Boolean circuits. Hence a collection of circuits $\{ C_n \}$ that computes $\ensuremath{\mathit{TWINP}}$ certainly exists. The difficulty in computing $\ensuremath{\mathit{TWINP}}$ using Python or any other programming language arises from the fact that we do not know for each particular $n$ what is the circuit $C_n$ in this collection.

Deterministic finite automata (optional)

All our computational models so far - Boolean circuits and straight-line programs - were only applicable for finite functions.

In Chapter 7, we will present Turing machines, which are the central models of computation for unbounded input length functions. However, in this section we present the more basic model of deterministic finite automata (DFA). Automata can serve as a good stepping-stone for Turing machines, though they will not be used much in later parts of this book, and so the reader can feel free to skip ahead to Chapter 7. DFAs turn out to be equivalent in power to regular expressions: a powerful mechanism to specify patterns, which is widely used in practice. Our treatment of automata is relatively brief. There are plenty of resources that help you get more comfortable with DFAs. In particular, Chapter 1 of Sipser’s book (Sipser, 1997) contains an excellent exposition of this material. There are also many websites with online simulators for automata, as well as translators from regular expressions to automata and vice versa (see for example here and here).

At a high level, an algorithm is a recipe for computing an output from an input via a combination of the following steps:

1. Read a bit from the input
2. Update the state (working memory)
3. Stop and produce an output

For example, recall the Python program that computes the $\ensuremath{\mathit{XOR}}$ function:

def XOR(X):
    '''Takes list X of 0's and 1's
       Outputs 1 if the number of 1's is odd and outputs 0 otherwise'''
    result = 0
    for i in range(len(X)):
        result = (result + X[i]) % 2
    return result

In each step, this program reads a single bit X[i] and updates its state result based on that bit (flipping result if X[i] is $1$ and keeping it the same otherwise). When it is done transversing the input, the program outputs result. In computer science, such a program is called a single-pass constant-memory algorithm since it makes a single pass over the input and its working memory is finite. (Indeed, in this case, result can either be $0$ or $1$.) Such an algorithm is also known as a Deterministic Finite Automaton or DFA (another name for DFAs is a finite state machine). We can think of such an algorithm as a “machine” that can be in one of $C$ states, for some constant $C$. The machine starts in some initial state and then reads its input $x\in \{0,1\}^*$ one bit at a time. Whenever the machine reads a bit $\sigma \in \{0,1\}$, it transitions into a new state based on $\sigma$ and its prior state. The output of the machine is based on the final state. Every single-pass constant-memory algorithm corresponds to such a machine. If an algorithm uses $c$ bits of memory, then the contents of its memory can be represented as a string of length $c$. Therefore such an algorithm can be in one of at most $2^c$ states at any point in the execution.

We can specify a DFA of $C$ states by a list of $C \cdot 2$ rules. Each rule will be of the form “If the DFA is in state $v$ and the bit read from the input is $\sigma$ then the new state is $v'$”. At the end of the computation, we will also have a rule of the form “If the final state is one of the following … then output $1$, otherwise output $0$”. For example, the Python program above can be represented by a two-state automaton for computing $\ensuremath{\mathit{XOR}}$ of the following form:

• Initialize in the state $0$.
• For every state $s \in \{0,1\}$ and input bit $\sigma$ read, if $\sigma =1$ then change to state $1-s$, otherwise stay in state $s$.
• At the end output $1$ iff $s=1$.

We can also describe a $C$-state DFA as a labeled graph of $C$ vertices. For every state $s$ and bit $\sigma$, we add a directed edge labeled with $\sigma$ between $s$ and the state $s'$ such that if the DFA is at state $s$ and reads $\sigma$ then it transitions to $s'$. (If the state stays the same then this edge will be a self-loop; similarly, if $s$ transitions to $s'$ in both the case $\sigma=0$ and $\sigma=1$ then the graph will contain two parallel edges.) We also label the set $\mathcal{S}$ of states on which the automaton will output $1$ at the end of the computation. This set is known as the set of accepting states. See Figure 6.3 for the graphical representation of the XOR automaton.

Formally, a DFA is specified by (1) the table of the $C \cdot 2$ rules, which can be represented as a transition function $T$ that maps a state $s \in [C]$ and bit $\sigma \in \{0,1\}$ to the state $s' \in [C]$ which the DFA will transition to from state $s$ on input $\sigma$ and (2) the set $\mathcal{S}$ of accepting states. This leads to the following definition.

def F(X):
    '''Return 1 iff X is a concatenation of zero/more copies of [0,1,0]'''
    if len(X) % 3 != 0:
        return False
    ultimate = 0
    penultimate = 1
    antepenultimate = 0
    for idx, b in enumerate(X):
        antepenultimate = penultimate
        penultimate = ultimate
        ultimate = b
        if idx % 3 == 2 and ((antepenultimate, penultimate, ultimate) != (0,1,0)):
            return False
    return True

Anatomy of an automaton (finite vs. unbounded)

Now that we are considering computational tasks with unbounded input sizes, it is crucial to distinguish between the components of our algorithm that have fixed length and the components that grow with the input size. For the case of DFAs these are the following:

Constant size components: Given a DFA $A$, the following quantities are fixed independent of the input size:

• The number of states $C$ in $A$.

• The transition function $T$ (which has $2C$ inputs, and so can be specified by a table of $2C$ rows, each entry in which is a number in $[C]$).

• The set $\mathcal{S} \subseteq [C]$ of accepting states. This set can be described by a string in $\{0,1\}^C$ specifiying which states are in $\mathcal{S}$ and which are not.

Together the above means that we can fully describe an automaton using finitely many symbols. This is a property we require out of any notion of “algorithm”: we should be able to write down a complete specification of how it produces an output from an input.

Components of unbounded size: The following quantities relating to a DFA are not bounded by any constant. We stress that these are still finite for any given input.

• The length of the input $x\in \{0,1\}^*$ that the DFA is provided. The input length is always finite, but not a priori bounded.

• The number of steps that the DFA takes can grow with the length of the input. Indeed, a DFA makes a single pass on the input and so it takes precisely $|x|$ steps on an input $x\in \{0,1\}^*$.

DFA-computable functions

We say that a function $F:\{0,1\}^* \rightarrow \{0,1\}$ is DFA computable if there exists some $\ensuremath{\mathit{DFA}}$ that computes $F$. In Chapter 4 we saw that every finite function is computable by some Boolean circuit. Thus, at this point, you might expect that every infinite function is computable by some DFA. However, this is very much not the case. We will soon see some simple examples of infinite functions that are not computable by DFAs, but for starters, let us prove that such functions exist.

Since the set of all Boolean functions is uncountable, we get the following corollary:

Regular expressions

Searching for a piece of text is a common task in computing. At its heart, the search problem is quite simple. We have a collection $X = \{ x_0, \ldots, x_k \}$ of strings (e.g., files on a hard-drive, or student records in a database), and the user wants to find out the subset of all the $x \in X$ that are matched by some pattern (e.g., all files whose names end with the string .txt). In full generality, we can allow the user to specify the pattern by specifying a (computable) function $F:\{0,1\}^* \rightarrow \{0,1\}$, where $F(x)=1$ corresponds to the pattern matching $x$. That is, the user provides a program $P$ in a programming language such as Python, and the system returns all $x \in X$ such that $P(x)=1$. For example, one could search for all text files that contain the string important document or perhaps (letting $P$ correspond to a neural-network based classifier) all images that contain a cat. However, we don’t want our system to get into an infinite loop just trying to evaluate the program $P$! For this reason, typical systems for searching files or databases do not allow users to specify the patterns using full-fledged programming languages. Rather, such systems use restricted computational models that on the one hand are rich enough to capture many of the queries needed in practice (e.g., all filenames ending with .txt, or all phone numbers of the form (617) xxx-xxxx), but on the other hand are restricted enough so that queries can be evaluated very efficiently on huge files and in particular cannot result in an infinite loop.

One of the most popular such computational models is regular expressions. If you ever used an advanced text editor, a command-line shell, or have done any kind of manipulation of text files, then you have probably come across regular expressions.

A regular expression over some alphabet $\Sigma$ is obtained by combining elements of $\Sigma$ with the operation of concatenation, as well as $|$ (corresponding to or) and $*$ (corresponding to repetition zero or more times). (Common implementations of regular expressions in programming languages and shells typically include some extra operations on top of $|$ and $*$, but these operations can be implemented as “syntactic sugar” using the operators $|$ and $*$.) For example, the following regular expression over the alphabet $\{0,1\}$ corresponds to the set of all strings $x\in \{0,1\}^*$ where every digit is repeated at least twice:

The following regular expression over the alphabet $\{ a,\ldots,z,0,\ldots,9 \}$ corresponds to the set of all strings that consist of a sequence of one or more of the letters $a$-$d$ followed by a sequence of one or more digits (without a leading zero):

Formally, regular expressions are defined by the following recursive definition:

We will drop parentheses when they can be inferred from the context. We also use the convention that OR and concatenation are left-associative, and we give highest precedence to $*$, then concatenation, and then OR. Thus for example we write $00^*|11$ instead of $((0)(0^*))|((1)(1))$.

Every regular expression $e$ corresponds to a function $\Phi_{e}:\Sigma^* \rightarrow \{0,1\}$ where $\Phi_{e}(x)=1$ if $x$ matches the regular expression. For example, if $e = (00|11)^*$ then $\Phi_e(110011)=1$ but $\Phi_e(101)=0$ (can you see why?).

A Boolean function is called “regular” if it outputs $1$ on precisely the set of strings that are matched by some regular expression. That is,

Regular expression can be defined over any finite alphabet $\Sigma$, but as usual, we will mostly focus our attention on the binary case, where $\Sigma = \{0,1\}$. Most (if not all) of the theoretical and practical general insights about regular expressions can be gleaned from studying the binary case.

Algorithms for matching regular expressions

Regular expressions would not be very useful for search if we could not evaluate, given a regular expression $e$, whether a string $x$ is matched by $e$. Luckily, there is an algorithm to do so. Specifically, there is an algorithm (think “Python program” though later we will formalize the notion of algorithms using Turing machines) that on input a regular expression $e$ over the alphabet $\{0,1\}$ and a string $x\in \{0,1\}^*$, outputs $1$ iff $e$ matches $x$ (i.e., outputs $\Phi_e(x)$).

Indeed, Definition 6.7 actually specifies a recursive algorithm for computing $\Phi_{e}$. Specifically, each one of our operations -concatenation, OR, and star- can be thought of as reducing the task of testing whether an expression $e$ matches a string $x$ to testing whether some sub-expressions of $e$ match substrings of $x$. Since these sub-expressions are always shorter than the original expression, this yields a recursive algorithm for checking if $e$ matches $x$, which will eventually terminate at the base cases of the expressions that correspond to a single symbol or the empty string.

We assume above that we have a procedure $\text{\textsc{MatchEmpty}}$ that on input a regular expression $e$ outputs $1$ if and only if $e$ matches the empty string $\ensuremath{\text{\texttt{""}}}$.

The key observation is that in our recursive definition of regular expressions, whenever $e$ is made up of one or two expressions $e',e''$ then these two regular expressions are smaller than $e$. Eventually (when they have size $1$) then they must correspond to the non-recursive case of a single alphabet symbol. Correspondingly, the recursive calls made in Algorithm 6.10 always correspond to a shorter expression or (in the case of an expression of the form $(e')^*$) a shorter input string. Thus, we can prove the correctness of Algorithm 6.10 on inputs of the form $(e,x)$ by induction over $\min \{ |e|, |x| \}$. The base case is when either $x=\ensuremath{\text{\texttt{""}}}$ or $e$ is a single alphabet symbol, $\ensuremath{\text{\texttt{""}}}$ or $\emptyset$. In the case the expression is of the form $e=(e'|e'')$ or $e=(e')(e'')$, we make recursive calls with the shorter expressions $e',e''$. In the case the expression is of the form $e=(e')^*$, we make recursive calls with either a shorter string $x$ and the same expression, or with the shorter expression $e'$ and a string $x'$ that is equal in length or shorter than $x$.

Efficient matching of regular expressions (optional)

Algorithm 6.10 is not very efficient. For example, given an expression involving concatenation or the “star” operation and a string of length $n$, it can make $n$ recursive calls, and hence it can be shown that in the worst case Algorithm 6.10 can take time exponential in the length of the input string $x$. Fortunately, it turns out that there is a much more efficient algorithm that can match regular expressions in linear (i.e., $O(n)$) time. Since we have not yet covered the topics of time and space complexity, we describe this algorithm in high level terms, without making the computational model precise. Rather we will use the colloquial notion of $O(n)$ running time as used in introduction to programming courses and whiteboard coding interviews. We will see a formal definition of time complexity in Chapter 13.

The implicit constant in the $O(n)$ term of Theorem 6.12 depends on the expression $e$. Thus, another way to state Theorem 6.12 is that for every expression $e$, there is some constant $c$ and an algorithm $A$ that computes $\Phi_e$ on $n$-bit inputs using at most $c\cdot n$ steps. This makes sense since in practice we often want to compute $\Phi_e(x)$ for a small regular expression $e$ and a large document $x$. Theorem 6.12 tells us that we can do so with running time that scales linearly with the size of the document, even if it has (potentially) worse dependence on the size of the regular expression.