Functions with Infinite domains, Automata, and Regular expressions

“An algorithm is a finite answer to an infinite number of questions.”, Attributed to Stephen Kleene.

The model of Boolean circuits (or equivalently, the NAND-CIRC programming language) has one very significant drawback: a Boolean circuit can only compute a finite function \(f\). In particular, since every gate has two inputs, a size \(s\) circuit can compute on an input of length at most \(2s\). Thus this model does not capture our intuitive notion of an algorithm as a single recipe to compute a potentially infinite function. For example, the standard elementary school multiplication algorithm is a single algorithm that multiplies numbers of all lengths. However, we cannot express this algorithm as a single circuit, but rather need a different circuit (or equivalently, a NAND-CIRC program) for every input length (see Figure 6.1).

In this chapter, we extend our definition of computational tasks to consider functions with the unbounded domain of \(\{0,1\}^*\). We focus on the question of defining what tasks to compute, mostly leaving the question of how to compute them to later chapters, where we will see Turing machines and other computational models for computing on unbounded inputs. However, we will see one example of a simple restricted model of computation - deterministic finite automata (DFAs).

Functions with inputs of unbounded length

Up until now, we considered the computational task of mapping some string of length \(n\) into a string of length \(m\). However, in general, computational tasks can involve inputs of unbounded length. For example, the following Python function computes the function \(\ensuremath{\mathit{XOR}}:\{0,1\}^* \rightarrow \{0,1\}\), where \(\ensuremath{\mathit{XOR}}(x)\) equals \(1\) iff the number of \(1\)’s in \(x\) is odd. (In other words, \(\ensuremath{\mathit{XOR}}(x) = \sum_{i=0}^{|x|-1} x_i \mod 2\) for every \(x\in \{0,1\}^*\).) As simple as it is, the \(\ensuremath{\mathit{XOR}}\) function cannot be computed by a Boolean circuit. Rather, for every \(n\), we can compute \(\ensuremath{\mathit{XOR}}_n\) (the restriction of \(\ensuremath{\mathit{XOR}}\) to \(\{0,1\}^n\)) using a different circuit (e.g., see Figure 6.2).

def XOR(X):
    '''Takes list X of 0's and 1's
       Outputs 1 if the number of 1's is odd and outputs 0 otherwise'''
    result = 0
    for i in range(len(X)):
        result = (result + X[i]) % 2
    return result

Previously in this book, we studied the computation of finite functions \(f:\{0,1\}^n \rightarrow \{0,1\}^m\). Such a function \(f\) can always be described by listing all the \(2^n\) values it takes on inputs \(x\in \{0,1\}^n\). In this chapter, we consider functions such as \(\ensuremath{\mathit{XOR}}\) that take inputs of unbounded size. While we can describe \(\ensuremath{\mathit{XOR}}\) using a finite number of symbols (in fact, we just did so above), it takes infinitely many possible inputs, and so we cannot just write down all of its values. The same is true for many other functions capturing important computational tasks, including addition, multiplication, sorting, finding paths in graphs, fitting curves to points, and so on. To contrast with the finite case, we will sometimes call a function \(F:\{0,1\}^* \rightarrow \{0,1\}\) (or \(F:\{0,1\}^* \rightarrow \{0,1\}^*\)) infinite. However, this does not mean that \(F\) takes as input strings of infinite length! It just means that \(F\) can take as input a string that can be arbitrarily long, and so we cannot simply write down a table of all the outputs of \(F\) on different inputs.

As we have seen before, restricting attention to functions that use binary strings as inputs and outputs does not detract from our generality, since other objects, including numbers, lists, matrices, images, videos, and more, can be encoded as binary strings.

As before, it is essential to differentiate between specification and implementation. For example, consider the following function:

This is a mathematically well-defined function. For every \(x\), \(\ensuremath{\mathit{TWINP}}(x)\) has a unique value which is either \(0\) or \(1\). However, at the moment, no one knows of a Python program that computes this function. The Twin prime conjecture posits that for every \(n\) there exists \(p>n\) such that both \(p\) and \(p+2\) are primes. If this conjecture is true, then \(T\) is easy to compute indeed - the program def T(x): return 1 will do the trick. However, mathematicians have tried unsuccessfully to prove this conjecture since 1849. That said, whether or not we know how to implement the function \(\ensuremath{\mathit{TWINP}}\), the definition above provides its specification.

Varying inputs and outputs

Many of the functions that interest us take more than one input. For example, the function

takes the binary representation of a pair of integers \(x,y \in \N\), and outputs the binary representation of their product \(x \cdot y\). However, since we can represent a pair of strings as a single string, we will consider functions such as MULT as mapping \(\{0,1\}^*\) to \(\{0,1\}^*\). We will typically not be concerned with low-level details such as the precise way to represent a pair of integers as a string, since virtually all choices will be equivalent for our purposes.

Another example of a function we want to compute is

\(\ensuremath{\mathit{PALINDROME}}\) has a single bit as output. Functions with a single bit of output are known as Boolean functions. Boolean functions are central to the theory of computation, and we will discuss them often in this book. Note that even though Boolean functions have a single bit of output, their input can be of arbitrary length. Thus they are still infinite functions that cannot be described via a finite table of values.

“Booleanizing” functions. Sometimes it might be convenient to obtain a Boolean variant for a non-Boolean function. For example, the following is a Boolean variant of \(\ensuremath{\mathit{MULT}}\).

If we can compute \(\ensuremath{\mathit{BMULT}}\) via any programming language such as Python, C, Java, etc., we can compute \(\ensuremath{\mathit{MULT}}\) as well, and vice versa.

def F(x):
    res = []
    i = 0
    while BF(x,i,1):
        res.append(BF(x,i,0))
        i += 1
    return res

Formal Languages

For every Boolean function \(F:\{0,1\}^* \rightarrow \{0,1\}\), we can define the set \(L_F = \{ x | F(x) = 1 \}\) of strings on which \(F\) outputs \(1\). Such sets are known as languages. This name is rooted in formal language theory as pursued by linguists such as Noam Chomsky. A formal language is a subset \(L \subseteq \{0,1\}^*\) (or more generally \(L \subseteq \Sigma^*\) for some finite alphabet \(\Sigma\)). The membership or decision problem for a language \(L\), is the task of determining, given \(x\in \{0,1\}^*\), whether or not \(x\in L\). If we can compute the function \(F\), then we can decide membership in the language \(L_F\) and vice versa. Hence, many texts such as (Sipser, 1997) refer to the task of computing a Boolean function as “deciding a language”. In this book, we mostly describe computational tasks using the function notation, which is easier to generalize to computation with more than one bit of output. However, since the language terminology is so popular in the literature, we will sometimes mention it.

Restrictions of functions

If \(F:\{0,1\}^* \rightarrow \{0,1\}\) is a Boolean function and \(n\in \N\) then the restriction of \(F\) to inputs of length \(n\), denoted as \(F_n\), is the finite function \(f:\{0,1\}^n \rightarrow \{0,1\}\) such that \(f(x) = F(x)\) for every \(x\in \{0,1\}^n\). That is, \(F_n\) is the finite function that is only defined on inputs in \(\{0,1\}^n\), but agrees with \(F\) on those inputs. Since \(F_n\) is a finite function, it can be computed by a Boolean circuit, implying the following theorem:

In particular, Theorem 6.1 implies that there exists such a circuit collection \(\{ C_n \}\) even for the \(\ensuremath{\mathit{TWINP}}\) function we described before, even though we do not know of any program to compute it. Indeed, this is not that surprising: for every particular \(n\in \N\), \(\ensuremath{\mathit{TWINP}}_n\) is either the constant zero function or the constant one function, both of which can be computed by very simple Boolean circuits. Hence a collection of circuits \(\{ C_n \}\) that computes \(\ensuremath{\mathit{TWINP}}\) certainly exists. The difficulty in computing \(\ensuremath{\mathit{TWINP}}\) using Python or any other programming language arises from the fact that we do not know for each particular \(n\) what is the circuit \(C_n\) in this collection.

Deterministic finite automata (optional)

All our computational models so far - Boolean circuits and straight-line programs - were only applicable for finite functions.

In Chapter 7, we will present Turing machines, which are the central models of computation for unbounded input length functions. However, in this section we present the more basic model of deterministic finite automata (DFA). Automata can serve as a good stepping-stone for Turing machines, though they will not be used much in later parts of this book, and so the reader can feel free to skip ahead to Chapter 7. DFAs turn out to be equivalent in power to regular expressions: a powerful mechanism to specify patterns, which is widely used in practice. Our treatment of automata is relatively brief. There are plenty of resources that help you get more comfortable with DFAs. In particular, Chapter 1 of Sipser’s book (Sipser, 1997) contains an excellent exposition of this material. There are also many websites with online simulators for automata, as well as translators from regular expressions to automata and vice versa (see for example here and here).

At a high level, an algorithm is a recipe for computing an output from an input via a combination of the following steps:

1. Read a bit from the input
2. Update the state (working memory)
3. Stop and produce an output

For example, recall the Python program that computes the \(\ensuremath{\mathit{XOR}}\) function:

def XOR(X):
    '''Takes list X of 0's and 1's
       Outputs 1 if the number of 1's is odd and outputs 0 otherwise'''
    result = 0
    for i in range(len(X)):
        result = (result + X[i]) % 2
    return result

In each step, this program reads a single bit X[i] and updates its state result based on that bit (flipping result if X[i] is \(1\) and keeping it the same otherwise). When it is done transversing the input, the program outputs result. In computer science, such a program is called a single-pass constant-memory algorithm since it makes a single pass over the input and its working memory is finite. (Indeed, in this case, result can either be \(0\) or \(1\).) Such an algorithm is also known as a Deterministic Finite Automaton or DFA (another name for DFAs is a finite state machine). We can think of such an algorithm as a “machine” that can be in one of \(C\) states, for some constant \(C\). The machine starts in some initial state and then reads its input \(x\in \{0,1\}^*\) one bit at a time. Whenever the machine reads a bit \(\sigma \in \{0,1\}\), it transitions into a new state based on \(\sigma\) and its prior state. The output of the machine is based on the final state. Every single-pass constant-memory algorithm corresponds to such a machine. If an algorithm uses \(c\) bits of memory, then the contents of its memory can be represented as a string of length \(c\). Therefore such an algorithm can be in one of at most \(2^c\) states at any point in the execution.

We can specify a DFA of \(C\) states by a list of \(C \cdot 2\) rules. Each rule will be of the form “If the DFA is in state \(v\) and the bit read from the input is \(\sigma\) then the new state is \(v'\)”. At the end of the computation, we will also have a rule of the form “If the final state is one of the following … then output \(1\), otherwise output \(0\)”. For example, the Python program above can be represented by a two-state automaton for computing \(\ensuremath{\mathit{XOR}}\) of the following form:

• Initialize in the state \(0\).
• For every state \(s \in \{0,1\}\) and input bit \(\sigma\) read, if \(\sigma =1\) then change to state \(1-s\), otherwise stay in state \(s\).
• At the end output \(1\) iff \(s=1\).

We can also describe a \(C\)-state DFA as a labeled graph of \(C\) vertices. For every state \(s\) and bit \(\sigma\), we add a directed edge labeled with \(\sigma\) between \(s\) and the state \(s'\) such that if the DFA is at state \(s\) and reads \(\sigma\) then it transitions to \(s'\). (If the state stays the same then this edge will be a self-loop; similarly, if \(s\) transitions to \(s'\) in both the case \(\sigma=0\) and \(\sigma=1\) then the graph will contain two parallel edges.) We also label the set \(\mathcal{S}\) of states on which the automaton will output \(1\) at the end of the computation. This set is known as the set of accepting states. See Figure 6.3 for the graphical representation of the XOR automaton.

Formally, a DFA is specified by (1) the table of the \(C \cdot 2\) rules, which can be represented as a transition function \(T\) that maps a state \(s \in [C]\) and bit \(\sigma \in \{0,1\}\) to the state \(s' \in [C]\) which the DFA will transition to from state \(s\) on input \(\sigma\) and (2) the set \(\mathcal{S}\) of accepting states. This leads to the following definition.

def F(X):
    '''Return 1 iff X is a concatenation of zero/more copies of [0,1,0]'''
    if len(X) % 3 != 0:
        return False
    ultimate = 0
    penultimate = 1
    antepenultimate = 0
    for idx, b in enumerate(X):
        antepenultimate = penultimate
        penultimate = ultimate
        ultimate = b
        if idx % 3 == 2 and ((antepenultimate, penultimate, ultimate) != (0,1,0)):
            return False
    return True

Anatomy of an automaton (finite vs. unbounded)

Now that we are considering computational tasks with unbounded input sizes, it is crucial to distinguish between the components of our algorithm that have fixed length and the components that grow with the input size. For the case of DFAs these are the following:

Constant size components: Given a DFA \(A\), the following quantities are fixed independent of the input size:

• The number of states \(C\) in \(A\).

• The transition function \(T\) (which has \(2C\) inputs, and so can be specified by a table of \(2C\) rows, each entry in which is a number in \([C]\)).

• The set \(\mathcal{S} \subseteq [C]\) of accepting states. This set can be described by a string in \(\{0,1\}^C\) specifiying which states are in \(\mathcal{S}\) and which are not.

Together the above means that we can fully describe an automaton using finitely many symbols. This is a property we require out of any notion of “algorithm”: we should be able to write down a complete specification of how it produces an output from an input.

Components of unbounded size: The following quantities relating to a DFA are not bounded by any constant. We stress that these are still finite for any given input.

• The length of the input \(x\in \{0,1\}^*\) that the DFA is provided. The input length is always finite, but not a priori bounded.

• The number of steps that the DFA takes can grow with the length of the input. Indeed, a DFA makes a single pass on the input and so it takes precisely \(|x|\) steps on an input \(x\in \{0,1\}^*\).

DFA-computable functions

We say that a function \(F:\{0,1\}^* \rightarrow \{0,1\}\) is DFA computable if there exists some \(\ensuremath{\mathit{DFA}}\) that computes \(F\). In Chapter 4 we saw that every finite function is computable by some Boolean circuit. Thus, at this point, you might expect that every infinite function is computable by some DFA. However, this is very much not the case. We will soon see some simple examples of infinite functions that are not computable by DFAs, but for starters, let us prove that such functions exist.

Since the set of all Boolean functions is uncountable, we get the following corollary:

Regular expressions

Searching for a piece of text is a common task in computing. At its heart, the search problem is quite simple. We have a collection \(X = \{ x_0, \ldots, x_k \}\) of strings (e.g., files on a hard-drive, or student records in a database), and the user wants to find out the subset of all the \(x \in X\) that are matched by some pattern (e.g., all files whose names end with the string .txt). In full generality, we can allow the user to specify the pattern by specifying a (computable) function \(F:\{0,1\}^* \rightarrow \{0,1\}\), where \(F(x)=1\) corresponds to the pattern matching \(x\). That is, the user provides a program \(P\) in a programming language such as Python, and the system returns all \(x \in X\) such that \(P(x)=1\). For example, one could search for all text files that contain the string important document or perhaps (letting \(P\) correspond to a neural-network based classifier) all images that contain a cat. However, we don’t want our system to get into an infinite loop just trying to evaluate the program \(P\)! For this reason, typical systems for searching files or databases do not allow users to specify the patterns using full-fledged programming languages. Rather, such systems use restricted computational models that on the one hand are rich enough to capture many of the queries needed in practice (e.g., all filenames ending with .txt, or all phone numbers of the form (617) xxx-xxxx), but on the other hand are restricted enough so that queries can be evaluated very efficiently on huge files and in particular cannot result in an infinite loop.

One of the most popular such computational models is regular expressions. If you ever used an advanced text editor, a command-line shell, or have done any kind of manipulation of text files, then you have probably come across regular expressions.

A regular expression over some alphabet \(\Sigma\) is obtained by combining elements of \(\Sigma\) with the operation of concatenation, as well as \(|\) (corresponding to or) and \(*\) (corresponding to repetition zero or more times). (Common implementations of regular expressions in programming languages and shells typically include some extra operations on top of \(|\) and \(*\), but these operations can be implemented as “syntactic sugar” using the operators \(|\) and \(*\).) For example, the following regular expression over the alphabet \(\{0,1\}\) corresponds to the set of all strings \(x\in \{0,1\}^*\) where every digit is repeated at least twice:

The following regular expression over the alphabet \(\{ a,\ldots,z,0,\ldots,9 \}\) corresponds to the set of all strings that consist of a sequence of one or more of the letters \(a\)-\(d\) followed by a sequence of one or more digits (without a leading zero):

Formally, regular expressions are defined by the following recursive definition:

We will drop parentheses when they can be inferred from the context. We also use the convention that OR and concatenation are left-associative, and we give highest precedence to \(*\), then concatenation, and then OR. Thus for example we write \(00^*|11\) instead of \(((0)(0^*))|((1)(1))\).

Every regular expression \(e\) corresponds to a function \(\Phi_{e}:\Sigma^* \rightarrow \{0,1\}\) where \(\Phi_{e}(x)=1\) if \(x\) matches the regular expression. For example, if \(e = (00|11)^*\) then \(\Phi_e(110011)=1\) but \(\Phi_e(101)=0\) (can you see why?).

A Boolean function is called “regular” if it outputs \(1\) on precisely the set of strings that are matched by some regular expression. That is,

Regular expression can be defined over any finite alphabet \(\Sigma\), but as usual, we will mostly focus our attention on the binary case, where \(\Sigma = \{0,1\}\). Most (if not all) of the theoretical and practical general insights about regular expressions can be gleaned from studying the binary case.

Algorithms for matching regular expressions

Regular expressions would not be very useful for search if we could not evaluate, given a regular expression \(e\), whether a string \(x\) is matched by \(e\). Luckily, there is an algorithm to do so. Specifically, there is an algorithm (think “Python program” though later we will formalize the notion of algorithms using Turing machines) that on input a regular expression \(e\) over the alphabet \(\{0,1\}\) and a string \(x\in \{0,1\}^*\), outputs \(1\) iff \(e\) matches \(x\) (i.e., outputs \(\Phi_e(x)\)).

Indeed, Definition 6.7 actually specifies a recursive algorithm for computing \(\Phi_{e}\). Specifically, each one of our operations -concatenation, OR, and star- can be thought of as reducing the task of testing whether an expression \(e\) matches a string \(x\) to testing whether some sub-expressions of \(e\) match substrings of \(x\). Since these sub-expressions are always shorter than the original expression, this yields a recursive algorithm for checking if \(e\) matches \(x\), which will eventually terminate at the base cases of the expressions that correspond to a single symbol or the empty string.

We assume above that we have a procedure \(\text{\textsc{MatchEmpty}}\) that on input a regular expression \(e\) outputs \(1\) if and only if \(e\) matches the empty string \(\ensuremath{\text{\texttt{""}}}\).

The key observation is that in our recursive definition of regular expressions, whenever \(e\) is made up of one or two expressions \(e',e''\) then these two regular expressions are smaller than \(e\). Eventually (when they have size \(1\)) then they must correspond to the non-recursive case of a single alphabet symbol. Correspondingly, the recursive calls made in Algorithm 6.10 always correspond to a shorter expression or (in the case of an expression of the form \((e')^*\)) a shorter input string. Thus, we can prove the correctness of Algorithm 6.10 on inputs of the form \((e,x)\) by induction over \(\min \{ |e|, |x| \}\). The base case is when either \(x=\ensuremath{\text{\texttt{""}}}\) or \(e\) is a single alphabet symbol, \(\ensuremath{\text{\texttt{""}}}\) or \(\emptyset\). In the case the expression is of the form \(e=(e'|e'')\) or \(e=(e')(e'')\), we make recursive calls with the shorter expressions \(e',e''\). In the case the expression is of the form \(e=(e')^*\), we make recursive calls with either a shorter string \(x\) and the same expression, or with the shorter expression \(e'\) and a string \(x'\) that is equal in length or shorter than \(x\).

Efficient matching of regular expressions (optional)

Algorithm 6.10 is not very efficient. For example, given an expression involving concatenation or the “star” operation and a string of length \(n\), it can make \(n\) recursive calls, and hence it can be shown that in the worst case Algorithm 6.10 can take time exponential in the length of the input string \(x\). Fortunately, it turns out that there is a much more efficient algorithm that can match regular expressions in linear (i.e., \(O(n)\)) time. Since we have not yet covered the topics of time and space complexity, we describe this algorithm in high level terms, without making the computational model precise. Rather we will use the colloquial notion of \(O(n)\) running time as used in introduction to programming courses and whiteboard coding interviews. We will see a formal definition of time complexity in Chapter 13.

The implicit constant in the \(O(n)\) term of Theorem 6.12 depends on the expression \(e\). Thus, another way to state Theorem 6.12 is that for every expression \(e\), there is some constant \(c\) and an algorithm \(A\) that computes \(\Phi_e\) on \(n\)-bit inputs using at most \(c\cdot n\) steps. This makes sense since in practice we often want to compute \(\Phi_e(x)\) for a small regular expression \(e\) and a large document \(x\). Theorem 6.12 tells us that we can do so with running time that scales linearly with the size of the document, even if it has (potentially) worse dependence on the size of the regular expression.

We prove Theorem 6.12 by obtaining more efficient recursive algorithm, that determines whether \(e\) matches a string \(x\in \{0,1\}^n\) by reducing this task to determining whether a related expression \(e'\) matches \(x_0,\ldots,x_{n-2}\). This will result in an expression for the running time of the form \(T(n) = T(n-1) + O(1)\) which solves to \(T(n)=O(n)\).

Restrictions of regular expressions. The central definition for the algorithm behind Theorem 6.12 is the notion of a restriction of a regular expression. The idea is that for every regular expression \(e\) and symbol \(\sigma\) in its alphabet, it is possible to define a regular expression \(e[\sigma]\) such that \(e[\sigma]\) matches a string \(x\) if and only if \(e\) matches the string \(x\sigma\). For example, if \(e\) is the regular expression \((01)^*(01)\) (i.e., one or more occurrences of \(01\)) then \(e[1]\) is equal to \((01)^*0\) and \(e[0]\) will be \(\emptyset\). (Can you see why?)

Algorithm 6.13 computes the restriction \(e[\sigma]\) given a regular expression \(e\) and an alphabet symbol \(\sigma\). It always terminates, since the recursive calls it makes are always on expressions smaller than the input expression. Its correctness can be proven by induction on the length of the regular expression \(e\), with the base cases being when \(e\) is \(\ensuremath{\text{\texttt{""}}}\), \(\emptyset\), or a single alphabet symbol \(\tau\).

Using this notion of restriction, we can define the following recursive algorithm for regular expression matching:

By the definition of a restriction, for every \(\sigma\in \Sigma\) and \(x'\in \Sigma^*\), the expression \(e\) matches \(x'\sigma\) if and only if \(e[\sigma]\) matches \(x'\). Hence for every \(e\) and \(x\in \Sigma^n\), \(\Phi_{e[x_{n-1}]}(x_0\cdots x_{n-2}) = \Phi_e(x)\) and Algorithm 6.14 does return the correct answer. The only remaining task is to analyze its running time. Note that Algorithm 6.14 uses the \(\text{\textsc{MatchEmpty}}\) procedure of Solved Exercise 6.3 in the base case that \(x=\ensuremath{\text{\texttt{""}}}\). However, this is OK since this procedure’s running time depends only on \(e\) and is independent of the length of the original input.

For simplicity, let us restrict our attention to the case that the alphabet \(\Sigma\) is equal to \(\{0,1\}\). Define \(C(\ell)\) to be the maximum number of operations that Algorithm 6.13 takes when given as input a regular expression \(e\) over \(\{0,1\}\) of at most \(\ell\) symbols. The value \(C(\ell)\) can be shown to be polynomial in \(\ell\), though this is not important for this theorem, since we only care about the dependence of the time to compute \(\Phi_e(x)\) on the length of \(x\) and not about the dependence of this time on the length of \(e\).

Algorithm 6.14 is a recursive algorithm that input an expression \(e\) and a string \(x\in \{0,1\}^n\), does computation of at most \(C(|e|)\) steps and then calls itself with input some expression \(e'\) and a string \(x'\) of length \(n-1\). It will terminate after \(n\) steps when it reaches a string of length \(0\). So, the running time \(T(e,n)\) that it takes for Algorithm 6.14 to compute \(\Phi_e\) for inputs of length \(n\) satisfies the recursive equation:

(In the base case \(n=0\), \(T(e,0)\) is equal to some constant depending only on \(e\).) To get some intuition for the expression Equation 6.2, let us open up the recursion for one level, writing \(T(e,n)\) as

Continuing this way, we can see that \(T(e,n) \leq n \cdot C(L) + O(1)\) where \(L\) is the largest length of any expression \(e'\) that we encounter along the way. Therefore, the following claim suffices to show that Algorithm 6.14 runs in \(O(n)\) time:

Claim: Let \(e\) be a regular expression over \(\{0,1\}\), then there is a number \(L(e) \in \N\), such that for every sequence of symbols \(\alpha_0,\ldots,\alpha_{n-1}\), if we define \(e' = e[\alpha_0][\alpha_1]\cdots [\alpha_{n-1}]\) (i.e., restricting \(e\) to \(\alpha_0\), and then \(\alpha_1\) and so on and so forth), then \(|e'| \leq L(e)\).

Proof of claim: For a regular expression \(e\) over \(\{0,1\}\) and \(\alpha\in \{0,1\}^m\), we denote by \(e[\alpha]\) the expression \(e[\alpha_0][\alpha_1]\cdots [\alpha_{m-1}]\) obtained by restricting \(e\) to \(\alpha_0\) and then to \(\alpha_1\) and so on. We let \(S(e) = \{ e[\alpha] | \alpha \in \{0,1\}^* \}\). We will prove the claim by showing that for every \(e\), the set \(S(e)\) is finite, and hence so is the number \(L(e)\) which is the maximum length of \(e'\) for \(e'\in S(e)\).

We prove this by induction on the structure of \(e\). If \(e\) is a symbol, the empty string, or the empty set, then this is straightforward to show as the most expressions \(S(e)\) can contain are the expression itself, \(\ensuremath{\text{\texttt{""}}}\), and \(\emptyset\). Otherwise we split to the two cases (i) \(e = e'^*\) and (ii) \(e = e'e''\), where \(e',e''\) are smaller expressions (and hence by the induction hypothesis \(S(e')\) and \(S(e'')\) are finite). In the case (i), if \(e = (e')^*\) then \(e[\alpha]\) is either equal to \((e')^* e'[\alpha]\) or it is simply the empty set if \(e'[\alpha]=\emptyset\). Since \(e'[\alpha]\) is in the set \(S(e')\), the number of distinct expressions in \(S(e)\) is at most \(|S(e')|+1\). In the case (ii), if \(e = e' e''\) then all the restrictions of \(e\) to strings \(\alpha\) will either have the form \(e' e''[\alpha]\) or the form \(e' e''[\alpha] | e'[\alpha']\) where \(\alpha'\) is some string such that \(\alpha = \alpha' \alpha''\) and \(e''[\alpha'']\) matches the empty string. Since \(e''[\alpha] \in S(e'')\) and \(e'[\alpha'] \in S(e')\), the number of the possible distinct expressions of the form \(e[\alpha]\) is at most \(|S(e'')| + |S(e'')|\cdot |S(e')|\). This completes the proof of the claim.

The bottom line is that while running Algorithm 6.14 on a regular expression \(e\), all the expressions we ever encounter are in the finite set \(S(e)\), no matter how large the input \(x\) is, and so the running time of Algorithm 6.14 satisfies the equation \(T(n) = T(n-1) + C'\) for some constant \(C'\) depending on \(e\). This solves to \(O(n)\) where the implicit constant in the O notation can (and will) depend on \(e\) but crucially, not on the length of the input \(x\).

Matching regular expressions using DFAs

Theorem 6.12 is already quite impressive, but we can do even better. Specifically, no matter how long the string \(x\) is, we can compute \(\Phi_e(x)\) by maintaining only a constant amount of memory and moreover making a single pass over \(x\). That is, the algorithm will scan the input \(x\) once from start to finish, and then determine whether or not \(x\) is matched by the expression \(e\). This is important in the common case of trying to match a short regular expression over a huge file or document that might not even fit in our computer’s memory. Of course, as we have seen before, a single-pass constant-memory algorithm is simply a deterministic finite automaton. As we will see in Theorem 6.17, a function can be computed by regular expression if and only if it can be computed by a DFA. We start with showing the “only if” direction:

Equivalence of regular expressions and automata

Recall that a Boolean function \(F:\{0,1\}^* \rightarrow \{0,1\}\) is defined to be regular if it is equal to \(\Phi_e\) for some regular expression \(e\). (Equivalently, a language \(L \subseteq \{0,1\}^*\) is defined to be regular if there is a regular expression \(e\) such that \(e\) matches \(x\) iff \(x\in L\).) The following theorem is the central result of automata theory:

Closure properties of regular expressions

If \(F\) and \(G\) are regular functions computed by the expressions \(e\) and \(f\) respectively, then the expression \(e|f\) computes the function \(H = F \vee G\) defined as \(H(x) = F(x) \vee G(x)\). Another way to say this is that the set of regular functions is closed under the OR operation. That is, if \(F\) and \(G\) are regular then so is \(F \vee G\). An important corollary of Theorem 6.17 is that this set is also closed under the NOT operation:

Since \(a \wedge b = \overline{\overline{a} \vee \overline{b}}\), Lemma 6.18 implies that the set of regular functions is closed under the AND operation as well. Moreover, since OR, NOT and AND are a universal basis, this set is also closed under NAND, XOR, and any other finite function. That is, we have the following corollary:

Limitations of regular expressions and the pumping lemma

The efficiency of regular expression matching makes them very useful. This is why operating systems and text editors often restrict their search interface to regular expressions and do not allow searching by specifying an arbitrary function. However, this efficiency comes at a cost. As we have seen, regular expressions cannot compute every function. In fact, there are some very simple (and useful!) functions that they cannot compute. Here is one example:

Lemma 6.20 is a consequence of the following result, which is known as the pumping lemma:

Using the pumping lemma, we can easily prove Lemma 6.20 (i.e., the non-regularity of the “matching parenthesis” function):

The pumping lemma is a very useful tool to show that certain functions are not computable by a regular expression. However, it is not an “if and only if” condition for regularity: there are non-regular functions that still satisfy the pumping lemma conditions. To understand the pumping lemma, it is crucial to follow the order of quantifiers in Theorem 6.21. In particular, the number \(n_0\) in the statement of Theorem 6.21 depends on the regular expression (in the proof we chose \(n_0\) to be twice the number of symbols in the expression). So, if we want to use the pumping lemma to rule out the existence of a regular expression \(e\) computing some function \(F\), we need to be able to choose an appropriate input \(w\in \{0,1\}^*\) that can be arbitrarily large and satisfies \(F(w)=1\). This makes sense if you think about the intuition behind the pumping lemma: we need \(w\) to be large enough as to force the use of the star operator.

For yet another example of a pumping-lemma based proof, see Figure 6.10 which illustrates a cartoon of the proof of the non-regularity of the function \(F:\{0,1\}^* \rightarrow \{0,1\}\) which is defined as \(F(x)=1\) iff \(x=0^n1^n\) for some \(n\in \N\) (i.e., \(x\) consists of a string of consecutive zeroes, followed by a string of consecutive ones of the same length).

Answering semantic questions about regular expressions

Regular expressions have applications beyond search. For example, regular expressions are often used to define tokens (such as what is a valid variable identifier, or keyword) in the design of parsers, compilers and interpreters for programming languages. Regular expressions have other applications too: for example, in recent years, the world of networking moved from fixed topologies to “software defined networks”. Such networks are routed by programmable switches that can implement policies such as “if packet is secured by SSL then forward it to A, otherwise forward it to B”. To represent such policies we need a language that is on one hand sufficiently expressive to capture the policies we want to implement, but on the other hand sufficiently restrictive so that we can quickly execute them at network speed and also be able to answer questions such as “can C see the packets moved from A to B?”. The NetKAT network programming language uses a variant of regular expressions to achieve precisely that. For this application, it is important that we are not merely able to answer whether an expression \(e\) matches a string \(x\) but also answer semantic questions about regular expressions such as “do expressions \(e\) and \(e'\) compute the same function?” and “does there exist a string \(x\) that is matched by the expression \(e\)?”. The following theorem shows that we can answer the latter question:

Using Theorem 6.23, we can obtain an algorithm that determines whether or not two regular expressions \(e\) and \(e'\) are equivalent, in the sense that they compute the same function.

Exercises

Bibliographical notes

The relation of regular expressions with finite automata is a beautiful topic, on which we only touch upon in this text. It is covered more extensively in (Sipser, 1997) (Hopcroft, Motwani, Ullman, 2014) (Kozen, 1997) . These texts also discuss topics such as non-deterministic finite automata (NFA) and the relation between context-free grammars and pushdown automata.

The automaton of Figure 6.4 was generated using the FSM simulator of Ivan Zuzak and Vedrana Jankovic. Our proof of Theorem 6.12 is closely related to the Myhill-Nerode Theorem. One direction of the Myhill-Nerode theorem can be stated as saying that if \(e\) is a regular expression then there is at most a finite number of strings \(z_0,\ldots,z_{k-1}\) such that \(\Phi_{e[z_i]} \neq \Phi_{e[z_j]}\) for every \(0 \leq i\neq j < k\).

Comments

Comments are posted on the GitHub repository using the utteranc.es app. A GitHub login is required to comment. If you don't want to authorize the app to post on your behalf, you can also comment directly on the GitHub issue for this page.

Compiled on 12/06/2023 00:07:03

Copyright 2023, Boaz Barak.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Produced using pandoc and panflute with templates derived from gitbook and bookdown.