Fabulous Adventures In Coding : Regular Expressions

Regular Expressions From Scratch, Part Twelve: Superposition of States

Eric Lippert — Fri, 20 Jan 2006 22:47:00 GMT

Happy New Year everyone. Over the break I had a wonderful time reconnecting with my friends and family. And of course I came back to a huge pile of work! We're going through the flaws that were discovered in C# 2.0 too late in the cycle to risk fixing, and some of them illustrate interesting corner cases in the language. But those will have to wait, as we've still got a lot of ground to cover before I'm done this crazy long series on regular expressions.

To summarize the story so far: we've defined alphabets and languages over those alphabets. We've shown how to create a special "regular expression" alphabet and language for any given alphabet. We've also come up with a rule that associates a language with a regular expression. This language is the set of strings which "matches" the regular expression.

We've determined that there are deterministic finite state computers which consume one character of a string at a time that can "recognize" strings from some regular languages. We've also determined that there are "nondeterministic" finite automata that somehow "magically" figure out which rules to apply at any time to match a string.

We've also shown that any NFA that has rules that act on more than one character can be turned into an equivalent NFA which only has single-character or no-character transitions.

What we would like to do is show the following three facts: first, that every NFA is equivalent to a DFA. Second, that every DFA/NFA recognizes a regular language. And third, that every regular language has a DFA/NFA that recognizes it.

We've still got a ways to go to show that first result. Rather than prove the general result, which will be tedious, I'll just go through an example and hope that it is illustrative enough to convince you that we could give this treatment to any NFA.

Let M be an NFA with alphabet {a, b}, states {1, 2, 3, 4}, start state is 1, and the set of acceptable states is {4}. We'll assume that we've already eliminated all the multi-character rules and have a set of rules as follows:

(1, e) → 2
(1, b) → 3
(2, a) → 1
(2, e) → 3
(2, a) → 4
(3, b) → 4

This NFA accepts the language a*( bb ∪ b ∪ a ), and as you can see, we've got lots of empty and ambiguous rules.

Here's the trick: given an NFA with 4 states like this one, we can find a DFA which is equivalent to this guy that has 2⁴ = 16 states or fewer. We can think of the NFA as at any time living in a "superstate" that consists of all of the states that it COULD be in right now.

Let me try to explain that a little better. We start in state 1, right? But since (1, e) → 2 and (2, e) → 3, we could also be starting in states 2 or 3 before we process any input. Let's create a new automaton with a start state that represents the concept "right now M could be in states 1, 2, or 3." We'll call that state 123x, which is one big symbol, not a string of four symbols. In our new automaton, the start state is 123x.

We're trying to write a DFA here, so there needs to be a rule for every input and every state:

(123x, a) → ?
(123x, b) → ?

Look at the original automaton. When we were in states 1, 2 or 3, what were the possible state transitions for a? The only ones were (2, a) → 1 and (2, a) → 4. But we're not finished! Again, we need to consider what could happen from states 1 and 4 if we processed the "empty string" transition rules. Since (1, e) → 2 and (2, e) → 3, we could end up in state 1, 2, 3 or 4. Let's create a new state called 1234 for our DFA and finish off that rule:

(123x, a) → 1234

Now do the same analysis for (123x, b) and we'll see that in the original automaton, the only possible resulting states starting in 1, 2 or 3, and processing a b are 3 and 4. Add another new state:

(123x, b) → xx34

Are we done? No. We've added two more states, 1234 and xx34, so we need to figure out the state transitions for them too, which we do by the same process. We discover that

(1234, a) → 1234
(1234, b) → xx34
(xx34, a) → ?
(xx34, b) → xxx4

Uh oh. There are no state transitions in the original NFA that start in states 3 or 4 and take an a. Remember that in that case we assume that the NFA goes into a special "reject" state. Let's call the reject state xxxx.

We now have two more new states to work out the transitions for. If we're in the reject state, we stay in the reject state, and we see from the original NFA that there are no transitions out of state 4, so we can round out our list with:

(xx34, a) → xxxx
(xxxx, a) → xxxx
(xxxx, b) → xxxx
(xxx4, a) → xxxx
(xxx4, b) → xxxx

Since 4 was the acceptable state in the original NFA, any "superstate" that contains 4 must be an acceptable state. So our new DFA is a machine M₂ with alphabet {a}, b}, states {1234, 123x, xx34, xxx4, xxxx}, start state is 123x, and the set of acceptable states is {1234, xx34, xxx4}. The rules are

(1234, a) → 1234
(1234, b) → xx34
(123x, a) → 1234
(123x, b) → xx34
(xx34, a) → xxxx
(xx34, b) → xxx4
(xxx4, a) → xxxx
(xxx4, b) → xxxx
(xxxx, a) → xxxx
(xxxx, b) → xxxx

This DFA accepts the language a*( bb ∪ b ∪ a ), so we've found an equivalent DFA to our NFA. We've turned an NFA with four states and five rules into an equivalent DFA with five states and ten rules. We did well -- you can find NFAs that require 2ⁿ new states, where n is the number of NFA states. An NFA with a thousand states could require 2¹⁰⁰⁰ states to represent as a DFA! 2¹⁰⁰⁰ is a finite, albeit rather large number. But it's not that big. Clearly we could build a 2¹⁰⁰⁰ state machine with only 1000 bits to store the state information. It's the state transition rules that are the pain to work out!

Using the techniques from this and the previous entry we can turn any NFA into an equivalent DFA, so NFAs are really not magic at all. They're just a convenient way to talk about DFAs.

Now, obviously what I've done here isn't a proof; one example does not a proof make. But the proof is both tedious and complicated, so I'm going to skip it. The key result here is that we can stop saying "deterministic finite automaton" and "nondeterministic finite automaton" and just start saying "finite automaton", because they're essentially the same thing.

Now that we know that we can use NFAs with impunity, we can try to answer questions such as:

Is every regular language recognizable by a finite automaton?

Are there any FAs that recognize irregular languages?

Tune in next time to find out!

Regular Expressions From Scratch, Part Eleven: Eliminating Multi-Symbol Rules

Eric Lippert — Thu, 22 Dec 2005 18:00:00 GMT

The story so far: we have deterministic and nondeterministic finite automata. DFAs follow a rigid, fully specified set of rules which, on any string, yield either an accept or reject state after exactly as many moves as the string has characters. NFAs, on the other hand, are poorly specified. They somehow magically are able to choose which rules to apply given the input to find a path that yields an accept state, if one exists.

NFA magic buys us a lot, but at the high price that we can now no longer easily see a clear relationship between this magic box and a computer we could actually build. We need to get back to reality. Here is the amazing result that I'm going to try to justify in your minds:

Every NFA is equivalent to some DFA. Even better: given a description of an NFA, we can construct a description of an equivalent DFA.

In other words, NFA magic doesn't buy us any extra power - no NFA can recognize a language that some DFA cannot also recognize.

This is excellent news, because it means that we can describe and reason about machines using short, convenient, vague nondeterministic notation, but still have confidence that we could build a fully deterministic machine that did exactly the same job.

But how the heck are we going to motivate that amazing result?

One step at a time. We're going to take some NFAs and make simple transformations that gradually turn them into different but equivalent NFAs, and eventually one of those NFAs will actually be a DFA. Rather than giving a full formal proof, which would be tedious and boring, I'm going to rely on the sketch being convincing enough that you believe me that we can take any NFA and turn it into a DFA. Maybe later in this series we'll actually write some C# code that implements the transformation.

The magical bits that we need to remove are: rules can have multiple characters in state transitions, rules can have "empty" state transitions, rules can have ambiguous state transitions, there can be situations for which no stated rule applies, and there's a magical 'crash' state. Once we eliminate all those weirdnesses, we'll be left with a DFA.

Step one: Any NFA that has rules that have multiple-character state transitions is equivalent to some NFA that has only single-character or empty-string transitions.

We can get rid of the multiple-character transitions by adding one new state for each extra character. Recall our previous example:

Alphabet: S = {a, b}
States: K = {0}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,ab) → 0
(0,aba) → 0

Suppose we added three new (non-acceptable) states, 1, 2 and 3. You agree I hope that the first rule is equivalent to these two rules:

(0,a) → 1
(1,b) → 0

And the second rule is equivalent to these three rules:

(0,a) → 2
(2,b) → 3
(3,a) → 0

We never need to add more than a finite number of new states or rules, so the new machine is always still an NFA.

Throw away the original two rules and hey presto, we have an equivalent NFA where every rule only has a zero or one-symbol string. Multi-symbol transition rules are no longer magical; we can eliminate them easily.

This example above has no empty-string transitions. Next time we'll pick another example that has empty string transitions and one-character transitions, and then show that we can turn something that looks like that into a DFA.

"Next time" will likely be in the new year, as I am heading back to my ancestral home for the next week to visit friends and relatives during this festive holiday season. I hope all of you have a fun and safe holiday; thanks for reading and we'll see you in 2006.

Regular Expressions From Scratch, Part Ten: Magic!

Eric Lippert — Mon, 19 Dec 2005 18:00:00 GMT

Let's recap the story so far.

Starting only with basic set theory, sequences, symbols and numbers, we've defined alphabets, languages, regular expressions, and the mapping between regular expressions and regular languages. We've also defined deterministic finite automata as machines that take in strings one character at a time, change their internal state to one of a finite set according to strict rules, and end up in either an accept or reject state as output.

Where we're going with this is towards unification of these two concepts. We want to show that for every DFA there's a regexp, and vice versa. But to get there is going to take some magic.

The trouble with DFAs is that they're kind of a pain to specify. You have an alphabet with, say, a hundred symbols, and you have, say, fifty states. You need to come up with a state transition rule for 100 x 50 = 5000 possible combinations. Since most of those will likely be transitions to rejection states anyway, that's majorly boring.

I'm hereby declaring that we have nondeterministic finite automata. An NFA works just like a DFA, except that the "rules" for determining the state transitions can be ambiguous and weird. We'll write an NFA like this:

Let M₁ be an NFA such that:

Alphabet: S = {a, b}
States: K = {0, 1, 2}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,a) → 1
(1,b) → 2
(2,a) → 0
(2,e) → 0

Notice that we're not specifying the six rules we ought to be, and one of those rules is a transition on an "empty input"! This last rule means that if we are in state 2, we can go to state 0 "for free", without consuming any characters.

We say that M₁ accepts a string if there is any way to make it eventually yield an acceptable state with no more input. For example, suppose we start with ababa:

(0, ababa) → (1, baba) → (2, aba) → (0, aba) → (1, ba) → (2, a) → (0, e)

is one path to an acceptable state. Even though

(0, ababa) → (1, baba) → (2, aba) → (0, ba) gets us stuck in a state that we can go no further in, that's OK. NFAs are magic. If there exists any path such that (0, ababa) ⶒ (0, e) then the NFA will find it and end in an accepting state. The fact that some nonsensical or rejecting paths exist doesn't matter if an accepting path exists. Only if every possible path either ends in a rejecting state, or leaves us unable to find any applicable rule, does the NFA reject the string.

In fact, this machine accepts the regular language (ab∪aba)*

(Recall that I am being sloppy about parenthesizing. This isn't a "real" regular expression but you get the point that when I say that I mean L(((ab)∪((ab)a))*) I'm sure.)

We've got lots of magic already, so let's add more. We'll say that a rule can consume any number of matching symbols that it wants.

Let M₂ be an NFA such that:

Alphabet: S = {a, b}
States: K = {0}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,ab) → 0
(0,aba) → 0

This machine clearly accepts (ab∪aba)* except for the fact that it is now totally unclear what we mean by "accepts" vs. "rejects" in a machine with only one state! How does it reject anything?

Let's assume that NFAs have a magic implicit "crash" state which means "For this input I was unable to find any path that did not end in a state where I lacked a rule telling me what to do next." The crash state is always a rejecting state.

Clearly M₁ and M₂ accept the same language. Let's make up a word for that relationship.

Definition 15: Two machines are equivalent if they both accept exactly the same language.

Next time: Can we in fact build a device which does magic?

Regular Expressions From Scratch, Part Nine: A Dream of a Machine

Eric Lippert — Thu, 15 Dec 2005 18:28:00 GMT

I want to come up with the simplest possible device that can identify whether a given string is a member of a given regular language. We need some kind of computer, but to make it easy to analyze, I want to put as many restrictions upon it as possible. For example, I want there to be very limited memory storage, I want to process only one character at a time, and so on.

Definition 14: A deterministic finite automaton (DFA) is an idealized machine. It has an input tape with a finite string written down on it in a given alphabet. The tape is exactly as long as the string. The DFA has a finite number of distinct states that it can be in. A DFA can be in only one state at a time, called the current state. The DFA always starts in a particular state called the start state. The DFA reads the string from left to right, one symbol at a time. There is no backtracking allowed. Each new symbol on the input tape may cause the machine to change the current state, but this choice can only be based on the current symbol and current state, and the rules for how that choice is made must be specified in advance and cannot change. Every state is either an acceptable state or an unacceptable state. When the machine is done reading the string, if it is in an acceptable state then the machine accepts the string, otherwise it rejects it.

I think you'll agree that this is a very limiting set of restrictions to place upon a computer.

We'll write down DFAs like this example:

Let M be a DFA such that:
Alphabet: S = {a, b}
States: K = {0, 1}
Start: s = 0
Acceptable: F = {1}

Every DFA has rules for determining what the state transitions are, one rule for every possible combination of states and symbols. For example, M might have rules:

current state	current input	new state
0	a	1
0	b	1
1	a	0
1	b	1

Note that there MUST be a rule for every combination of K and S. Since it will get cumbersome to write out these tables constantly I'm going to declare a new notation:

(0, a) → 1
(0, b) → 1
(1, a) → 0
(1, b) → 1

The arrow is read as "yields".

Consider the action of M on a tape containing aab. The machine uses these rules in this order:

(0, a) → 1
(1, a) → 0
(0, b) → 1

The string is accepted because we've run out of string and we're in an acceptable state.

Since any set of strings is a language, the set of strings which a machine M accepts forms a language. We can construct a finite sequence of sets (alphabet, states, transition rules, etc) that exactly characterizes M, so we can build a function L which maps M onto the language which M accepts. Call the language L(M).

Convince yourself that in the case above, L(M) = { (a(aa)*) ∪ (((a ∪ b)*b) (aa)*) } - that is, the regular language where every string is either an odd number of as or any string ending in a b followed by an even number of as.

This idea of listing the rules used by the machine is pretty good, but it requires us to mentally keep track of how far along the input tape the machine is. Also, there's some redundancy there, since obviously the state of the next line is going to be the same as the new state of the current line. I'm therefore going to change the notation I use to describe how a machine is working. For the example above we'd say in our new notation

(0, aab) → (1, ab) → (0, b) → (1, e)

Clearly this is the very definition of a mechanical process. It's going to get tedious to write out all of the steps in more complex machines. Let me declare a new "super arrow", which means "yields eventually".

(0, aab) ⇒ (1, e)

I'll use the super arrow to mean "yields in zero or more steps". Maybe one, maybe a thousand, but some finite number of steps.

What's the point of a DFA? It's pretty much the simplest thing we can possibly call a computer. It's got input and output and storage, but is very limited. The input, sure, it can be as long or as short as you want. But the output and the storage consists of a single "register" which can only hold one of a finite number of states. In our example today, we've got only a single bit of storage and four rules, and we can already accept a fairly complex language.

If we can reason about the limits of a DFA then we can determine whether a machine with finite storage is buff enough to recognize any interesting languages. Today's machine recognizes a regular language, and that's a start.

Next time, we'll add a little magic to a DFA.

Regular Expressions From Scratch, Part Eight: The Diagonal Argument

Eric Lippert — Mon, 12 Dec 2005 18:00:00 GMT

As we know, each regular expression is associated with a language by our function L. We also determined last time that we could list all members of S* and R in order first by length and then by alphabetical order. Here's a weird question: is the nth string in S*'s alphabetical ordering a member of the language associated with the nth alphabetical regular expression?

Let's make a table and see if we can discern any pattern.

s	r	s in L(r)?
e	a	no
a	b	no
b	Ø	no
aa	a*	yes
ab	b*	no
ba	Ø*	no
bb	a**	no
aaa	b**	no
aab	Ø**	no
aba	(aa)	no

We could continue this table forever, of course, and it seems that sometimes, pretty much at random, we're going to get a match. Matches might not be very often, but they will happen from time to time. As it turns out, it doesn't really matter whether we can come up with some pattern here. All that matters is that we can answer the question definitively for every possible entry.

Now define the language D such that D = { w in S* such that w has "no" in the column above }

D = {e, a, b, ab, ba, bb, aaa, aab, aba, abb, … }

This is a weird but perfectly well-defined property. Given any string we can determine very quickly whether it is a member of D or not. Just figure out where on the table it is, compute the nth regular expression, and see if it is a member of that regular language. If it is not, then it is in D.

Is there any regular expression that specifies the language D?

Which one would it be? We've got a complete alphabetical list of regular expressions, so let's just go down the list. By our definition of D, clearly it can't be any of them. If the third column is "yes" then the nth regexp matches a string not in D. If it is "no" then the nth regexp fails to match a string in D. Therefore there is no such regular expression that matches everything in D. Therefore D is not a regular language.

By a similar argument we can show that for every nonempty alphabet there exists a language which is not regular.

That's a pretty unlikely example of an irregular language though. Later we'll see that many perfectly normal languages are not regular, including, ironically enough, the regular expression language itself.

Remember that the reason why we came up with regular expressions in the first place is because we wanted a concise way to characterize languages using short, simple expressions. We of course could come up with other mechanical means to cleverly map between strings in one alphabet and languages in another, but ultimately it wouldn't do us much good if our aim is to capture all possible languages. The diagonal argument above doesn't depend upon any special features of regular expressions; it applies to any function that maps strings onto languages.

No language description system which maps finite strings onto languages can describe every language.

This is a stunning result; there are languages which we cannot characterize in any finite number of symbols, no matter how clever we are with our symbolic manipulations!

It's simply a fundamental fact that the definition of "language" we've chosen - an arbitrary set of finite strings - affords an immense number of possible results, more immense than the number of strings in any description language you'd care to name. Some - infinitely many - will be indefinable.

You might have noticed a bit of a hand-wave in today's entry: I've made the claim that we can easily take a regular expression and determine if a string is in its regular language. Over the next few entries we'll justify that claim by building some simple abstract machines that can make this determination.

Regular Expressions From Scratch, Part Seven: Listing All Members Of A Language In Order

Eric Lippert — Thu, 08 Dec 2005 18:00:00 GMT

Regular languages are by definition those languages which can be described by a regular expression. It should be clear from the definition that the union of any finite number of regular languages is a regular language, the concatenation of any finite number of regular languages is a regular language, and the Kleene Closure of any regular language is a regular language.

That's rather a lot of languages. Are all languages regular languages?

We should have a strong intuition that regular languages are a very limited subset of all languages. It seems like it would be hard to come up with a finite regular expression that, say, matches the language {w is in 1* such that there are a prime number of 1s in w}.

But we can do better than intuition. There are many interesting ways to prove that there exists at least one non-regular language. Over the next couple entries we'll find a non-regular language by using a clever trick. The first thing we'll need to do is enumerate every member of a language.

It doesn't really matter how we enumerate the language, as long as we guarantee that we eventually hit every member of the language. Here's one way to do such an enumeration.

Consider an alphabet, say S = {a, b}. We can enumerate all the strings of S* first by length and then by alphabetical order. Of course, we'll need to define an order for our alphabet, but since alphabets are by definition finite sets, we can choose any old order. In general, for alphabets consisting of Roman alphabet characters and numbers we'll use the standard alphabetical ordering we're all used to. We'll start with the one zero-length string, then the two one-symbol strings, then the four two-symbol strings, and so on:

e, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, …

(A commenter correctly pointed out that there are languages which cannot be easily enumerated in this way, but my coming argument does not rely upon an alphabetical ordering. All we need is some schema for enumerating a language which guarantees that we eventually get all of them.)

Consider R, the regular expression language of S. We can enumerate it too, first by length and then by alphabetical order. Let's say that alphabetical order of R is * ( ) a b Ø ∪ just to pick an arbitrary ordering for the symbols. We can then enumerate R in the same way like this:

a, b, Ø, a*, b*, Ø*, a**, b**, Ø**, (aa), (ab), (aØ), (ba), (bb), (bØ), (Øa), (Øb), (ØØ), …

Of course we are leaving out any strings which are not in R, such as * or ))(.

Next time we'll use the fact that we can make both these lists to show that a non-regular language exists.

Regular Expressions From Scratch, Part Six: The Insanely Clever Bit

Eric Lippert — Mon, 05 Dec 2005 18:10:00 GMT

Let's start with an easy one today, because things are about to get a little tricky.

Definition 10: a pair is a finite sequence with exactly two members.

Definition 11: a function is a set of pairs, where the second member of each pair is the value associated with the first member by the function. If F is a function { (a1, b1), (a2, b2) } then we might write F(a1) = b1, F(a2) = b2. (Again, we will figure out from context whether parentheses mean function mapping, grouping or a finite sequence.)

Definition 12: Take any alphabet S. The regular expression language generating function for S is a function where the first member of the pair is a string in the regular expression language of S, and the second member is a language in S.

Let's call the function L and the regular expression language R. The pairs in this set go like this.

The string Ø is paired with the empty language. Or, in our new notation, L(Ø) = {}
The strings in R that are just single members of S are paired with the languages that are just single members of S. For example, if S = {a, b} then L(a) = {a}, L(b) = {b},
If x and w are in R then L((xw)) = L(x)L(w)
Similarly, L((x∪w)) = L(x) ∪ L(w)
Similarly, L(x*) = L(x)*

OK, let's try it out. What's L(((ab)∪a*))?

L(((ab)∪a*))
= L((ab)) ∪ L(a*)
= L(a)L(b) ∪ L(a*)
= {a}{b} ∪ L(a)*
= {a}{b} ∪ {a}*
= ((ab) ∪ a*)
= {ab, e, a, aa, aaa, aaaa, …}

Hey, wouldja look at that: L(((ab)∪a*)) = ((ab) ∪ a*)

What an amazing coincidence!

Obviously this is no coincidence at all. We have just defined a mapping between regular expressions and the languages which consist of all strings which match those regular expressions, and the mapping is basically "turn the regular expression string into an expression via the obvious substitutions."

Since we have this very strong mapping, I am probably going to become very sloppy about making a distinction between L(((ab)∪a*)) and ((ab ∪ a*). If I say "the language ((ab)∪a*) what I mean is L(((ab)∪a*)).

Note also that L(((ab)a)) = L((a(ba))) = { {aba}. In general, the concatenation and union operators do not require parens. Therefore, from now on I will also be very sloppy with my parens. Even though aba is not a "real" regular expression by the rules laid out earlier, I will assume that you can mentally transform that back into the well-formed (a(ba)) string.

Let's end off today's descent into the bowels of computer science with a rather obvious definition:

Definition 13: A language K over an alphabet S is called a regular language if and only if there exists a string r in S's regular expression language such that L(r) = K. That is, a language is called a "regular language" if and only if it can be described by a regular expression.

Is every language regular? One would suspect not, given the incredible variety of languages that I mentioned earlier. Actually proving that a nonregular language exists is both amusing and character-building, so we'll do that next.

Regular Expressions From Scratch, Part Five: The Regular Expression Language

Eric Lippert — Thu, 01 Dec 2005 18:00:00 GMT

Now things start to get really weird.

Definition 9: Take any alphabet S. The regular expression alphabet of S is S plus a bunch of extra symbols; it's S ∪ {(, ), *, ∪, Ø} (I assume that none of those symbols are already in S.)

I'm doing something that I said earlier that I would try not to do. I'm using symbols in an alphabet that I also use in expressions that talk about strings in that alphabet. (Of course, I also said that this would all fall apart when we got to regular expressions, and sure enough, it did. Foreshadowing: the sign of a quality blog.)

Again, keep a careful eye on when I'm using fixed-width blue, because those are the "meaningless" symbols, not expression notation.

Definition 10: Take any alphabet S. The regular expression language R of an alphabet S is a language formed from strings of the regular expression language of S, and is defined as follows:

Ø is in R.
Every member of S is in R
If u and w are strings in R then (uw) is in R.
Similarly, (u∪w) is in R.
Similarly, u* is in R.
Nothing else is in R

An example might help. Suppose that S = {a, b}. The regular expression alphabet of S is {a, b, (, ), *, ∪, Ø}. The regular expression language of S is R = {Ø, a, b, (ØØ), (Øa), (Øb), (aØ), (aa), (ab), (bØ), (ba), (bb), (Ø∪Ø), (Ø∪a), (Ø∪b), (a∪Ø), (a∪a), (a∪b), (b∪Ø), (b∪a), (b∪b), Ø*, a*, b*, …}

We've defined an alphabet, we've defined a language -- a language which looks suspiciously like the expressions we've been using to talk about languages. Next time we'll do something insanely clever to bridge the gap between strings in the regular expression language and the languages which these strings would define if they were interpreted as expressions.

Regular Expressions From Scratch, Part Four: The Kleene Closure of a Language

Eric Lippert — Mon, 28 Nov 2005 18:00:00 GMT

Languages are sets, so we can take any two languages (over the same alphabet) and take their union to form a new language. Just as a reminder:

Definition 7: The union of two sets L and K is the set with all the members found in either, and is written L ∪ K.

We're going to take the Kleene Star one level further and say that it applies to languages too.

Definition 8: The Kleene Closure of a language L is L* = { e } ∪ L ∪ LL ∪ LLL …

Got it? L* is the language consisting of set of strings, where those strings are any number of strings in L concatenated together.

I want to extend our notion of concatenation in one more way. When I write a string or string variable next to a language, I mean "concatenate the language onto the end of the language that consists of only this string". So when I say abLa I mean {ab}L{a} Again, you can infer the braces from the context.

Let's do an example of using * and string concatenation of languages, just so its clear.

S = {0, 1}
L = {011*}
1L* = {1, 101, 1011, 101101, 1011011, …}

Or, in English, this is the set of strings w in S* such that w:

starts with a single 1,
has no more than one 0 in a row, and
ends with 1

We need one more thing to make this notation sufficiently terse. You probably noticed that the Kleene Star above only applied to the thing immediately to the star's left, whether the language or the symbol. Suppose we've got four languages: H, J, K and L. We can form two new languages through concatenation: G = HJ and F = KL. And then we can form a seventh language, D = G*F*. But we can't write D = HJ*KL*, because that's a different language. What we mean is D = (HJ)*(KL)*.

Therefore I'm declaring that we can use parenthesis in the normal way we're all used to for order of operations. We will figure out from context whether parens mean "this is a finite sequence" or "this is a grouping of concatenations".

Remember, we're looking for a way to concisely characterize languages, and clearly we're onto something here. We now have union, grouping, concatenation and Kleene closure. In other words, we have enough tools to define regular expressions, which we'll do next time. Stay tuned!

Regular Expressions From Scratch, Part Three: Concatenation

Eric Lippert — Fri, 25 Nov 2005 18:00:00 GMT

You probably intuitively understood concatenation already, but let me define it anyway.

Definition 5: The concatenation of two strings w and u (over the same alphabet) makes a string consisting of the sequence of every element in w followed by every element in u. We write concatenations the same way as before: all run together. So if we have S = { a, b, c} and two strings in S* called w and u, where w=abc and u=cccaaabbb then then wu=abccccaaabbb.

Be very clear on this point: w and u are variables which represent string values. wu is an expression which represents concatenating the value of u onto the end of the value of w. Similarly, waaaa would be the result of concatenating aaaa ont w, so waaaa = abcaaaa.

Recall that I'm using the variable e to represent the empty string. You can concatenate the empty string onto any old string, and the result is unchanged. So we = w, for any string w.

We can concatenate any finite number of strings together with impunity. If v=ab then wuvevaeaea = abccccaaabbbababaaa.

We cannot concatenate an infinite number of non-empty strings together to form a string, because strings are by definition finite sequences.

Note that with this definition we can say that the set S* is the set of all finite concatenations of any members of S, plus the empty string.

That pretty much takes care of making strings out of other strings. We're going to need to make new languages from old languages as well. Can we meaningfully concatenate two languages together? Sure, why not?

Definition 6: Two languages L and K over the same alphabet may be concatenated together form a new language. Concatenated languages are notated as you'd expect: by writing one after the other. If u is in L, and v is in K, then uv is in LK.

If one of those languages is the empty language - the language with no strings, not even the empty string - then the concatenation is also the empty language.

For example, suppose

S = {a, b}
L = {e, a, aa, aaa, aaaa, …}
K = {e, b, bb, bbb, bbbb, …}

then

LK = {w in S* such that w has any number of as followed by any number of bs}

That's a little irksome. We need a more compact notation for "any number of as followed by any number of bs". Fortunately we already have such a notation -- we have the Kleene Closure over alphabets. We'll just create two one-character alphabets, star them to form languages, and concatenate the languages:

LK = {a}*{b}*

Let's make that notation a little more compact. From now on when I say

X = a*

what I mean is

X = {a}*

That is, I'm going to omit the set braces, because you can figure them out from context.

We can write that last example much more compactly:

S = {a, b}
L = a*
K = b*
LK = a*b*

Perhaps you can see that we are in fact heading towards regular expressions.

Next time: The Kleene Closure applies to languages too.

Regular Expressions From Scratch, Part Two: Some Examples of Languages

Eric Lippert — Tue, 22 Nov 2005 18:00:00 GMT

Let's look at some sample languages to get a sense of just how flexible languages can be.

For example, here are some languages over the alphabet S = {0, 1}

L₁ = all members of S*, period -- the language where every string is in the language

L₂ = all members of S* such that there are more 1's than 0's. So L₂ = {1, 11, 011, 101, 110, 111, …}

L₃ = all members of S* such that the string is a prime number of 1's. So L₃ = {11, 111, 11111, 1111111, 11111111111, …}

L₄ = all members of S* such that there are n 1's and no 0's in the string, and furthermore, n is such that there is a nontrivial solution to Fermat's Last Theorem: aⁿ + bⁿ = cⁿ in natural numbers.

That is a perfectly good description of a language. In fact, it's a description of the finite language L₄ = {1, 11}. We know that now, as Fermat's Last Theorem was proven. Twenty years ago though we would not have been able to enumerate all the members of this language with any confidence.

We can even come up with language descriptions that we cannot possibly know their members!

L₅ = all members of S* such that there are n 1's and no 0's in the string, and furthermore, n is the largest number of quarters I had in my pocket at any time on the 13th of December, 1991.

That language has only one member, but darned if I know what it is!

This silly example raises an important point: in general, given a description of a language, we cannot know how many members the language has, whether it is finite or infinite, or whether any particular string is a member of the language.

We will be seeing later that there are definitely languages which cannot in principle be determined. That is, there is no way whatsoever of computing the members of the language.

For some languages though, we can build and analyze devices which recognize them - that is, when given a string and a language, a device can tell you whether or not the string is a member of the language.

To do that, we're going to need a way to describe languages in a very precise manner - we need a "language definition language" of some sort. The relationship between language recognizers and various metalanguages for describing languages will be a fundamental focus of this series.

Next time: concatenation of strings and languages

Regular Expressions From Scratch, Part One: Defining Terms

Eric Lippert — Fri, 18 Nov 2005 21:00:00 GMT

Over the years that I've been writing this blog some of the most positive feedback I've received has been for those entries where I've explored fundamental concepts in computer science. I thought that I might take that to its logical extreme, and do a series on what exactly a "regular expression" is.

Most script developers are familiar with regular expressions - they're those dense, hard-to-read, patterns that you can use to write self-obfuscating code that does string matching. I'd like to start by throwing out everything that we know about practical, real-world regular expressions and start from set theory, of all things.

This is going to be a very long series I think. But it will build character!

Here's what I'm going to assume:

We have an intuitive understanding of sets. Sets are collections of stuff. They can be empty, finite, infinite, you name it. They can contain other sets. I am not going to start with the axioms of set theory, but trust me, we could justify our intuitive understanding of sets axiomatically if we wanted to.
We have an intuitive understanding of the set of natural numbers ℕ = {0, 1, 2, 3, …}
We can create arbitrary sets of symbols. For clarity, throughout this series I'll be displaying symbols in a blue fixed-width typeface. We could have a set of symbols S = {a, b, c, +, -, 0, 1, &} for instance.
We have an intuitive understanding of finite sequences. Remember, sets are by definition unordered, may be infinite, and contain no duplicates. A finite sequence is like a set but is finite, has an order, and may contain duplicates. A finite sequence may be empty. (Again, we could derive a suitable definition of a finite sequence from axiomatic set theory, but I won't.) To distinguish finite sequences from sets, I'll use curly braces for sets and round parentheses for finite sequences, but not for long.

That's not starting with much. Let's see what we can do.

Definition 1: An alphabet is a finite set of symbols.

For example, we could have the alphabet of capital Roman letters R = {A, B, C, …, Z} or the alphabet of binary digits B = {0, 1}.

The empty alphabet is a pretty boring alphabet, but it is a legal alphabet.

Definition 2: a string is a finite sequence of symbols drawn from a given alphabet.

It is a pain in the rear to continually say "(b, b, c) is a string over alphabet {a, b, c}" so for the vast majority of cases, I'll be writing strings all run together like this: bbc and assuming that you can mentally treat this as a sequence.

I'll also be assuming that you can figure out the alphabet for a given string from context. If the alphabet is ever unclear then I'll call it out specifically.

I will sometimes use "variables" to refer to strings and languages. You'll be able to distinguish variables from symbols because symbols are always written in blue fixed-width font. I shall endeavour to also avoid saying something confusing such as "Suppose b = bbc" though as we'll see later, this will become unavoidable when we talk about regular expressions.

There's no good way to show the empty string, not without getting into a whole lot of issues with quotation marks. I'm hereby declaring that unless I say otherwise, the variable e represents the empty string in whatever alphabet we're presently talking about.

Note that the empty string is the only string that can be formed from the empty alphabet.

I'll almost always use the variable S to represent an alphabet and u, v and w to represent strings.

Definition 3: Consider an alphabet S. The Kleene Closure of S is the set of all strings that can be formed from that alphabet and will be written as S*.

For example, let S = {0, 1}. Then S* = {e, 0, 1, 00, 01, 10, 11, 000, …}. That is, every string of finite length that can be formed from only 0 and 1, including the empty string.

S* is of course an infinite set, though I hasten to emphasize that no member of S* is infinitely long, because strings are by definition finite sequences.

Here comes our most important definition today:

Definition 4: A language over an alphabet S is any subset of S*.

This is a rather broad definition of "language" -- any subset whatsoever of the set of all finite strings in any alphabet forms a language.

Next time we'll look at some examples of languages, both sensible and crazy.