Regular Expression Sub-Matching using Partial Derivativessuma0002/talks/ppdp12-part... · 2013. 8....

Regular Expression Sub-Matchingusing Partial Derivatives

Martin Sulzmann Kenny Zhuo Ming Lu

Hochschule Karlsruhe Nanyang Polytechnic

Regular Expression Sub-Matching using Partial Derivatives – p. 1/18

Regular Expressions - The Basics

Words: w ::= Σ∗

Regular expressions

(A + (BC))∗ denotes a regular language

L( (A + (BC))∗ ) = {ǫ, A, BC, ABC, ...}

Regular ExpressionSub-Matching

Matching

w matches r iff w ∈ L(r)

ABAAC matches (A + AB)(BAA + A)(AC + C)

L( (A + AB)(BAA + A)(AC + C) ) ={ABAAAC, ABAAC, AAAC, AAC, ABBAAAC, ABBAAC, ABAC}

Matching

Which sub-parts are matched?

Matching

(x1 : (A + AB))(x2 : (BAA + A))(x3 : (AC + C))

Matching

(x1 : (A + AB))(x2 : (BAA + A))(x3 : (AC + C))

Sub-matchings for ABAAC arex1 = AB

x2 = A

x3 = AC

Matching

(x1 : (A + AB))(x2 : (BAA + A))(x3 : (AC + C))

Sub-matchings for ABAAC arex1 = AB

x2 = A

x3 = AC

Now that the difference is clear:Matching = Sub-matching

Slow Regular Expression Matching

ExampleAn “A n-times”(x : A?n)(y : An) r? = ǫ + r

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

Fail ⇒ Backtrack

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

Fail ⇒ BacktrackAA ⊢ (x : A?A?)(y : AA)

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

...AA ⊢ (x : A?A?)(y : AA)

Consider n = 2

AA ⊢ (x : A?A?)(y : AA)

...AA ⊢ (x : A?A?)(y : AA)

Success but exponential complexity

Fast Regular Expression Matching

For brevity, we ignore sub-matching locations

Convert A?A?AA to NFA

�� A//�� A

//�� A//�� A

//��

�� A//�� A

//�� A//�� A

//��

Simultaneous search for match AA

�� A//• A

//• A//• A

//��

�� A//�� A

//• A//• A

//��

�� A//�� A

//• A//• A

//��

No backtracking, linear searchTo be tracked states linear in the size of regularexpressionLinear complexity!So far Thompson and Glushkov NFA constructionSee Russ Cox, Alain Frisch et. al., Ville Laurikari,...

Our Contributions

Matching automata construction based onBrzozowski’s Derivatives (DFA)Antimirov’s Partial Derivatives (NFA)

Fast and elegant algorithms forPOSIX matchinggreedy left-most matching

Implementation in Haskell supporting real-world regularexpressions

Brzozowski’s Derivatives

r\l “take away the leading l”

r\l derivative of r w.r.t. l

L(r\l) = {w | lw ∈ L(r)}

Compute r\l by induction, e.g.A\A = ǫ, B\A = φ, r∗\l = (r\l)r∗

r1r2\l = (r1\l)r2 + r2\l if ǫ ∈ L(r1)

Matching derivation:

l→ r2 iff r2 = r1\l

w = l1...ln check if rl1→ ...

ln→ r′ where ǫ ∈ L(r′)

Matching with Derivatives

Example: Match AB against (x : A + y : AB + z : B)∗

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

= (ǫ + B) (A + AB + B)∗

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

= (ǫ + B)1 (A + AB + B)∗ {x1 : A, y1 : A, z1 : ǫ}

Record matchings for each iteration

Paper records matchings within pattern

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

= (ǫ + B)1 (A + AB + B)∗ {x1 : A, y1 : A, z1 : ǫ}

B→ ((ǫ + B)1

︸︷︷︸

(A + AB + B)∗︸︷︷︸

(p1p2)\l = (p1\l, p2) + (empty(p1)p2\l) if p1 empty

Choice of matchings. Don’t drop p1,

keep p1 and make p1 “empty” (ǫ + B)1 ⇒ (ǫ + φ)1

so p1 won’t contribute further matchingsRegular Expression Sub-Matching using Partial Derivatives – p. 8/18

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

= (ǫ + B)1 (A + AB + B)∗ {x1 : A, y1 : A, z1 : ǫ}

B→ ((ǫ + B)1 (A + AB + B)∗)\B

= ((ǫ+B)1\B (A+AB +B)∗)+(ǫ+φ)1((A+AB +B)∗\B)

... = ǫ1(A + AB + B)∗ + (ǫ + φ)1ǫ2(A + AB + B)∗

(A + AB + B)∗ {x : ǫ, y : ǫ, z : ǫ}

A→ (A + AB + B)∗\A

= (A + AB + B)\A (A + AB + B)∗

= (A\A + AB\A + B\A) (A + AB + B)∗

= (ǫ + B)1 (A + AB + B)∗ {x1 : A, y1 : A, z1 : ǫ}

B→ ((ǫ + B)1 (A + AB + B)∗)\B

= ((ǫ+B)1\B (A+AB +B)∗)+(ǫ+φ)1((A+AB +B)∗\B)

... = ǫ1(A + AB + B)∗ + (ǫ + φ)1ǫ2(A + AB + B)∗

{y1 : AB} and {x1 : A, z2 : B}

POSIX and greedy left-most match

Derivatives Matching Summary

Computes all matchings ⇒ exponential complexity

Optimization:

Simplify, e.g. aggressively to the left

(1) r + r ⇒ r keep left r (2) φ r ⇒ φ (3) ǫ r ⇒ r

ǫ1(A + AB + B)∗︸︷︷︸

+ (ǫ + φ)1ǫ2(A + AB + B)∗︸︷︷︸

⇒∗ ǫ1(A + AB + B)∗ + ǫ2(A + AB + B)∗

⇒ ǫ1(A + AB + B)∗

⇒ (A + AB + B)∗

Yields POSIX match

Matching with Partial Derivatives

Derivatives represent states of a DFA·\· : r 7→ L 7→ r

(A + AB + B)∗\A = (ǫ + B) (A + AB + B)∗

On the fly DFA construction.

Partial derivatives represent states of an NFA·\p· : r 7→ L 7→ 2r

L(r\l) = L(r1 + ... + rn) where r\pl = {r1, ..., rn}

(A+AB+B)∗\pA = {ǫ(A+AB+B)∗, B(A+AB+B)∗}

Set of partial derivatives finite and linear in size ofregular expression.Build NFA match automata.

Matching with Partial Derivatives

(Sketch of) NFA match automata

76540123p2

// 76540123p1A

76540123p3 ...

(A + AB + B)∗︸︷︷︸

\pA = {ǫ(A + AB + B)∗︸︷︷︸

, B(A + AB + B)∗︸︷︷︸

Depth-first left-most traversal⇒ greedy left-most match

Not POSIX because structure is broken apart

See paper for details of construction

POSIX versus Greedy Left-Most Match

Derivatives for POSIX matching

POSIX = maximal match w.r.t structureAB matches (A + AB + B)∗

(A + AB + B)∗\A = (ǫ + B) (A + AB + B)∗

Partial derivatives for greedy left-most matchingGreedy left-most = maximal match ignoring anystructureAB matches (A + AB + B)∗

(A + AB + B)∗\pA ={ǫ(A + AB + B)∗,

B(A + AB + B)∗}

Implementation

Fine-tuned Haskell implementation of greedy left-mostusing partial derivative NFAs.

Real-world extensions:Group matchings, anchored match, ...

Competitive performance (see paper for details):C-based: RE2, PCREHaskell-based: Weighted, TDFA

Partial derivative NFA construction “smaller” comparedto Thompon and Glushkov NFA construction

Reference implementation of Thompon, Glushkovand Partial Derivative NFA construction

Conclusion

First application of derivatives and partial derivatives forregular expression sub-matching

Future work:Implementation in other languagesEfficient POSIX implementation

Tricky for NFA but see “backwards scanning” trickby Russ CoxExploiting laziness of the on-the fly derivativeconstruction

Error explanationWhy is there no match?

Errata - Simplifications

, left colomn: The derivation should be as followswhere we have underlined the corrected parts andexpressions involving φ have been alredy removed.

(x|ǫ : A∗, y|ǫ : A∗)A→ (x|A : A∗, y|ǫ : A∗) + (x|ǫ : ǫ, y|A : A∗)A→ ((x|AA : A∗, y|ǫ : A∗) + (x|A : ǫ, y|A : A∗))+

(x|ǫ : ǫ, y|AA : A∗))A→ ...

Simplifications at the pattern and regular expression level.

Errata - Derivative Matching

Figure 8 “Derivative Matching”:env(·) :: p → {Γ}

env((x|w : r)) =

{{(x, w)}} if ǫ ∈ L(r)

{} otherwise

env((x|w : p)) = {{(x, w)} ⊎ es|es ∈ env(p)}

env((p1, p2)) = {e1 ⊎ e2|e1 ∈ env(p1), e2 ∈ env(p2)}

env((p1 + p2)) = env(p1) ⊎ env(p2)

env(p∗) = env(p)

match(·, ·) :: p → w → {Γ}

match(p, w) = env(p\w)

There’s an issue:

In case env(p) yields {} but the pattern is empty, weshould actually return instead {{}}.

Errata - Derivative Matching (2)

To explain the issue, consider the initial pattern(x|ǫ : (y|ǫ : A)∗).Building the derivative w.r.t A yields

(x|A : (y|A : ǫ, (y|ǫ : A)∗))

Applying env() on the subpattern (y|ǫ : A)∗ yields {}because the underlying pattern (y|ǫ : A) is not empty.Clearly, (y|ǫ : A)∗ contains empty (zero iterations). Hence, inthis situation, we shouldn’t return {} (“no match”) but ratherreport {{}} (“empty match”).

Errata - Derivative Matching (3)

Here’s the fix:env(·) :: p → {Γ}

env((x|w : r)) =

{{(x, w)}} if ǫ ∈ L(r)

{} otherwise

env((x|w : p)) = envH (((x|w : p), {{(x, w)} ⊎ es|es ∈ env(p)}))

env((p1, p2)) = envH ((p1, p2), {e1 ⊎ e2|e1 ∈ env(p1), e2 ∈ env(p2)}))

env((p1 + p2)) = envH ((p1 + p2, env(p1) ⊎ env(p2)))

env(p∗) = envH ((p∗, env(p)))

envH ((p, e)) =

{{}} if ǫ ∈ L(p ↓) and e = {}

e otherwise

Regular Expression Sub-Matching using Partial Derivativessuma0002/talks/ppdp12-part... · 2013. 8....

Documents

Transcript of Regular Expression Sub-Matching using Partial Derivativessuma0002/talks/ppdp12-part... · 2013. 8....

12-Dec-15 Regular Expressions. About “Regular” Expressions In a theory course you should have learned about regular expressions Regular expressions describe.

Extremal 1-codes in distance-regular graphs of diameter 3 · Distance-regular graphs Codes in distance-regular graphs Triple intersection numbers Codes in distance-regular graphs

New Algorithms For Regular Expression Matchingphbi/files/talks/2006nafremS.pdf · New Algorithms For Regular Expression Matching ICALP 2006 Philip Bille IT University of Copenhagen.

Regular Languages: Regular Expressions and Finite Automatasupratik/courses/cs208/spr13/ashutosh... · A. Trivedi { 1 of 20 Regular Languages: Regular Expressions and Finite Automata

MODUL 2: Bahasa Regular dan Ekspresi Regular fileModul 2: Bahasa Regular dan Ekspresi Regular Slide 10 dari 21 Hirarki antar Operator Seperti halnya dalam ekspresi aritmetis, untuk

Partial Differential Equations - tat.physik.uni-tuebingen.dekokkotas/Teaching/Num_Methods... · Partial Di erential Equations Kostas Kokkotas December 15, 2014 Kostas Kokkotas Partial

String Matching

Impedance Matching

Transforms and Partial Differential Equations

Opposites matching cards · 2020. 7. 24. · Opposites word and picture matching cards © Copyright 2014, αυτή. Title: Opposites matching cards Author: HP_Administrator Created

Equivalence of NFAs and regular expressionsperugini.cps.udayton.edu/teaching/courses/Spring2020/cps...Regular Expression to NFA •Goal: to show that every regular expression defines

Matching Shapes with a Reference Point

IMPEDANCE MATCHING - Hong Kong Polytechnic …cktse.eie.polyu.edu.hk/eie403/impedancematching.pdf · Michael Tse: Impedance Matching 5 The Q factor approach to matching The Q factor

Partial Redundancy Elimination

Regular language and Regular expression

Resolution Lower Bounds for Perfect Matching Principles

Efficient algorithms for ( δ , γ , α )-matching

Faster Regular Expression Matchingphbi/files/talks/2009fremS.pdf• History tour of regular expression matching • Thompson’s algorithm • Myers’ algorithm • New algorithm

Regular Expressions - Cornell University€¦ · Regular Expressions CS 2800: Discrete Structures, Fall 2014 Sid Chaudhuri. Regular expressions (“regex”-es) are defned inductively

Regular expression with DFA