Xquery Streaming à la Carte &

ΠανεπιστήμιοΚρήτης ΠανεπιστήμιοΚρήτης ΣχολήΘετικώνΕπιστημών ΣχολήΘετικώνΕπιστημών

ΤμήμαΕπιστήμης Υπολογιστών ΤμήμαΕπιστήμης Υπολογιστών

-56ΗΥ -56ΗΥ 11: Διαχείριση Δεδομένων: Διαχείριση Δεδομένων στονΠαγκόσμιο Ιστό στονΠαγκόσμιο Ιστό

Xquery Streaming à la CarteXquery Streaming à la Carte

&&

Combined Static and Dynamic Analysis for Combined Static and Dynamic Analysis for Effective Buffer Minimization in Streaming Effective Buffer Minimization in Streaming

Xquery EvaluationXquery Evaluation

XQuery Streaming à la CarteXQuery Streaming à la CarteIntroductionIntroduction

Existing XML query evaluation techniquesExisting XML query evaluation techniques Algebraic optimization with algorithms for persistent Algebraic optimization with algorithms for persistent

datadata Streaming algorithms for transient dataStreaming algorithms for transient data

New IdeaNew Idea Physical algebra for XQuery Physical algebra for XQuery

À la carte use of streaming algorithms & optimization techniquesÀ la carte use of streaming algorithms & optimization techniques

HY-561 Paper Presentation Konstantinos Galanakis Slide 2

IntroductionIntroduction

XQuery Streaming à la CarteXQuery Streaming à la CarteIntroductionIntroduction


Diverse Data SourcesDiverse Data Sources

Join of local repository and streaming sourceJoin of local repository and streaming source

ListList Immutable ordered sequence of homogenous valuesImmutable ordered sequence of homogenous values

CursorCursor Mutable ordered sequence of homogenous valuesMutable ordered sequence of homogenous values DestructiveDestructive C(C( )α)α : Cursor containing values of type : Cursor containing values of type αα OperatorsOperators

fromListfromList nextnext peekpeek


XQuery Streaming à la CarteXQuery Streaming à la CartePreliminariesPreliminaries

PreliminariesPreliminaries


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Data ModelPhysical Data Model

Physical Data Model 1/2Physical Data Model 1/2 Physical ValuePhysical Value

Physical XML value, Physical XML value, ((Xml)Xml) Cursor of XML tokens, Cursor of XML tokens, C(Tok)C(Tok) List of tree values, List of tree values, LL((Tree)Tree)

Physical table, Physical table, (Table)(Table) Cursor of tuplesCursor of tuples, , CC( )τ( )τ

• Physical Tuple, Physical Tuple, ττ: record of fields containing physical XML values: record of fields containing physical XML values List of tuplesList of tuples, , LL( )τ( )τ

XML Token, (Tok): XML Token, (Tok): Parsing event produced by SAX parserParsing event produced by SAX parser


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Data ModelPhysical Data Model

Physical Data Model 2/2Physical Data Model 2/2 XML Token, (Tok) : XML Token, (Tok) : Parsing event produced by SAX Parsing event produced by SAX

parserparser startElemstartElem endElemendElem texttext atomicatomic holehole


Physical Representation & ConversionPhysical Representation & Conversion

XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Representation & ConversionPhysical Representation & Conversion


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Overview & OperatorsPhysical Algebra – Overview & Operators

Physical algebra for logical Algebra proposed in Physical algebra for logical Algebra proposed in C. Re, J. Simeon and M. Fernandez , C. Re, J. Simeon and M. Fernandez , ““A complete and efficient algebraic compiler for XQuery”, In ICDE 2006A complete and efficient algebraic compiler for XQuery”, In ICDE 2006


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra - ConstructorsPhysical Algebra - Constructors

ConstructorsConstructors

TreeProjectTreeProject Projection of path expressions on a tree.Projection of path expressions on a tree. Injected after Injected after ParseParse to reduce the plan input size. to reduce the plan input size.

TreeJoinTreeJoin Returns a node sequence in document order with no Returns a node sequence in document order with no

duplicateduplicate Strictly-forwardStrictly-forward path expressions path expressions

self self axesaxes child child axesaxes descendant descendant axesaxes descendant-or-self descendant-or-self axesaxes attribute attribute axesaxes


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Navigation OperatorsPhysical Algebra – Navigation Operators

Navigation Operators 1/3Navigation Operators 1/3


Navigation Operators 2/3Navigation Operators 2/3

XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Navigation OperatorsPhysical Algebra – Navigation Operators

desc-or-self::sectiondesc-or-self::section child::titlechild::title

Compiled in physical planCompiled in physical planApplying the plan toApplying the plan toan input documentan input document

Polymorphic Operators except MapFromItemPolymorphic Operators except MapFromItem MapFromItem MapFromItem

Input Input →→ Item sequence Item sequence Output Output → → Tuple for each itemTuple for each item 2 implementations2 implementations

For Lists of trees and for token cursorsFor Lists of trees and for token cursors

Relies to Relies to map map and and splitsplit


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Tuple OperatorsPhysical Algebra – Tuple Operators

Tuple operators 1/2Tuple operators 1/2


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Tuple OperatorsPhysical Algebra – Tuple Operators

Tuple operators 2/2Tuple operators 2/2

Mapping from a logical plan (Op) to a physical plan Mapping from a logical plan (Op) to a physical plan (POp).(POp). CS(Op) CS(Op) → → POpPOp

Physical plan correctnessPhysical plan correctness Stream safetyStream safety

Sufficient to ensure correctnessSufficient to ensure correctness


XQuery Streaming à la CarteXQuery Streaming à la CartePhysical Algebra – Code SelectionPhysical Algebra – Code Selection

Code selection 1/4Code selection 1/4



Code selection 2/4Code selection 2/4Conditions for Stream SafetyConditions for Stream Safety Navigational access on the XML Navigational access on the XML

values returned by Op is strictly values returned by Op is strictly forwardforward

Tuples returned by Op consumed Tuples returned by Op consumed in the order of creationin the order of creation

Tuple fields returned by Op Tuple fields returned by Op accessed at most onceaccessed at most once

OpOp

Code selection heuristic based assumptionsCode selection heuristic based assumptions conversion between physical representations is expensiveconversion between physical representations is expensive streaming operators are more efficient on streamed sourcesstreaming operators are more efficient on streamed sources copying whole sub-trees is expensive and should be avoidedcopying whole sub-trees is expensive and should be avoided

Following rules are applied to each subplan Op of a Following rules are applied to each subplan Op of a whole plan Opwhole plan Op00, bottom-up, bottom-up

1.1. If a)inputs of Op are streamed, b)streaming operators POp If a)inputs of Op are streamed, b)streaming operators POp exists for OP c)Op is stream-safe, then exists for OP c)Op is stream-safe, then CS(CS(Op) selects OpOp) selects Op

2.2. If Op is a constructor operator, If Op is a constructor operator, CS(CS(Op) uses a streaming Op) uses a streaming operator.operator.



Code selection 3/4Code selection 3/4

Experiments on synthetic dataExperiments on synthetic data verify linear scalability of streaming operators w.r.t. verify linear scalability of streaming operators w.r.t.

query and document sizesquery and document sizes run over MemBeR documents in XCheck frameworkrun over MemBeR documents in XCheck framework

XMark benchmarksXMark benchmarks Q2, 6, 15 are Q2, 6, 15 are fully streamablefully streamable Q1, 4, 5, 7, 14, 16 – 19 are partially streamableQ1, 4, 5, 7, 14, 16 – 19 are partially streamable Self-join queries Q8 – 12 /Q20 not streamableSelf-join queries Q8 – 12 /Q20 not streamable


XQuery Streaming à la CarteXQuery Streaming à la CarteExperimental EvaluationExperimental Evaluation

Experimental Evaluation 1/2Experimental Evaluation 1/2


Experimental Evaluation 1/2Experimental Evaluation 1/2

XQuery Streaming à la CarteXQuery Streaming à la CarteExperimental EvaluationExperimental Evaluation

Buffer manager of a streaming Xquery willBuffer manager of a streaming Xquery will1.1. Only relevant query evaluation data put into bufferOnly relevant query evaluation data put into buffer

2.2. Avoid keeping data buffered longer than necessaryAvoid keeping data buffered longer than necessary

3.3. Avoid keeping multiple copies of the data in buffersAvoid keeping multiple copies of the data in buffers

Claim: Combination of static analysis and dynamic Claim: Combination of static analysis and dynamic buffer minimization techniques neededbuffer minimization techniques needed


Combined Static and Dynamic Analysis for Effective Combined Static and Dynamic Analysis for Effective Buffer Minimization in Streaming Xquery EvaluationBuffer Minimization in Streaming Xquery Evaluation


GeneralGeneral




Previous Work 1/2Previous Work 1/2

<q> {<q> {for $b in /bib/bookfor $b in /bib/bookwhere ($b/author=where ($b/author=““A. TuringA. Turing”” and and fn:exists($b/price))fn:exists($b/price))return $b/titlereturn $b/title} </q>} </q>

{ { /bib/book,/bib/book, /bib/book/author//bib/book/author/

dos::node(),dos::node(), /bib/book/price,/bib/book/price, /bib/book/title//bib/book/title/

dos::node()dos::node()}}

XQueryXQuery Projection PathsProjection Paths

bibbib

bookbook

authorauthor priceprice titletitle

bookbook

authorauthor priceprice titletitle

…… …… …… ……

articlearticle

…… …… ……isbnisbn isbnisbn

…… …… …… ……

XML DocumentXML Document


Previous Work 2/2Previous Work 2/2



<q> {<q> { for $x1 in //book returnfor $x1 in //book return for $x2 in //* returnfor $x2 in //* return for $x3 in //article returnfor $x3 in //article return <node/><node/>} </q>} </q>

XQueryXQuery

Two approaches:Two approaches:

(1) Single DOM-tree(1) Single DOM-tree

(2) Buffers for variables(2) Buffers for variables

Buffer management technique for Xquery EnginesBuffer management technique for Xquery Engines Both static and dynamic analysis is exploitedBoth static and dynamic analysis is exploited Basic ideaBasic idea

Which data objects won’t be accessed in the futureWhich data objects won’t be accessed in the future A.G.C. StrategyA.G.C. Strategy

Reference countingReference counting New approachNew approach

Roles assigned to nodesRoles assigned to nodes Multiple roles per nodeMultiple roles per node Multiple nodes per roleMultiple nodes per role

signOffsignOff-statement-statementHY-561 Paper Presentation Konstantinos Galanakis Slide 22


Active Garbage CollectionActive Garbage Collection

Active Garbage CollectionActive Garbage Collection


Role removalRole removal(A.G.C.)(A.G.C.)

VariableVariablebindingsbindings


Main IdeaMain Idea

XQ is an XQuery FragmentXQ is an XQuery Fragment Nested for-expressionsNested for-expressions ConditionsConditions JoinsJoins

Covers syntactically simple fragments of XqueryCovers syntactically simple fragments of Xquery Assume that syntactically richer fragment could be Assume that syntactically richer fragment could be

evaluatedevaluated Remove let-expressions Remove let-expressions → Query normalization→ Query normalization Rewrite where-conditions to if-then-else expressionsRewrite where-conditions to if-then-else expressions Replace for-loop with nested single step for-loopsReplace for-loop with nested single step for-loopsHY-561 Paper Presentation Konstantinos Galanakis Slide 24


Query LanguageQuery Language



<r> {<r> { for $b in /bibfor $b in /bib where (fn:exists($b/book))where (fn:exists($b/book)) return return <books>{ $b/book }</books><books>{ $b/book }</books>} </r>} </r>

<r> {<r> { for $b in /bib returnfor $b in /bib return (( if (fn:exists($b/book))if (fn:exists($b/book)) then then <books><books> else (), else (),

if (fn:exists($b/book))if (fn:exists($b/book)) then then $b/book$b/book else (), else (),

if (fn:exists($b/book))if (fn:exists($b/book)) then then </books></books> else () else () ))} </r>} </r>

where-expressions where-expressions → → if-statementif-statement




<r> {<r> { for $b in /bibfor $b in /bib where (fn:exists($b/book))where (fn:exists($b/book)) return return <books>{ $b/book }</books><books>{ $b/book }</books>} </r>} </r>

<r> {<r> { for $b in /bib returnfor $b in /bib return (( if (fn:exists($b/book))if (fn:exists($b/book)) then then <books><books> else (), else (),

if (fn:exists($b/book))if (fn:exists($b/book)) then then $b/book$b/book else (), else (),

if (fn:exists($b/book))if (fn:exists($b/book)) then then </books></books> else () else () ))} </r>} </r>

i.i.where-expressions where-expressions → → if-statementif-statementii.ii.pushing if-statementspushing if-statements






<r> {<r> { for $bib in /bib returnfor $bib in /bib return (for $x in $bib/* return(for $x in $bib/* return if (not(fn:exists($x/price)))if (not(fn:exists($x/price))) then $x else (),then $x else (), for $b in $bib/book for $b in $bib/book return $b/title )return $b/title )} </r>} </r>

/bib/bib

/*/* /book/book

//

/title/dos::node()/title/dos::node()/price[1]/price[1] dos::node()dos::node()

Role extractionRole extraction




Role assignmentRole assignment

bibibb

bookbook

authorauthortitltitlee

{ r{ r22 } }

{ r{ r3,3, r r5,5, r r6 6 }}

{ r{ r55 } }{ r{ r5,5, rr7 7 }}

rr1 1 / /

rr22 /bib /bib

rr33 /bib/* /bib/*

rr44 /bib/*/price[1] /bib/*/price[1]

rr55 /bib/*/dos::node() /bib/*/dos::node()

rr66 /bib/book /bib/book

rr77

/bib/book/title/dos::node()/bib/book/title/dos::node()

RolesRoles XML documentXML document

Roles assigned to document node when projected into bufferRoles assigned to document node when projected into buffer On-the-fly role assignmentOn-the-fly role assignment Nodes without roles and role-carrying ancestors need not to Nodes without roles and role-carrying ancestors need not to

be bufferedbe buffered




Role update insertingRole update inserting<r> {<r> { for $bib in /bib returnfor $bib in /bib return (for $x in $bib/* return(for $x in $bib/* return if (not(fn:exists($x/price)))if (not(fn:exists($x/price))) then $x else (),then $x else (), for $b in $bib/book for $b in $bib/book return $b/title)return $b/title)} </r>} </r>

<r> {<r> { for $bib in /bib returnfor $bib in /bib return (( for $x in $bib/* returnfor $x in $bib/* return (( if (not(exists($x/price)))if (not(exists($x/price))) then $xthen $x else (),else (), signOff($x,r3),signOff($x,r3), signOff($x/price[1],r4),signOff($x/price[1],r4), signOff($x/dos::node(),r5)signOff($x/dos::node(),r5) ),), for $b in $bib/book returnfor $b in $bib/book return (( $b/title,$b/title, signOff($b,r6),signOff($b,r6), signOff($b/title/dos::node(),r7)signOff($b/title/dos::node(),r7))))) ),), signOff($bib,r2)signOff($bib,r2) ) ) } </r>} </r>

rr11 / /

rr22 /bib /bib $bib $bib

rr33 /bib/* /bib/* $x $x

rr44 /bib/*/price[1] /bib/*/price[1]

$x/price$x/pricerr55 /bib/*/dos::node() /bib/*/dos::node() $x $x

rr66 /bib/book /bib/book $b $b

rr77 /bib/book/title/dos::node() /bib/book/title/dos::node()

$b/title$b/title




Active Garbage CollectionActive Garbage Collection<r> {<r> { for $bib in /bib returnfor $bib in /bib return (( for $x in $bib/* returnfor $x in $bib/* return (( if (not(exists($x/price)))if (not(exists($x/price))) then $xthen $x else (),else (), signOff($x,r3),signOff($x,r3), signOff($x/price[1],r4),signOff($x/price[1],r4), signOff($x/dos::node(),r5)signOff($x/dos::node(),r5) ),), for $b in $bib/book returnfor $b in $bib/book return (( $b/title,$b/title, signOff($b,r6),signOff($b,r6), signOff($b/title/dos::node(),r7)))signOff($b/title/dos::node(),r7))) ),), signOff($bib,r2)signOff($bib,r2) ) ) } </r>} </r>

Buffer:Buffer:

Output stream:Output stream:

Input stream:Input stream:

<bib<bib>>

<book<book>>

<title/><title/><author/><author/></book></book>……

<r><r><book><book><title/><title/><author/><author/></book></book>

bibibb

bookbook

titletitle

{r{r22}}

{r{r3 3 , r, r5 5 , r, r66}}

{r{r55 , r , r77}} authorauthor{r{r55}}

{r{r55 , r , r66}}

{r{r77}} {}{}

{r{r66}}

Aggregated rolesAggregated roles Remove redundant rolesRemove redundant roles


>>rr} <} < for $bib in /bibfor $bib in /bib return $bib/bookreturn $bib/book

/> {/> {rr<<

>>rr} <} < for $bib in /bibfor $bib in /bib) ) return $bib/bookreturn $bib/book,, signOff($bib,rsignOff($bib,r11)),,

signOff($bib/book/dos::node(),rsignOff($bib/book/dos::node(),r22))((/> {/> {rr<<

Path steps Path steps → → for-expressionsfor-expressions


OptimizationsOptimizations

<r> {<r> { for $bib in /bib returnfor $bib in /bib return (for $_1 in $bib/book(for $_1 in $bib/book (return $_1/book,(return $_1/book, signOff($_1/book/dos::node(),rsignOff($_1/book/dos::node(),r22)),)), signOff($bib,rsignOff($bib,r11))))} </r>} </r>

<r> {<r> { for $bib in /bib returnfor $bib in /bib return for $_1 in $bib/bookfor $_1 in $bib/book return $_1/bookreturn $_1/book} </r>} </r>


Time and memory consumptionTime and memory consumption Queries and documents from the XMark BenchmarkQueries and documents from the XMark Benchmark Queries and documents modified to match the supported fragmentQueries and documents modified to match the supported fragment 3GHz CPU Intel Pentium IV with 2GB RAM 3GHz CPU Intel Pentium IV with 2GB RAM SuSe Linux 10.0, J2RE v1.4.2 for Java-based systemsSuSe Linux 10.0, J2RE v1.4.2 for Java-based systems Time limit: 1 hourTime limit: 1 hour Benchmarks against the following systemsBenchmarks against the following systems

FluXFluXJava in-memory engine for streaming XQuery evaluation.Java in-memory engine for streaming XQuery evaluation.

MonetDB v4.12.0/XQuery v0.12.0MonetDB v4.12.0/XQuery v0.12.0A A secondary storagesecondary storage engine written in C++. Loading of the document is engine written in C++. Loading of the document is included in time measurements.included in time measurements.

QizX/open v1.1QizX/open v1.1Free in-memory XQuery engine written in Java.Free in-memory XQuery engine written in Java.

Saxon v8.7.1Saxon v8.7.1Free in-memory XQuery engine written in Java.Free in-memory XQuery engine written in Java.


BenchmarkingBenchmarking

Benchmark Results 1/5Benchmark Results 1/5




Benchmark Results 2/5Benchmark Results 2/5<query1> <query1> { { for $s in /site returnfor $s in /site return for $p in $s/people return for $p in $s/people return for $pe in $pe/person return for $pe in $pe/person return if ($pe/person_id="person0") if ($pe/person_id="person0") then <result>{ $pe/name }</result> then <result>{ $pe/name }</result> else () else ()}}</query1></query1>

XMark XMark Q1:Q1:

0

2

4

6

8

10

12

14

16

10MB 50MB 100MB 200MB

GCX

FluxQuery

MonetDB

Saxon

Qizx/open

Running time Running time (s)(s)




Benchmark Results 3/5Benchmark Results 3/5

0

100

200

300

400

500

600

700

800

900

1000

10MB 50MB 100MB 200MB

GCX

FluxQuery

MonetDB

Saxon

Qizx/open

Memory Consumption Memory Consumption (MB)(MB)

<query1> <query1> { { for $s in /site returnfor $s in /site return for $p in $s/people return for $p in $s/people return for $pe in $pe/person return for $pe in $pe/person return if ($pe/person_id="person0") if ($pe/person_id="person0") then <result>{ $pe/name }</result> then <result>{ $pe/name }</result> else () else ()}}</query1></query1>

XMark XMark Q1:Q1:




Benchmark Results 4/5Benchmark Results 4/5<query8> { <query8> { for $root in (/) returnfor $root in (/) return for $site in $root/site return for $site in $root/site return for $people in $site/people return for $people in $site/people return for for $person$person in $people/person return in $people/person return <item> { <item> { ( <person>{ $person/name }</person>,( <person>{ $person/name }</person>, <items_bought> { <items_bought> { for $site2 in $root/site return for $site2 in $root/site return for $cas in $site2/closed_auctions return for $cas in $site2/closed_auctions return for $ca in $cas/closed_auction return for $ca in $cas/closed_auction return for for $buyer$buyer in $ca/buyer return in $ca/buyer return if ( if ($buyer/buyer_person=$person/person_id$buyer/buyer_person=$person/person_id)) then <result> { $ca } </result> then <result> { $ca } </result> else () } </items_bought> ) else () } </items_bought> ) } </item> } </item> } </query8>} </query8>

XMark XMark Q8:Q8:




Benchmark Results 5/5Benchmark Results 5/5XMark Q8XMark Q8

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

10MB 50MB 100MB 200MB

Failure for 100MB: MonetDB – Failure for 200MB: GCX, FluxQuery, Failure for 100MB: MonetDB – Failure for 200MB: GCX, FluxQuery, MonetDBMonetDB

0

500

1000

1500

2000

2500

3000

3500

10MB 50MB 100MB 200MB

GCX

FluxQuery

MonetDB

Saxon

Qizx/open

Xquery Streaming à la Carte &

Documents

Transcript of Xquery Streaming à la Carte &