BeepBeep 3: A declarative event stream query engine (EDOC 2015)

25
A declarative event stream query engine

Transcript of BeepBeep 3: A declarative event stream query engine (EDOC 2015)

A declarative event streamquery engine

Booleans

B

Numbers

R

234

π

Strings

S

abc

Functions

X Y→

Sets

X2

A processor is a function that takes 0 or moreevent traces as input, and returns 0 or 1event trace as output

. . . . . .

Processors can be composed: the output ofa processor can be given as the input ofanother ("piping")

BeepBeep is an event stream query engine that provides...

A set of basic processors (independent fromany event type)

A core grammar and interpreter tocompose ("pipe") processors

Mechanisms to extend the grammarwith user-defined events, processors, etc.

abc. . . a . . .n cΨ

abc. . . b . . .n c

EVERY nTH OF (T)

TRIM n FROM (T)

WHERE2 1 5 0 2 0

C

C2 1 5 0

(T) WHERE condition

COMBINE (T) WITH f

Σ+2 1 5 0 2 3 8

f

f

8

2 1 5 0 2 1 5 0

f1 5 0 1 5 0

f5 0 5 0

++

+

a . . .b

a . . .b

FILE "filename"

SAVE (T) TO "filename"

WHEN @P IS A PROCESSOR:

THE SUM OF ( @P )

IS THE PROCESSOR

COMBINE (@P) WITH SUM.

Arbitrarysymbol

Grammar rule this symbolmust parse against

New grammar case

Grammar rule the caseis added toExpression the new casestands for

WHEN @P IS A PROCESSOR: THE COUNT OF ( @P ) IS THE PROCESSOR COMBINE (SELECT 1 FROM (@P)) WITH SUM.

WHEN @P IS A PROCESSOR: THE SUM OF ( @P ) IS THE PROCESSOR COMBINE (@P) WITH SUM.

WHEN @P IS A PROCESSOR: THE AVERAGE OF ( @P ) IS THE PROCESSOR SELECT (T.*) ÷ (U.*) FROM ( THE SUM OF (@P) AS T, THE COUNT OF (@P) AS U).

On every fifth trading day starting today,calculate the average closing price of MSFT for the five most recent trading days, and keepthe query standing for fifty trading days.

timestamp stockSymbol closingPrice

0 APPL 1039.30 MSFT 950.00 GOGL 433.31 MSFT 951.21 APPL 1038.3... ... ...

On every fifth trading day starting today,calculate the average closing price of MSFT for the five most recent trading days, and keepthe query standing for fifty trading days.

String line = br.readLine().trim();if (!line.isEmpty()) { String[] parts = line.split(","); if (parts[0].compareTo("ABC") != 0) { value_index++; sum += Double.parseDouble(parts[1]); if (value_index == 5) { double average = sum / 5; value_index = 0; sum = 0; return average;}}}

On every fifth trading day starting today,calculate the average closing price of MSFT for the five most recent trading days, and keepthe query standing for fifty trading days.

SELECT afd FROM ( SELECT S1.timestamp AS ts, AVG(S2.closingPrice) AS afd FROM (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S1, (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S2 WHERE (S2.timestamp - S1.timestamp) < 5 GROUP BY S1.timestamp) AS S3WHERE MOD(ts, 5) = 0;

On every fifth trading day starting today,calculate the average closing price of MSFT for the five most recent trading days, and keepthe query standing for fifty trading days.

EVERY 5TH OF ( APPLY (THE AVERAGE OF (*)) TO ( SELECT closingPrice FROM stocks) WHERE (stockSymbol) = ("MSFT")))) ON A WINDOW OF 5).

Calculate how many times the closing priceof MSFT is greater than 20 and the nextday, its closing price is less than 10.

Calculate how many times the closing priceof MSFT is greater than 20 and the nextday, its closing price is less than 10.

SELECT COUNT(*) FROM (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S1, (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S2 WHERE (S2.timestamp - S1.timestamp) = 1 AND S1.closingPrice > 20 AND S2.closingPrice < 10;

Calculate how many times the closing priceof MSFT is greater than 20 and the nextday, its closing price is less than 10.

WHEN @P IS A PROCESSOR: MY PATTERN IN ( @P ) IS THE PROCESSOR (SELECT (closingPrice) LESS THAN (20) FROM (@P)) AND (NEXT (SELECT (closingPrice) GREATER THAN (10) FROM (@P))).

THE COUNT OF (( MY PATTERN IN ( (SELECT closingPrice FROM stocks) WHERE (stockSymbol) = ("MSFT"))) WHERE (*) = (true)).

LinearTemporalLogic!

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

public Queue<Vector<Object>> compute(Vector<Object> inputs) {

}

public void build(Stack<Object> s) {

}}

. . . Create output events from input . . .

. . . Instantiate processor from parse stack . . .

<processor> := . . .<number> := . . .<string> := . . .

Add new rules to any symbol fromthe basic grammar

<processor> := <my_processor> ;

<my_processor> := INCREMENT ( <processor> ) BY <number> ;

Symbols already defined in basic grammar

Adds a new case to an existing rule

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

private int increment;

public Queue<Vector<Object>> compute(Vector<Object> inputs) { Queue<Vector<Object>> out = new Queue<Vector<Object>>(); Vector<Object> v = new Vector<Object>(); Integer i = (Integer) inputs.firstElement() + increment; v.addElement(i); out.put(v); return out; }

. . .

. . .

public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop();

increment = n.intValue();

Connector.connect(p, this);

s.push(this); }}

Read contents ofparse stack

<number>BY(<processor>)INCREMENT

Set processor's statePipe it to its inputPut on parse stack

Some pre-packaged grammar extensions:

Manipulation of name-value tuples

Set theory

Formatted input (CSV, XML, JSON)

Graphing (histograms, scatterplots, ...)

Basic signal processing (smoothing, peak detection, ...)

Create your own!

https://bitbucket.org/sylvainhalle/beepbeep-3

import ca.uqac.lif.cep.*;import ca.uqac.lif.cep.eml.tuples.*;

public class MyExample {

public static void main(String[] args) { Interpreter my_int = new Interpreter();

my_int.extendGrammar(TupleGrammar.class);

Pullable p = my_int.executeQuery( "\"HELLO WORLD\"");

for (int i = 0; i < 10; i++) { EmlString s = (EmlString) p.pull(); System.out.println(s); } }}

Create query interpreterLoad a grammar extension

Execute a query

Pull an output event