Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for...

19
Live Traffic Ali Ghandi December 2020 alighandij.github.io www.linkedin.com/in/alighandi/

Transcript of Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for...

Page 1: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Live TrafficAli Ghandi

December 2020

alighandij.github.io

www.linkedin.com/in/alighandi/

Page 2: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

2

Page 3: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Balad

2

• Start: December 2016

• Release: May 2018

Page 4: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Balad

2

• Team members: 110

• Users: +5 million

• POI’s: +700,000

Page 5: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

What is Live?

3

xt

Page 6: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

4

What is Live?

1 minutes window

Δx/Δt

GPS location error

Page 7: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

5

What is Live?

9:039:049:109:01

Car #1Car #2Car #3Car #4

edgeV

ttt

e v

e

Page 8: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

6

Change point

Page 9: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

BOCD

7

( )mr n

( ), (0)t tr

Bayesian online change point detection

Page 10: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

8

BOCD

1

1/ , 0

( | ) 1 1/ , 1

0 .

t t

m

P r m r n m n

o w

Hazard:

let’s assume that the

‘natural chance’ of a

change point is 1/λ

Page 11: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

9

BOCD

1

1

1

1 1:

1 1: 1 1 1: 1

1 1 1: 1 1: 1

( , , )

( , | , ) ( , )

( | ) ( | , ) ( , )

t

t

t

t t tr

t t t t t tr

t t t t t t tr

P r r x

P r x r x P r x

P r r P x r x P r x

1:( , )t tP r x

Page 12: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

9

BOCD

1

1

1

1 1:

1 1: 1 1 1: 1

1 1 1: 1 1: 1

( , , )

( , | , ) ( , )

( | ) ( | , ) ( , )

t

t

t

t t tr

t t t t t tr

t t t t t t tr

P r r x

P r x r x P r x

P r r P x r x P r x

1:( , )t tP r x

Page 13: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

9

BOCD

1

1

1

1 1:

1 1: 1 1 1: 1

1 1 1: 1 1: 1

( , , )

( , | , ) ( , )

( | ) ( | , ) ( , )

t

t

t

t t tr

t t t t t tr

t t t t t t tr

P r r x

P r x r x P r x

P r r P x r x P r x

1:( , )t tP r x

Page 14: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

9

BOCD

1

1

1

1 1:

1 1: 1 1 1: 1

1 1 1: 1 1: 1

( , , )

( , | , ) ( , )

( | ) ( | , ) ( , )

t

t

t

t t tr

t t t t t tr

t t t t t t tr

P r r x

P r x r x P r x

P r r P x r x P r x

1:( , )t tP r x

Page 15: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

10

BOCD

1:1:

1:

( , )( ) ( | )

( )

t tt t t

t

P r xr n P r x

p x

Bayesian online change point detection

Page 16: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Stateful Stream Processing

11

Online algorithms based on state• Can use Flink for production

• Last 10 minutes of events are state

• Parameters of change point detection are state

1 minutes window 1 minutes window 1 minutes window

states states states

Parameters update

base on online data.

System recover itself

Page 17: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Future Work

12

• Graph neural networks (GNNs)

• Road ranking

Page 18: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

Future Work

13

NLP-Topic modeling

Each road can be a document and

Events can be it’s words.

Page 19: Live TrafficStateful Stream Processing 11 Online algorithms based on state • Can use Flink for production • Last 10 minutes of events are state • Parameters of change point detection

▪ Problems and Challenges (Cont.)Questions

14