Post on 16-Apr-2017
Reactive Data System in Practice
Or how to solve Fast Data Problem and reacting to the World faster
By Triều Nguyễn at http://trieu.xyz http://www.rfxlab.com (Reactive Data System Lab)
λ(x)
2008: Java Developer, develop Social Trading Network for a startup (Yopco)2011: joined FPT Online, software engineer, worked in FPT ID project and VnExpress Mobile (Backend)2012: backend engineer at Greengar (Scaling server and real-time services)12/2012 to 05/2015: back to FPT Online, lead engineer at eClick Ad Platform06/2015 to now: lead engineer at iTVad, a startup project at FPT for OTT and TV Advertising Solutions
about me
1. Just some concepts 2. Ideas in practice3. What is RFX framework ?4. Why is RFX ?5. How to solve problems with RFX ?
a. Counting pageview of websiteb. Counting unique user of websitec. Querying data more reactive and adaptive
6. Final ideas
Contents of this talk
1 - Just some concepts
Fast Data vs Big Data
What is Reactive Data System ?
is a software design approach that uses the principles of event driven architecture for the design and implementation of data intensive applications.
Event Driven ArchitectureEDA is a framework that promotes production, detection, consumption and reaction to the events. Event based applications will run on RDS framework.
Why is Reactive Data SystemWe want to ask bigger questions faster
2 - Ideas in practice
Problem (User Story in Practice)
● User does login with FB Connect at Ecommerce website
● User does a click on an item, then system should do the following tasks:1. Find best items to recommend 2. Find a best item to send email marketing3. If there are more than 100 users, that click on a item,
run a marketing campaign on Facebook automatically.
An example from E-Commerce
3 - What is RFX framework ?
● A framework for big fast data problems● A collection of Open Source Tools from Netty to Spark● The mission of RFX
1. Build data product quickly with design patterns2. React to critical events in real-time
What is RFX or Reactive Function X ?
RFX is inspired by the Actor Model Theory
Paper: http://vialab.science.uoit.ca/textvis2011/papers/textvis%202011-rohrdantz.pdf
RFX is designed for startup business
I design RFX because I have to solve following problems both at FPT and at RFXLab.com● Marketing Automation● Product UX personalization● Real-time Advertising● System monitoring (frontend and backend)● Email marketing● Product Analytics● Customer Analytics
→ So I build a full-stack framework (a collection of tools for both backend and frontend)
Philosophy of RFX
Sub modules in RFX framework
4 - Why is RFX framework ?
Why Rfx ?
● Ideas since 2012 (from Actor model theory)● R&D and Deployed in Production since 2013
● Open Source: Apache License, Version 2.0 ● Full Stack: from Frontend to Backend● Agile for Data Analytics● Lightweight Lambda Architecture
● Really fast and near-real-time processing● Tested with 1.000.000 events / second● Simple development model for data engineer
Roadmap to the future
5 - How to solve problems with RFX ?
ProblemsUser story in Digital Marketing: 1. Counting pageview of website2. Counting unique user of website3. Real-time marketing
Find rush hour in your system
Which device user reach our content
Solution Architecture
Tracking(RFX-track)
Event Queue (Apache Kafka)
Event Processor(RFX-stream)
RedisMongoDB Synch Data Jobs(RFX-job)
Report(RFX-report) Event Actor
(RFX-actor)Ad Server
An example topology
6 - Final ideas
Summary
● Fast Data is cool● Fast data: The next step after big data● RFX is designed to solve common Fast Data
problems in Digital Marketing and Advertising
The end and thank youhttps://github.com/rfxlab
http://rfxlab.com
λ