Introduction to RFX for Backend Developer

Reactive Function (X) the Open Source Framework for solving Fast Data Problem

and reacting to the World with Deep Learning

By Triều Nguyễn, the creator of RFX http://mc2ads.com (Reactive Big Data Lab)

λ(x)

2008: Java Developer, develop Social Trading Network for a startup (Yopco)2011: joined FPT Online, software engineer, worked in banbe.net social network and VnExpress Mobile Restful API2012: backend engineer at Greengar Studios12/2012 to now - back to FPT Online, lead engineer, developed new version Data Analytics Platform (ad-network eclick.vn and VnExpress News)

Introduction about myself

1. What is Rfx ?2. Inception and Ideas3. How Rfx was born4. Why is Rfx ?

a. from big picture view (business)b. from business viewc. from specific problems

5. Concepts and architecture: The BIG picture6. Coding and tutorials from practical problems7. Resources for self-studying

Contents of this talk

● A framework for reactive real-time big fast data● A collection of Open Source Tools ● The mission of RFX

→ “BUILD digital data-driven brain for every company in the World”

What is RFX or Reactive Function X ?

INCEPTION and Ideas

Ideas when I was student, internship at DRD, non-profit OrganizationMore info at http://activefunctor.blogspot.com

Challenges for today computing

Ask bigger questions

http://singularityhub.com/2014/04/20/new-imaging-method-shows-young-neurons-making-connections-exchanging-information/

Complex brain network topology

Digital Brain Topology

Think about this module

Why Rfx ?

● Ideas since 2007 (from Haskell and Actor model theory)● R&D and Deployed in Production since 2013

● Open Source: Apache License, Version 2.0 ● Full Stack: Front-end and Back-end● Apply Agile for Analytics and Data Science● Apply Reactive Lambda Architecture

● Really fast and near-real-time processing● Tested with 1.000.000 logs / second (1 million in 1 second)

● Simple development model for big data developer

What problems can be solved with Rfx

Domain (in business) where Rfx can be used

● real-time data analytics for digital marketing, advertising● hospital systems● personal banking system● financial institution to detect frauds● manufacturing plant● airline systems● online trading system● emergency control system● manufacturing plant management system● road tolling system detects● social networking site

Paper: http://vialab.science.uoit.ca/textvis2011/papers/textvis%202011-rohrdantz.pdf

Problem: How to monitor Mobile Web Performance and react to slow response time

http://sixrevisions.com/mobile/pay-attention-to-mobile-web-performance

Luggage management system, events are produced by the check-in process and by the various radio-frequency identification (RFID) readers, which emit events about the movement of the luggage in the system. The events generated by the event processing system are consumed by the luggage control system itself, by airport staff, or even by the passengers themselves.

Problem: Monitoring sensor data and real-time security checking

ActorUser, Mobile, Browser, ...

Reactive Lambda Architecture

System Rfx-Topology

data + context + metadata

useful (data + relationship)

DatabaseNoSQL

1. Actor → System 2. System → Database3. Database → System4. System → Actor

Concepts● Each user, who uses the services and creates data, is the

actor in system

● Actor is the source of all events (aka: logs), (click, reading

news, sending message to friends, playing games, ...)

● Functor (aka: neuron) is a computing object, used for

storing, processing data and emitting results to subscribed

functors

● Topology is the directed graph, define how functors that

are connected with stream data and process data

Philosophy and the Architecture

There are 3 demos, from simple to advanced user story

User story 1: Counting Real-time URL PageviewUser story 2: Monitoring Social Media StatisticsUser story 3: Social Ranking for Recommendation Engine

User Story

Domain problem: Reactive Real-time MarketingUser story’s details:1. User does read news from a website

→ tracking user activities (pageview, time on site)2. User does login with Facebook Account3. User clicks on like Facebook button

→ tracking what user liked, commented4. The marketer/data analyst should see the trending most

read article in real-time● → Personalized articles for the reader ● → Native advertising in real-time

Demo user story 1: Counting Real-time URL Pageview

Input:1. The pageview logs from HTTP

Output:1. The total number of page-view2. The total number of page-view per hour3. The total number of page-view per minute4. The total number of page-view per second5. The total number of page-view for URL

Demo user story 2: Monitoring Social Media Statistics

Input:1. The pageview logs from HTTP

Output:The social media statistics from:1. Facebook: Like, Share, Comment2. Twitter: Tweet Count3. LinkedIn: Share Count4. Geolocation heat-map report

Demo user story 3: Social Ranking for Recommendation EngineInput:● Data: the URL of article● Context: where (User's Location), when (time visit), from

where (referer url)● Metadata: keywords, category of article

Output:Real-time Statistics about pageview, social media statistics (Share, Like, Comment), recommended articles

The list of articles are ranked by:● most liked and same category● most viewed and same category● most liked, same category and near user's location

Reference ResourcesMain website for Rfx: http://www.mc2ads.comIdeas:

● http://journal.frontiersin.org/Journal/10.3389/fninf.2010.00112/full● http://singularityhub.com/2014/04/20/new-imaging-method-shows-young-neurons-making-

connections-exchanging-information● http://en.wikipedia.org/wiki/Actor_model_theory● http://java.dzone.com/articles/introduction-event-processing● http://www.technologyreview.com/featuredstory/526501/brain-mapping● http://www.technologyreview.com/featuredstory/513696/deep-learning

Apache Storm: http://storm.incubator.apache.orgApache Kafka: http://kafka.apache.orgIn-memory NoSQL: http://redis.ioDeep Learning for Java: http://deeplearning4j.orgDistributed processing with Actor Model: http://akka.ioPapers: ● Real-Time Visualization of Streaming Text Data● Hypernetworks for the Science of. Complex Systems

Main Blogs: http://www.mc2ads.org

The end and thank youhttps://github.com/mc2ads/rfx

http://www.mc2ads.org

λ(x)