Lightbend Fast Data Platform

48
Lightbend Fast Data Platform A Technical Overview

Transcript of Lightbend Fast Data Platform

Lightbend Fast Data Platform

A Technical Overview

A Bit of History:Monoliths to Microservices

lightbend.com/reactive-microservices-architecture

• Tangled responsibilities, lead to… • Infrequent, “big-bang” deployments • Runtimes: months to years • Latencies: μsecs to seconds

Monoliths

• Each does one thing, so… • Updates easier to deploy frequently • Message driven & asynchronous • Runtimes: minutes! to forever • Latencies: higher than function calls

Microservices

Reactive Systems

reactivemanifesto.org

OpenSourceCoreForDevelopmentinScala/Java

Produc8onSuiteForOpera8ons/DevOps

Lightbend Reactive Platform

A Bit of History:Big Data

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode

DiskDiskDiskDiskDisk

NodeMgr

DataNode

Master

ResourceManager

NameNode

Hadoop

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode

DiskDiskDiskDiskDisk

NodeMgr

DataNode

Master

ResourceManager

NameNode

Hadoop 2013: Embrace Spark

Sparkjob#1

Sparkjob#2

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode

DiskDiskDiskDiskDisk

NodeMgr

DataNode

Master

ResourceManager

NameNode

Go beyond batch?

Sparkjob#1

Sparkjob#2

The Emergence of Fast Data:(Time Is Money)

Respond to change

IoT

16

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

Streaming

Flink

Fast Data and Microservices; Are they Converging?

• Each (stream|μservice): • does one responsibility • has unending (data|messages)

Synergies

18

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

Streaming

Flink

• Each (stream|μservice): • encourages asynchrony • offers never-ending service

Synergies

19

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

Streaming

Flink

• Reasons these architectures will converge: 1.Similar design problems 2.Data becomes dominate problem

Thesis

20

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

Streaming

Flink

Lightbend Fast Data Platform

1. An accelerated on ramp for building streaming data systems and applications.

Value Three Ways

2. Best practices guidance, through sample apps, documentation, and enablement services for what tools and techniques to use for solving specific design problems.

Value Three Ways

3. Intelligent monitoring and management, driven by machine learning, to keep your systems resilient, scalable, and responsive with minimal user intervention.

Value Three Ways

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Service 1

Log & Other Files

Internet

Services

Service 2

Service 3

Services

Services

N * M links ConsumersProducers

Service 1

Log & Other Files

Internet

Services

Service 2

Service 3

Services

Services

N + M links ConsumersProducers

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

• Low latency? How low? • High volume? How high?

Streaming Tradeoffs (1/3)

33

• Which kinds of data processing & analytics are required?

• How will this processing be done? • Individual processing of events? • Bulk processing of records?

Streaming Tradeoffs (2/3)

34

• Which tools and data sources/sinks must interoperate with your streaming tool?

Streaming Tradeoffs (3/3)

35

36

37

• Low latency • Low volume • Complex flows • Complex Event

Processing

38

• Med. latency • High volume • Data flows, SQL • En masse

processing

39

• Low latency • High volume • Data flows,

correctness • En masse

processing

40

• Low latency • Med. volume

• ETL, “tables” • Data flow or Per

Event

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS,MarathonOnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

GravitonClusterAnalysis

MachineLearningMicroservices

ProducConSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

For More Information:

lightbend.com/fast-data-platform

bit.ly/lightbend-fast-data

For More Information

• FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data