Lightbend Fast Data Platform
-
Upload
lightbend -
Category
Technology
-
view
211 -
download
0
Transcript of Lightbend Fast Data Platform
lightbend.com/reactive-microservices-architecture
• Tangled responsibilities, lead to… • Infrequent, “big-bang” deployments • Runtimes: months to years • Latencies: μsecs to seconds
Monoliths
• Each does one thing, so… • Updates easier to deploy frequently • Message driven & asynchronous • Runtimes: minutes! to forever • Latencies: higher than function calls
Microservices
OpenSourceCoreForDevelopmentinScala/Java
Produc8onSuiteForOpera8ons/DevOps
Lightbend Reactive Platform
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Hadoop
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Hadoop 2013: Embrace Spark
Sparkjob#1
Sparkjob#2
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Go beyond batch?
Sparkjob#1
Sparkjob#2
16
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
• Each (stream|μservice): • does one responsibility • has unending (data|messages)
Synergies
18
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
• Each (stream|μservice): • encourages asynchrony • offers never-ending service
Synergies
19
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
• Reasons these architectures will converge: 1.Similar design problems 2.Data becomes dominate problem
Thesis
20
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
2. Best practices guidance, through sample apps, documentation, and enablement services for what tools and techniques to use for solving specific design problems.
Value Three Ways
3. Intelligent monitoring and management, driven by machine learning, to keep your systems resilient, scalable, and responsive with minimal user intervention.
Value Three Ways
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Service 1
Log & Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers
Service 1
Log & Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
• Which kinds of data processing & analytics are required?
• How will this processing be done? • Individual processing of events? • Bulk processing of records?
Streaming Tradeoffs (2/3)
34
• Which tools and data sources/sinks must interoperate with your streaming tool?
Streaming Tradeoffs (3/3)
35
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS,MarathonOnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
GravitonClusterAnalysis
MachineLearningMicroservices
ProducConSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
For More Information:
lightbend.com/fast-data-platform
• FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data