Partition-Tolerant Distributed Publish/Subscribe System

download Partition-Tolerant Distributed Publish/Subscribe System

of 49

  • date post

    08-Jul-2015
  • Category

    Technology

  • view

    132
  • download

    1

Embed Size (px)

description

Introduction to the design choices behind partition-tolerant distributed pub-sub system

Transcript of Partition-Tolerant Distributed Publish/Subscribe System

Slide 1

Partition-Tolerant Distributed Publish/Subscribe System

MotivationChallenges

Focus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

SolutionFocus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

SolutionFocus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

Solution1-neighborhoodFocus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

Solution2-neighborhood1-neighborhoodFocus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

Solution2-neighborhood1-neighborhoodFocus on:fault tolerancereliabilityBased on:tree-overlay neighborhood knowledge - configuration parameter

Solution3-neighborhood2-neighborhood1-neighborhoodArchitecture

Architecture

Overlay partitionsAn island :

Overlay partitionsABCDEFSPDsourcedestinationAn island :

A barrier:

Partition identifier (PID) = (pd, i, pnodes)

Overlay partitionsABCDEFSPDEFABCDEFSPDsourcedestinationdestinationsourceSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscriptionssSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscriptionsssssssSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscriptionsConfirmationsssssssconfSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscriptionsConfirmationsssssssconfconfconfconfconfconfSubscription is accepted when it is added into routing tablesThat requires acknowledgments from whole outgoing setSubscription propagation when there are no overlay partitionsABCDEPSSubscriptionsConfirmationsssssssconfconfconfconfconfconfBrokers B FD detects partition, and connects to first alive broker along the pathIt removes identified nodes from Outs list and sends confirmation to upper brokers with included PID of partitionSubscription is accepted when all ACK messages are received from brokers in Outs list

Subscription propagation protocol in presence of overlay partitionsABCDEPSConfirmationsSubscriptionsCDBBrokers B FD detects partition, and connects to first alive broker along the pathIt removes identified nodes from Outs list and sends confirmation to upper brokers with included PID of partitionSubscription is accepted when all ACK messages are received from brokers in Outs list

Subscription propagation protocol in presence of overlay partitionsABCDEPSConfirmationsSubscriptionsCDBsssBrokers B FD detects partition, and connects to first alive broker along the pathIt removes identified nodes from Outs list and sends confirmation to upper brokers with included PID of partitionSubscription is accepted when all ACK messages are received from brokers in Outs list

Subscription propagation protocol in presence of overlay partitionsABCDEPSConfirmationsSubscriptionsCDBsconfssconfBrokers B FD detects partition, and connects to first alive broker along the pathIt removes identified nodes from Outs list and sends confirmation to upper brokers with included PID of partitionSubscription is accepted when all ACK messages are received from brokers in Outs list

Subscription propagation protocol in presence of overlay partitionsABCDEPSConfirmationsSubscriptionsCDBsconfssconfconf** Tag conf with pidBrokers B FD detects partition, and connects to first alive broker along the pathIt removes identified nodes from Outs list and sends confirmation to upper brokers with included PID of partitionSubscription is accepted when all ACK messages are received from brokers in Outs list

Subscription propagation protocol in presence of overlay partitionsABCDEPSConfirmationsSubscriptionsCDBsconfssconfconf*conf**pid tag is alsostored alongwith s* Tag conf with pidForwarding compromises of five steps:QueuingBarrier checkingMatchingRoutingcleanup

Publication forwardingForwarding only uses subscriptions accepted brokers.Steps in forwarding of publication p:Identify broker of accepted subscriptions that match pDetermine active connections towards matching subscriptions brokersSend p on those active connections and wait for confirmationsIf there are local matching subscribers, deliver to themIf no downstream matching subscriber exists, issue confirmation towards POnce confirmations arrive, discard p and send a conf towards PPublication propagation when there are no overlay partitionsPublicationsABCDEPSSubscriptionspCEpppppDeliver to localsubscribersconfconfconfconfconfconfpKey forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsPublicationsABCDEPSSubscriptionsKey forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsPublicationsABCDEPSSubscriptions*Key forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsPublicationsABCDEPSSubscriptionspCBD*pKey forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsPublicationsABCDEPSSubscriptionspCBD*pppKey forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsconfconfconfPublicationsABCDEPSSubscriptionspCBD*ppconfpKey forwarding invariant to ensure reliability: ensuring that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription

Publication propagation protocol in presence of overlay partitionsconfconfconfPublicationsABCDEPSSubscriptionspCBD*ppDepending on when this link has been establishedeither recovery or subscription propagation ensureC accepts s prior to receiving pconfpIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoveryIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoveryABCDEXRNew sessionIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoverysisiABCDEXRNew sessionsisisisiIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoverysisiABCDEXRNew sessioncsisicsicsicsicsicsisisisiAck messagesIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoverysisiABCDEXRNew sessioncsi*sicsicsicsicsicsisisisiAck messagesIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoverysisiABCDEXRNew sessioncsi*sicsicsicsicsicsisisisiAck messagesIs initiated upon activation of a new session.Have five steps:Notify about active sessionReply by sending a summary of subscriptionsSummary is compared to local list, missing subscriptions are transferred tooSubscriptions are accepted by R and sent to its downstream networkPartition information is updated within distance 2

Partial recoverysisiABCDEXRNew sessioncs