OceanStore: An Architecture for Global - Scale Persistent Storage

John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski, Patric Eaton, Dennis Geels,

Ramakrishna Gummadi, Sean Rhea, Hakim Weatherspoon, Westley Weimer, Chris Wells,

and Ben Zhao

Ελευθερία Φιλτζαντζίδη, 2002

OceanStore

Ubiquitous Computing: Car, Clothing, Books, Houses.Computing devices must have high

performance.Computing devices should consume low power.Computing devices must be transparent to the

user.

Persistent information is necessary for transparency.

Where does persistent information reside?

OceanStore

Requirements for a persistent infrastructure. Connectivity through: cable-modems, DSL, cell phones

and wireless data services. Information must be kept secure. Information must be extremely durable. Archiving of information should be automatic and

reliable. Information must be divorced from location.

OceanStore is a utility infrastructure for persistent storage.

OceanStore

As a rough estimate, OceanStore will provide service to roughly 1010 users, each with at least 10,000 files.

OceanStore must therefore support over 1014 files. Consumers will pay a monthly fee in exchange for

acess to persistent storage follow services. Companies buy and sell capacity from each other. The core of the system is composed of a multitude of

highly connected “pools”.

The OceanStore system

Two Unique Goals

Untrusted Infrastructure Servers may crash without warning or leak information

to third parties.

Only clients can be trusted.

All the information that enters the OceanStore is

encrypted.

Nomadic Data: Data that is allowed to flow freely. Promiscuous Caching: Data can be cached anywhere,

anytime.

Applications

Groupware and personal information management tools. (calendars, email, contact lists and distributed design tools)

OceanStore can be used to create large digital libraries and repositories for scientific data.

OceanStore provides an ideal platform for new streaming applications, such as sensor data aggregation and dissemination.

System Architecture

System OverviewNamingAccess ControlData Location and RoutingUpdate ModelDeep Archival StorageIntrospection

System Overview

The fundamental unit is the persistent object Objects exist in both active and archival forms.

Active Object: Is the latest version of its data together with a handle for update.

Archival Object: Permanent read-only version of the object. Archival Objects are encoded with an erasure code.

The OceanStore API provides: sessions, session guarantees, updates and callbacks.

OceanStore provides an array of familiar interfaces such as the Unix and a transactional interface.

Naming

Objects are identified by a GUID, a pseudo-random, fixed-length bit string.

An object GUID is the secure hash of the owner’s key and some human-readable name (Self-certifying path).

Certain objects act as directories, mapping human-readable names to GUIDs (SDSI).

A user can choose several directories as “roots”. The system as a whole has no “roots”.

Access Control

Reader restrictionAll data that is not completely public is

encrypted. The encryption key is distributed to those users with read permission.

Writer restrictionAll writes can be verified against an access

control list (ACL).An owner of an object can choose the ACL x

for an object foo by providing a signed certificate.

Data Location and Routing

OceanStore messages are labeled with a destination GUID, a random number, and a small predicate.

OceanStore combines data location and routing The task of routing is handled by the aggregate resources

of many different node. Messages route directly to destinations. The underlying infrastructure has more up-to-date routing

information.

Routing mechanism is a two-tiered approach. Probabilistic algorithm. Deterministic algorithm.

Probabilistic Algorithm Is fully distributed and uses a constant amount of

storage per server. Using an array of D normal Bloom filter the attenuated

Bloom filter. The first filter contains the objects which are locally on the

node. The ith Bloom Filter is the union of all the Bloom filters for all

of the nodes a distance I through any path from the current node.

Bloom Filter : A method for representing a set A. It consists of a vector B of m bits and k hash functions

h1,h2,..hk of range {1..n}. For each element of the set A, the bits at positions

h1(a),h2(a)..hk(a) of the vector B is set to 1.

Given a query for b we check the bits at positions h1(a), h2(a), ..hk(a).

The Probabilistic Query Process

The Global Algorithm

OceanStore uses a variation on Plaxton et. al.’s hierarchical distributed data structure.

The Basic Plaxton scheme Every server in the system is assigned a random node-ID. Each link is labeled with a level number.

In OceanStore each object is mapped to a single node whose node-ID matches the object’s GUID in the most bits. This node is called object’s root.

The location of a replica is “published” in the infrastructure.

This process requires O(logn) hops.

The Global Algorithm

The Global Algorithm

Inserting Object #62942

116

479

529

629

116

479

529

629

675

109

Searching Object #62942

Root Node

Search Client

Object Location

Update Model

Update model base on Conflict Resolution - Bayou System.

Update: list of predicates associated with actions. Commit Abort

Possible Predicates compare-version, compare size : applied to unencrypted

meta-data of an object. compare-block: the encryption technology is a position-

dependent block cipher. search: is preformed directly to cipher data.

Update Model

Operations applied to ciphertext replace block, append block: a position dependent block

cipher. insert block, delete block

Two sets of blocks: index blocks, data blocks.

Update Model Someone must choose the final commit order of

updates. OceanStore choose two tier of replicas

A primary tier of replicas: Byzantine agreement protocol. Small number of replicas located in high bandwidth, high -

connectivity regions of the network. Stronger consistency guarantees.

A secondary tier of replicas: Epidemic Algorithm. They are organized into dissemination trees. Contain both tentative and committed data. Secondary replicas order tentative updates in timestamp

order. Lesser degree of consistency.

Update Model

After generating an update, a client send it to the object’s primary tier,as well as to several random replicas for that object.

The primary tier performs the Byzantine protocol. The secondary replicas propagate the update among themselves epidemically.

The result is multicast down the dissemination tree.

Deep Archival Storage

The archival mechanism employs erasure codes (interleaved Read-Solomon, Tornado codes).

Erasure coding treats data as a series of fragments and transforms these fragments into a greater number of fragments.

The fragments are spread widely. Any n of the coded fragments are sufficient to construct the original data.

Fragmentation increases reliability and survivability.

Introspection

Introspection augments a system’s normal operation (computation) with observation and optimization.

The Cycle of Introspection An architecture for introspective systems in OceanStore

Status

The first implementation is deployed in Java. They use the Unix file system interface and a read-

only proxy for the WWW. They have explored the security guarantees that

are required for the OceanStore. Included Components

A prototype for the probabilistic algorithm. Prototype archival systems that use Read-Solomon and

Tornado codes for redundancy encoding. An introspective prefetching mechanism for a local file

system.