Post on 21-Dec-2015
1
Lessons from Giant-Scale Services
IEEE Internet Computing, Vol. 5, No. 4., July/August 2001
Eric A. BrewerUniversity of California, Berkeley,
and Iktomi Corporation
Παρουσίαση: Ηλίας Τσιγαρίδας (Μ484)
2
Examples of Giant-scale services
Aol Microsoft network Yahoo eBay CNN Instant messaging Napster Many more…
The demand
They must be always available, despite their scale, growth rate, rapid evolution of content and features, etc
3
Article Characteristics
Characteristics “Experience”
article No literature
points Principles
approaches Not quantitative
evaluation
The reasons Focusing on high
level design New area Proprietary nature
of the information
4
Article scope
Look at the Basic Model of the giant-scale services
Focusing the challenges of High availability Evolution Growth Principles for the above
Simplify the design of large systems
5
Basic Model (general)
The “infrastructure services” Internet-based systems that provide
instant messaging, wireless services and so on
6
Basic Model (general)
We discuss Single-site Single-owner Well-connected
cluster Perhaps a part of
a larger service
We do not discuss Wide are issues
Network partitioning Low or discontinuous
bandwidth Multiple admistrative
domains Service monitoring Network QoS Security Log and logging
analysis DBMS
7
Basic Model (general)
We focus on High availability Replication Degradation Disaster tolerance Online evolution
The scope is bridging the gap between
the basic building block of giant-scale services
and the real world scalability and availability they require
8
Basic Model (Advantages)
Access anywhere, anytime Availability via multiple devices Groupware support Lower overall cost Simplified service updates
9
Basic Model (Advantages)
Access anywhere, anytime The infrastructure is ubiquitous You can access the service from
home, work airport and so on
10
Basic Model (Advantages)
Availability via multiple devices The infrastructure handles the
processing (the most at least) User access the services via set-top
boxes, networks computer, smart phones and so on
In that way we have offer more functionality for a given cost and battery life
11
Basic Model (Advantages)
Groupware support Centralizing data from many users
allowing group-ware application like Calendar Teleconferencing systems, and so on
12
Basic Model (Advantages)
Lower overall cost Hard to measure overall cost but
Infrastructure services have an advantage over designs based on stand alone devices
High utilization Centralize administration reduce the cost,
but harder to quantify
13
Basic Model (Advantages)
Simplified service updates Updates without physical distribution The most powerful long term
advantage
14
Basic Model (Components)
15
Basic Model (Assumptions)
The service provider has limited control over the clients an the IP network
Queries drive the service Read only queries outnumber
greatly update queries Giant-scale services use CLUSTERS
16
Basic Model (Components)
Clients, such as Web browsers. Initiate the queries to the services
IP network, public Internet or a private network. Provides access to the service.
Load manager, provides indirection between the service’s external name and the servers’ physical names (IP addresses). Load balancing. Proxies or firewalls before the load manager.
Servers. Combining CPU, memory, and disks into an easy-to-replicate unit.
Persistent data store, replicated or partitioned database spread across the servers. Optional external DBMSs or RAID storage.
Backplane. Optional. Handles inter-server traffic.
17
Basic Model (Load Management)
Round Robin DNS “Layer-4” switches
understand TCP and port numbers
“Layer-7” switches parses URL
Custom “front-end” nodes They act like service specific “layer-7” routers
Include the clients in the load balancing Ex alternative DNS or Name Server
18
Basic Model (Load Management)
Two opposite approaches Simple Web Farm Search engine cluster
19
Basic Model (Load Management)Simple Web Farm
20
Basic Model (Load Management)Search engine cluster
21
High Availability (general)
Like telephone, rail or water systems Features
Extreme symmetry No people Few cables No external disks No monitors
Inkotomi in addition Manages the cluster offline Limit temperature and power variations
22
High Availability (metrics)
MTBF MTTRuptime
MTBF
queries completedyield
queries offered
data availableharvest
complete data
23
High Availability (DQ principle)
The systems overall capacity has a particular physical bottleneck Ex. Total I/O bandwidth, total seeks per second
Total amount of data to be moved per second
Measurable and tunable Ex. adding nodes, software optimization OR
faults
Data per query×Queries per second constant
24
High Availability (DQ principle)
Focus on the relative DQ value, not on the absolute
Define the DQ value of your system
Normally DQ values scales linearly with the number of the nodes
25
High Availability (DQ principle)
Analyzing the faults impact Focus on how DQ reduction
influence the three metrics Only for data-intensive sites
26
High AvailabilityReplication vs. Partitioning
Replication 100% harvest 50% yield DQ -= 50%
Maintain D Reduce Q
Partitioning 50% Harvest 100% yield DQ -= 50%
Reduce D Maintain Q
Example: 2-node cluster. One down
27
High AvailabilityReplication
28
High AvailabilityReplication vs. Partitioning
Replication wins if the bandwidth is the same.
Extra cost is on the bandwidth not on the disks
Easy recovering We might also use partial
replication and randomization
29
High AvailabilityGraceful degradation We can not avoid saturation, because
Peak-to-average ratio 1.6:1 to 6:1. Expensive to build capacity above the (normal) peak
Single events burst (ex. Online ticket sales for special events)
Faults like power failures or natural disaster affect substantially the overall DQ and the remaining nodes become saturated.
So, we MUST have mechanisms for degradation
30
High AvailabilityGraceful degradation
The DQ principle give us the options for Limit Q (capacity) to maintain D Reduce D and increase Q Focus on harvest by Admission Control (AC)
Reduce Q Reduce D on dynamic databases Both
Cut the effective database to half (new approach)
31
High AvailabilityGraceful degradation
More sophisticated techniques Cost based AC
Estimate query cost Reduce the data per query Augment Q
Priority (or value) based AC Drop low-valued queries Ex execute stock trade within 60s or the user pays
no commission Reduced data freshness
Reduce the freshness so reduce the work per query Increase yield at the expense of harvest
32
High AvailabilityDisaster Tolerance
Combination of managing replicas and graceful degradation
How many locations? How many replicas on each
location? Load management
“Layer-4” switch do not help with the loss of a whole cluster
Smart clients is the solution
33
Online Evolution & Growth
We must plan for continuous growth and frequent functionality updates
Maintenance and upgrades are controlled failures
Total loss of DQ value is ΔDQ = n · u · average DQ/node = DQ · u Where n is the number of nodes and u the
total amount per time a node requires for online evolution
34
Online Evolution & GrowthThree approaches
An example for a 4-node cluster
35
ConclusionsThe basic lessons learned
Get the basics right Professional data center, layer-7 switch,
symmetry Decide on your availability metrics
Everyone must agree on the goals Harvest and yield > uptime
Focus on MTTR at least as much as MTBF MTTR is easier and has the same impact
Understand load redirection during faults Data replication is insufficient, you need
excess DQ
36
ConclusionsThe basic lessons learned
Graceful degradation is a critical part Intelligent admission control and dynamic
database reduction Use DQ analysis on all upgrades
Capacity planning Automate upgrades as much as
possible Have a fast simple way to return to older
version
37
Final Statement
Smart clients could simplify all of the above