TheEssentialGuideTo QueueingTheory - Percona W q P (W q > 0) = A M M ! M M أ„ A M أ„ 1 i= 1 A...

download TheEssentialGuideTo QueueingTheory - Percona W q P (W q > 0) = A M M ! M M أ„ A M أ„ 1 i= 1 A i i! +

of 34

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of TheEssentialGuideTo QueueingTheory - Percona W q P (W q > 0) = A M M ! M M أ„ A M أ„ 1 i= 1 A...

  • Wq


    P(Wq > 0) =


    M ! M

    M − A M−1


    Ai i ! +


    M! M

    M − A

    The Essential Guide To

    Queueing Theory

    September 8, 2016 • Revision 3

  • Meet the Author

    Baron Schwartz

    Baron is well-known for his contributions to the MySQL,

    PostgreSQL, and Oracle communities. Baron has helped build

    and optimize database systems for some of the largest Internet

    properties. He has written several books, including O'Reilly's

    best-selling High Performance MySQL. Prior to founding

    VividCortex, Baron managed consulting, support, training, and

    software engineering at Percona. He has a CS degree from the

    University of Virginia.

    Table of Contents

    • Queueing Theory is for Everyone 3 • What is Queueing Theory? 4 • Why Queueing Theory is Tricky 5 • Queueing Systems in Real Life 6 • Why Does Queueing Happen? 9 • The Queueing Theory Framework 11 • Describing Queueing Systems 16 • Calculating Queue Lengths and Wait Times 19 • The Erlang Queueing Formulas 22 • Approximations to Erlang Formulas 23 • Many Queues or One Combined Queue? 25 • Tradeoffs Between Cost and Quality of Service 28 • Applying Queueing Theory in the Real World 29 • Conclusions 31 • Further Reading 32

    Copyright ©2015 VividCortex

  • Page 3

    Queueing Theory is for Everyone Whether you’re an entrepreneur, engineer, or manager, learning about queueing theory is a great way to be more effective. Queueing theory is fundamental to getting good return on your efforts. That’s because the results your systems and teams produce are heavily influenced by how much waiting takes place, and waiting is waste. Minimizing this waste is extremely important. It’s one of the biggest levers you will find for improving the cost and performance of your teams and systems.

    Unfortunately, queueing theory books are often terribly complicated and boring. But it doesn’t have to be that way! The concepts are easy to understand, so developing intuition is low-hanging fruit. Plus, queueing theory is super-cool, and it’s a pity if you don’t learn about it.

    There are plenty of 350-page books about queueing theory, and lots of high-level introductions too. But these all either dive deep into pages full of equations and Greek symbols, or gloss over (or just outright lie about) the big, important truths of queueing. We need a middle ground.

    My hope is that this book can help you achieve the following:

    • Develop awareness of how queueing appears in everyday life. Queueing rules everything around you, but many people probably aren’t aware of it.

    • Build intuition of important concepts and truths that can guide you in decision making.

    • Help you see where your intuition is a trusty guide and where queueing will surprise you.

    • Help you understand the landscape of queueing theory from a general framework, so you can quickly navigate more detailed treatises and find the right answers fast.

    Copyright ©2015 VividCortex

  • Page 4

    Armed with the knowledge in this book, you will be able to quickly understand common scenarios such as:

    • If web traffic increases 30%, will the server get overloaded?

    • Should I buy faster disks, or should I just buy a few more of what I already have?

    • What if I specialize tasks in my team and create a handoff between roles?

    Knowing how to think about these kinds of questions will help you anticipate bottlenecks. As a result, you’ll build your systems and teams to be more efficient, more scalable, higher performance, lower cost, and ultimately provide better service both for yourself and for your customers. And when you need to calculate exactly how much things will bottleneck and how much performance will suffer, you’ll know how to do that too.

    As for scope, the aim of this book is to help you get answers for common scenarios and point you to the right resources for other cases. I’ll go into details where they help illuminate important concepts.

    What is Queueing Theory? Queueing theory is a broad field of study that focuses on the many nuances of how waiting lines behave. It’s a fascinating topic if you’re of a theoretical bent. But it’s highly relevant even if you’re the impatient “just answer my question!” type, too.

    Queueing theory boils down to answering simple questions like the following:

    • How likely is it that things will queue up and wait in line?

    • How long will the line be?

    Copyright ©2015 VividCortex

  • Page 5

    • How long will the wait be?

    • How busy will the server/person/system servicing the line be?

    • How much capacity is needed to meet an expected level of demand?

    You can rearrange these questions easily to answer other questions, such as “how likely am I to lose business due to too-long waits or not enough capacity?” and “how much more demand can we satisfy without creating an unacceptably long wait?”

    And although I wrote the questions in simple terms, you can also answer sophisticated questions about the characteristics of the answers. For example, “how long will the wait be?” can be answered not only on average, but you can calculate things such as the variability of wait times, the distribution of wait times and how likely it is that someone will get extremely poor service, and so on.

    Why Queueing Theory is Tricky Bad news: queueing theory is based on probability, which is hard and weird. As Eric Bowman tweeted, “Not even God understands queueing theory.”1 Nothing leads you astray faster than trusting your intuition about probability. If your intuition worked, casinos would all go out of business, and your insurance rates would seem reasonable.

    In particular, human intuition is linear. Double the traffic, we think, and other things will double too. Wrong! Queueing systems are nonlinear. What’s worse is that they’re unpredictably nonlinear. Things seem to change gradually, until you reach some tipping point and boom, everything gets really bad, really fast. These transitions can be so sudden that they are very surprising. Our intuition isn’t geared to expect these disproportionate, sudden changes.


    Copyright ©2015 VividCortex

  • Page 6

    The classic example you might recognize is the so-called hockey-stick curve. This is the most famous queueing theory picture of all time:

    0.0 0.2 0.4 0.6 0.8 1.0

    0 5

    10 15

    20 25


    R es

    id en

    ce T

    im e

    0.0 0.2 0.4 0.6 0.8 1.0

    0 5

    10 15

    20 25

    This curve shows how a system’s total response time increases as utilization increases. As utilization approaches 100%, response time goes to infinity. The exact shape of this curve depends on the characteristics of the system and the work it’s doing, but the general principle applies to pretty much all systems.

    You’ll get familiar with nonlinearity over the course of this book. You might never be able to accurately estimate how much things will be nonlinear. But you’ll get pretty good at recognizing situations where someone needs to check whether bad stuff is going to happen. Then you’ll do a little bit of math or look things up in a table and find the answers you need.

    Queueing Systems in Real Life Most introductions to queueing theory talk about abstract things and use jargon a lot. They’ll usually discuss some real-life scenarios to help you understand the abstract things, but for some reason they always seem to pick bizarre situations I personally have never encountered.1 Then they

    1 Often with British terminology, something about hiring lorries and abseiling.

    Copyright ©2015 VividCortex

  • Page 7

    jump into discussions of Markovian something or other, and talk about births and deaths. At this point my eyes have usually glazed over.

    But life is full of queueing systems!

    • Your Kanban board

    • Getting a latte at the coffee shop

    • The line for the restroom after drinking the latte

    • Your computer’s CPUs and disks

    • At the traffic light

    • When the captain says you’re third in line for takeoff

    • Your load balancer

    • Pre-ordering the new iPhone

    • Trying to pick the best line at the grocery checkout counter

    • Merging onto the freeway

    • Connection pooling

    Importantly, queueing systems are present in much more subtle scenarios too. Any process for handling units of work has queueing systems built into it. When you ask your graphic designer to produce a beautiful new ebook, for example, queueing is at work.

    What is the real meaning of adding a queue into a system? At a higher level, although the mechanics of queues are simple, their ultimate results are often overlooked or misunderstood. To co-opt the old joke, the programmer with a problem thought, ”I know, I’ll just add a queue.” (Now the programmer has unbounded problems.) Let’s look at some of the consequences and purposes of queues.1

    1 I’m speaking here about queues as a concept, not specific technologies such as RabbitMQ.

    Copyright ©2015 VividCortex

  • Page 8

    Fundamentally, a queue is a mechanism for increasing availability. A server is available when it can accept and service a request. In the formal sense, a server is either busy or id