JavaOne 2015: Scaling micro services at Gilt

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of JavaOne 2015: Scaling micro services at Gilt

  • scaling -services at Gilt [email protected]

    San Francisco26th October 2015

    Adrian Trenaman, SVP Engineering, Gilt, @adrian_trenaman


  • gilt: luxury designer brands at discounted prices

  • we shoot the product in our studios

  • we receive, store, pick, pack and ship...

  • we sell every day at noon...

  • stampede...

  • this is what the stampede really looks like...

  • rails to riches: 2007 - ruby-on-rails monolith

  • 2011: java, loosely-typed, monolithic services

    (5) Hidden linkages; buried business logic

    (4) Monolithic Java App; huge bottleneck for innovation.

    (2) Lots of duplicated code :(

    (3) Teams focused on business lines

    (1) Large loosely-typed JSON/HTTP services

  • enter: -services

    How can we arrange our teams around strategic initiatives? How can we make it fast and easy to get to change to production?

  • 2015: micro-services

  • driving forces behind gilts emergent architecture

    team autonomy voluntary adoption (tools, techniques,

    processes) kpi or goal-driven initiatives failing fast and openly open and honest, even when its difficult

  • service growth over time: point of inflexion === scala.

  • anatomy of a gilt service

  • anatomy of a gilt service - typical choices


    log4j, cloudwatch Cave,

    , , javascript


  • service discovery: straight forward


    Brocade Traffic Manager (aka Zeus, Stringray, SteelApp,...)

  • what are all these services doing?

  • we used a spread sheet.The Gilt Genome Project

  • Its hard to think of architecture in one dimension.

    We added Functional Area, System and Subsystem columns to Gilt Genome; provides a stronger (although subjective) taxonomy than the previous tags.

    It turns out we have an elegant, emergent architecture.

    Some services / components are deceptively simple.

    Others are simply deceptive, and require knowledge of their surrounding constellation

    n = 265, where n is the number of services.

  • Deceptively Simple - many services are small; < 2048 loc

  • Deceptively Simple - many services are small, < 32 files.

  • Gilt Admin (Legacy Ruby on Rails Application)


    Discounts FinancialReporting

    Fraud Mgmt

    Gift Cards Inventory Mgmt Order Mgmt

    Sales Mgmt Product Catalog

    Purchase Orders



    Other Admin Applications (Scala + Play Framework)*

    City Creative (2) CS

    Discounts Distribution i18n Inventory (2)

    Order Processing


    Service Constellations (Scala, Java)*

    Auth (1) Billing (1) City (6) Creative (4) CS (2) Discounts (1) Distribution (9) i18n (3) inventory (6)

    Order Processing

    (8)Payments (3) Product Catalog (5) Referrals (1) Util (2)

    Core Database - db3

    Job System (Java, Ruby)

    Gilt Logical Architecture - Back Office Systems

    * counts denote number of service / app components.

    Simply deceptive: service context only make sense in constellation.

  • from bare-metal...


  • to vapour.

  • Lift-and-shift + elastic teams

    Existing Data Centre

    Dual 10Gb direct connect line, 2ms latency.

    Legacy VPC

    MobileCommon Person-alisation Admin Data

    (1) Deploy to VPC

    (2) Department accounts for elasticity & devops

  • single tenant: one EC2 instance per service instance

  • reproducible, immutable deployments: docker

  • service discovery: same pattern, different LB


    Amazon ELB

  • # running instances per service: rule of three

  • AWS instance sizing

  • evolution of architecture and tech organisation

  • Lessen dependencies between teams: faster code-to-prod

    Lots of initiatives in parallel

    Your favourite here

    We (heart) -servicesGraceful degradation of service

    Disposable Code: easy to innovate, easy to fail and move on.

  • We (heart) cloudDo devops in a meaningful way.Low barrier of entry for new tech (dynamoDB, Kinesis, ...)Isolation

    Cost visibilitySecurity tools (IAM)Well documentedResilience is easyHybrid is easyPerformance is great

  • seven -service challenges (& some solutions) no one ever said this was gonna be easy

  • 1. staging vs test-in-prodWe find it hard to maintain staging environments across multiple teams with lots of services.

    We think TiP is the way to go: invest in automation, use dark canaries in prod.

    However, some teams have found TiP counter-productive, and use minimal staging environments.

  • 2. ownershipWho owns that service? What happens if that person decides to work on something else?

    We have chosen for teams and departments to own and maintain their services. No throwing this stuff over the fence.

  • 1. Software is owned by departments, tracked in genome project. Directors assign services to teams.

    2. Teams are responsible for building & running their services; directors are accountable for their overall estate.

    bottom-up ownership, RACI-style

  • ownership donut informs tech strategy

    3. Ownership is classified: active, passive, at-risk.

    done === 0% at risk

  • 3. deploymentServices need somewhere to live. Weve open-sourced tooling over docker and AWS to give:

    elasticity + fast provisioning + service isolation+ fast rollback

    + repeatable, immutable deployment.

  • 4. lightweight APIsWeve settled on REST-style APIs, using Separate interface from implementation; an AVRO for REST (Mike Bryzek, Gilt Founder)

    We strongly recommend zero-dependency strongly-typed clients.


  • 5. audit + alertingHow do we stay compliant while giving engineers full autonomy in prod?

    Really smart alerting:

    orders[shipTo: US].count.5m == 0

  • 6. io explosionEach service call begets more service calls; some of which are redundant...=> unintended complexity and performance

    Looking to lambda architecture for critical-path APIs: precompute, real-time updates, O(1) lookup

  • 7. reportingMany services => many databases => data is centralized.

    Solution: real-time event queues to a data-lake.

  • scaling -services at Gilt [email protected]

    San Francisco26th October 2015

    Adrian Trenaman, SVP Engineering, Gilt, @adrian_trenaman