MesosCon NA 2016

Twitter’s compute platform manages hundreds and thousands of containers across tens and thousands of hosts. At that scale, problems we have observed are:
1. Containers are not often sized appropriately in a way to utilize underlying hardware resources efficiently
2. Lack of visibility into who owns what containers (and services)
3. Identifying and tracking the utilization of resources by individual containers to the org structure

We built a Chargeback system for Mesos that provides visibility into who owns what and how much resources they use. It also generates a "bill" based on usage data. The goal here is to incentivize users of Mesos through visibility into what they use & effectively right size their containers to use the underlying resources in a cost-effective way. In this talk, we describe how we built the chargeback service that
1. Ingests utilization and quota data from Mesos master & container stats
2. process around figuring out resources to charge on (ex, cpu cores used per day)
3. ownership mapping i.e how we built a identity service that ties ldap roles with organizational teams (takes a more project/team centric view)

Towards the end of the talk, we will share a case study of how chargeback at Twitter helped improve resource utilization (cores in this case) by over 30% and save millions in $ prior to implementing something like Chargeback.