Loading…
This event has ended. Create your own event → Check it out
This event has ended. Create your own
View analytic
Thursday, June 2 • 4:20pm - 5:10pm
Lessons Learned from Running Heterogeneous Workload on Apache Mesos - Imran Shaikh, Yellow Pages

Sign up or log in to save this to your schedule and see who's attending!

Mesos is by far the best orchestration system available out there for running containers. At YP, when we started using Mesos couple of years ago, we were overwhelmed with the robustness it provided. We quickly started on-boarding some of the apps. We soon were faced with multiple request from various developers to on-board their apps onto Mesos. These apps ranged from infrastructure tools to highly critical billing systems, from traditional java to web based apps, from map reduce jobs to batch processing jobs, from one off cron jobs to build pipeline jobs. These apps were written in Java, Go Lang, Node.js, Ruby/Rails, C, Scala as well as Python. And their configuration came from git repo, key value store, pom files or some static files managed somehow. We were made to provide a unified platform that served the use-cases for all of them or at least come up with the processes that everyone can use.

Initially, we relied heavily on the out of the box features that Mesos provided. But that wasn’t enough to support these myriad range of apps. For eg: redirecting all logs to STDOUT/STDERR wasn’t enough or possible for some of these apps. Managing app secrets was very insecure and exposed via ENV variables in Marathon. Also, managing app configs for various environment was tedious. We wanted to achieve persistence and fault tolerance with block devices to run databases through Mesos. There was no easy way to troubleshoot running processes in production. There was a need to manage multiple similar frameworks within the same Mesos cluster. And the list goes on and on.

We, at YP, having been exposed to this madness. While doing so, we have picked up valuable information along the way. We may not have solved all the issues, but we have made significant strides to resolve them.

Speakers
avatar for Imran Shaikh

Imran Shaikh

Lead/Architect, YP
Imran is a Lead/Architect working at YP. He is a proven technical leader with an industry experience of 13+ years working with Fortune 500 companies. He has worked extensively developing, architecting and managing cloud and distributed computing technologies at YP and Yahoo. He has substantial experience running a globally distributed production environment on thousands of systems running hundreds of application in a complex, fast moving and... Read More →



Thursday June 2, 2016 4:20pm - 5:10pm
Grand Mesa DE