MesosCon NA 2016

The reliable and scalable access to databases with a standardized interface like SQL or an ORM remains a real challenge. Crate is an open source, highly scalable, shared-nothing distributed SQL database. It offers the scalability and performance of a modern NoSQL database with the power of SQL. Still, the nodes forming such a cluster need to be orchestrated.

With the introduction of persistent volumes in Mesos 0.23, reliability of stateful services improved. After a task failure or a Mesos agent restart, the stateful framework guarantees the availability of stateful data. As such it is a natural fit for a clustered database like Crate.

The Crate Mesos Framework, developed as an open source project, integrates the data storage layer with Mesos. It is a good fit for stateful services that run on top of Mesos. It avoids operations teams to manually install Crate on their cluster with resources, such as disk space, statically partitioned. Instead, it scales infinitely due to its shared-nothing architecture. The framework is highly fault tolerant: If a node or task fails, Mesos reschedules Crate instances and attaches them to the existing Crate cluster. The framework makes use of Mesos-DNS for data rebalance operations and service discovery.

The Crate Mesos Framework API simplifies managing the cluster. Using the framework API, system managers scale the cluster simply by changing the number of instances. Scaling out is easy: By using constraint rules, the Mesos framework enforces only one Crate instance to run on each host in our configuration. Scaling back by gracefully removing nodes from the cluster, is a challenge, though. The Crate Mesos framework API takes care of this task, taing into account a number of corner cases.