MesosCon NA 2016

Apache Spark is one of the most widely used systems for big data processing, and it has been integrated with Mesos from day one -- in fact, Spark started as an example framework running on the Mesos cluster manager. In this talk, I'll discuss where Spark fits in the data processing ecosystem, and some exciting additions coming out soon in Spark 2.0. In particular, unlike Apache Hadoop, Spark was designed to be a computing engine agnostic of storage systems, so that it can connect to a variety of systems beyond HDFS. This choice proved useful because most organizations use a mix of storage systems and want to run Spark on elastic infrastructure such as Mesos. Spark 2.0 will go further in this direction through support for new data sources, high-level APIs that can push key parts of the computation into the storage systems, and a new Structured Streaming API that provides similar optimizations for stream processing.