Mesos + DCOS: Mesosphere's Vision for the Open Cloud

Editor’s Note: This article is paid for by Mesosphere as a Platinum-level sponsor of MesosCon, to be held Aug. 20-21, 2015, and was written by Linux.com.

Mesosphere last week announced a partnership to integrate Google’s Kubernetes container management tool with its own growing data center operating system (DCOS).

The move will allow developers to use the same DCOS command line interface to manage Kubernetes applications alongside their Apache Mesos workloads. But, perhaps more importantly, they can build and manage web-native applications with Kubernetes or Mesos and run them across private data centers and public cloud platforms.

Such true infrastructure portability has long been the holy grail for the visionaries and architects of the open cloud. But exactly how to achieve it, and the level of openness it requires, remains a source of contention.

Mesosphere, which builds commercial products around Mesos, presents a model for how Mesos and a data center operating system, such as the one it announced four months ago, could help facilitate interoperability, and portability, in the open cloud.

“We can get to the world where we have a POSIX-like API for distributed systems, so that anybody can build a distributed system and effectively compile it against this API,” said Benjamin Hindman, Founder of Mesosphere and former lead of the Mesos project at Twitter. “Mesos can provide this POSIX API.”

Mesos is the Data Center OS Kernel

In the data center OS model, Mesos behaves like the Linux kernel, managing hardware resources, including CPU and memory, and providing that information to application frameworks to schedule tasks across dozens or thousands of machines. Mesosphere’s DCOS is then akin to an enterprise Linux distribution like RHEL, where the Mesos kernel is packaged along with other services such as a user interface for operators.

The DCOS spans all of the machines in the cluster, but relies on Linux as the host operating system on each individual machine. Thus developers can still build and compile their applications to take full advantage of the Linux ecosystem.

In the same way that some enterprises go and download the Linux kernel directly to run their systems instead of using an enterprise distribution, there are hard core tech companies that go to mesos.apache.org, download Mesos and run with it, Hindman says. But most companies don’t have the IT resources, knowledge or desire to use Mesos directly. Mesosphere’s DCOS aims to make Mesos more consumable for companies.

“This part of distributed computing (a data center OS) is still very new – but that’s what we’re building with our DCOS product” Hindman said.

Distributed computing today is a bit like going back to the days when developers were writing the first applications for personal computers, doing their own manual memory management, Hindman says. Writing multiple applications crashed the system because the applications were agnostic of each other. Then virtual memory management (VMM) software was added to the kernel. Nobody does their own memory management anymore.

“We’re at the same stage of distributed computing,” Hindman says. “Our opportunity is to figure out the equivalencies of the VMM in the distributed space, that fail less and crash less and make people more productive when it comes to the stuff they’re trying to build.”

DCOS makes it easier to administer application frameworks built on top of Mesos, including Spark and Chronos. DCOS has extensions that make it easier to write and deploy distributed applications on top of the kernel. And there are many undiscovered opportunities for developers to build new frameworks that ease data center management and administration. To identify these sore spots and build out the solutions, more companies need to be thinking about and building distributed systems in this way, Hindman says.

Getting to Ease and Portability

The ultimate goal is to easily build applications that can move between different physical or cloud based infrastructures run by different organizations, just like you can move (most) applications from one computer to another, regardless of the operating system – Windows, Linux or OSX, Hindman says.

“Wouldn’t it be nice if there was one distributed systems kernel that you could build your software against? And then it doesn’t matter where your customers want to be, you run this thing first, it abstracts away all the hardware or cloud-specific stuff and you just run it and go,” Hindman said.

Rather than spend their engineering effort trying to build software assuming a specific API, like Amazon Web Services (AWS) or Google Compute Engine (GCE), developers can build against an open API from a project that’s in a third-party foundation, Hindman says. Mesos can then be built to run well on Microsoft Azure, AWS, Google Compute, Digital Ocean, or any other platform.

“You should be able to move from AWS to GCE without changing anything because you’re just running on Mesos,” Hindman said. “Your new level of abstraction is Mesos, not AWS.”