For the past year you probably read a lot about Docker container, micro services, big data, data mining and machine learning. The problems start when you try to introduce to any of these concepts and end up stuck on how to orchestrate all the different services you now use in your stack. This is a common problem, it's definitely not easy to manage all the servers and connect all your services in a way that can be scaled when needed.
A lof of different frameworks and tools are proposing different ways to solve these problems. Mesosphere just released DCOS (short for Data Center Operating System) to tackle this problems and more.
DCOS aims to leverage many open source frameworks such as Mesos, Marathon and Chronos, providing an easy way to deploy, config and maintain your cluster being it a small two server with a few docker containers to a mega cluster for thousands of server and many different softwares and services running concurrently.
Take a moment to view it in action:
Mesosphere's proposed solution enables you to deploy all your services in the same cluster while DCOS takes care of running them for you. They do this by abstracting all the servers resources into a pool of CPU, memory and disk while offering available resources to schedulers running tasks.
Centralize data center operations in a single command line and a single web user interface.
Deploy and scale services in seconds being it Docker containers, big Hadoop or Spark jobs, and even big distributed databases like Cassandra or Elasticsearch.
Run in any cloud provider (AWS, Google Cloud Platform and Azure), virtual machine or bare metal, or mix them to improve high availability and fault tolerance.
Auto scales the cluster.
Greatly increase installed capacity usage rate. By running all services concurrently on the same cluster, DCOS can really make use of capacity available. It is easy to get to 50%+ usage rate and not difficult to think of way to take to even greater numbers. Let's say we can easily plug in office client machines into a local DCOS cluster during night time when no one is using them to run big processing jobs and make use of all that capacity that has been sitting still every night.
Integrated service discovery with mesos-dns.
The future of DCOS?
The single best feature of Mesosphere DCOS is that is built on top of very well known open source projects actively used in production environments at Twitter, Airbnb, Apple among many other big players. You can get the full picture at Powered By Mesos.
This creates a great environment where many different solutions appear to tackle each problem that needs solving and many frameworks and services are being actively ported to work with mesos and DCOS.
I am looking forward for the day I can manage my MongoDB cluster within Mesos with DCOS, and I believe it will be sooner than one would expect.
What am I doing with DCOS?
At the moment I am exploring ideas and new possibilities. I believed DCOS was game changing the moment I started having ideas on how now I can improve my project by easily dividing them in very small microservices, add new functionality and control such as ELK to centralize logging and analytics of all my services, and one of my favorites, using Chronos to create and run many small jobs for automation. The possibilities are endless.
At the moment I am already running my CI/CD flow with Jenkins configured to run all Jenkins Slaves on a Mesos cluster.
I plan in the near future to also finally get a bit into Spark for even better analytics and machine learning to further improve my project and deliver better results to my customers.
Where to go from here?
You can find a lot of material online on all the subjects we mentioned here, I will just list a few good starting points: