Deploying and managing complex stateful applications on Kubernetes.

Kubernetes (aka K8s) is now the de facto container orchestration framework. Like other popular open-source technologies, Kubernetes has amassed a considerable ecosystem of complementary tools to address everything from storage to security. And although it was first created for running stateless applications, more and more organizations are interested in using Kubernetes for stateful applications.

However, while Kubernetes has advanced significantly in many areas during the past couple years, there still are considerable gaps when it comes to running complex stateful applications. It remains challenging to deploy and manage distributed stateful applications consisting of a multitude of co-operating services (such as for use cases with large-scale analytics and machine learning) with Kubernetes.

I've been focused on this space for the past several years as a co-founder of BlueData. During that time, I've worked with many teams at Global 2000 enterprises in several industries to deploy distributed stateful services successfully, such as Hadoop, Spark, Kafka, Cassandra, TensorFlow and other analytics, data science, machine learning (ML) and deep learning (DL) tools in containerized environments.

In that time, I've learned what it takes to deploy complex stateful applications like these with containers while ensuring enterprise-grade security, reliability and performance. Together with my colleagues at BlueData, we've broken new ground in using Docker containers for big data analytics, data science and ML/DL in highly distributed environments. We've developed new innovations to address requirements in areas like storage, security, networking, performance and lifecycle management.

Now we want to bring those innovations to the Open Source community—to ensure that these stateful services are supported in the Kubernetes ecosystem. BlueData's engineering team has been busy working with Kubernetes, developing prototypes with Kubernetes in our labs and collaborating with multiple enterprise organizations to evaluate the opportunities (and challenges) in using Kubernetes for complex stateful applications.

To that end, we recently introduced a new Kubernetes open-source initiative: BlueK8s. The BlueK8s initiative will be composed of several open-source projects that each will bring enterprise-level capabilities for stateful applications to Kubernetes.