Adventures in Kubernetes (Part 5)

Cultivating Culture

As we start to wind up our series, it's important to reflect on our adventure. Sure, we weren't the ideal team that one would expect to see adopting Kubernetes early, but we grew into it. We still aren't the textbook DORA elite team, able to deploy after every commit, but we are much closer to that goal than we thought was possible, and using Kubernetes is helping us get there.

Let's look back at where we started. We had large teams, working on features based on long detailed documents. We would deploy sporadically, with big bang releases. Our version updates at that time came with emails warning customers of expected downtime.

Initially, when we created our first microservices, we ran them on their own VMs, with pm2. Then we looked into moving them into containers and Docker Compose. These were important first steps to help our organization learn about the intricacies of containerized applications, and take safe, reversible learning steps along our journey. While we were experimenting with Docker Swarm and alternatives, we learned that Docker Enterprise was using Kubernetes, and not just Docker Swarm. This gave us the confidence and courage to get over the learning hump of Kubernetes, and its strange jargon of pods and services that didn't actually refer to the instances of our microservices.

Before we continue, it must be said, that much of the transformation described in this blog would have happened if we had chosen Docker Swarm or Mesos instead of Kubernetes. It's not so much the exact tool that matters here, but rather the move to microservices, and the complex orchestration requirements which that entails.

We introduced Kubernetes with just a small collection of three microservices, mostly just for logging and routing traffic to our .net monolith. A safe, small scale test that let us monitor traffic and the robustness of Kubernetes routing in a reversible way. Things went well, but with Kubernetes comes a cost of mental complexity. Developers needed to start learning YAML, new jargon, and how to get many containers working on their local setup. Deployments were easier but configurations became more complex. It became clear that if we kept going down this route, a developer's laptop would not be powerful enough to host all the containers. But these were challenges to overcome rather than obstacles that blocked us.

We learned more about DevOps, and why it's not a role but a culture, and assigned one or two people from each team to be responsible for learning Kubernetes, and keeping their team's dev environment up and running. After initial enthusiasm, reality kicked in and a dedicated DevOps team was created, to keep our Jenkins pipelines and Kubernetes clusters functional. By this time we had grown our clusters from three small services to about ten services, and routing all of our traffic through Kubernetes.

Another interesting thing that happened during this time, was a parallel shift in the way we worked. Our large documents grew smaller, and we began to question the need for downtime during deployments. Kubernetes documentation made it clear that we can roll out our services with zero downtime! We had some cultural and structural issues blocking us, but through experimentation with Kubernetes we saw that it was indeed possible to do, at least theoretically. And thus we gained a great insight, that Kubernetes was actually helping to reinforce the goal of a cultural shift. We didn't need a proper culture to make use of Kubernetes, Kubernetes made developing a better culture easier!

Today, we have regular deployments. Our deployments are separated from our releases with feature flags, and we do this with almost zero downtime deployments. Our teams have also gotten smaller and more independent, and the time between a new feature being thought about and it making it to production has been reduced. Could we have made this transition without Kubernetes? Probably. Would we have had the courage and trust in our processes to adapt so quickly? Unlikely.

This blog post is a summary of about three years of cultural transformation, and thus is greatly oversimplified. Other key ingredients for us to make this transition was adopting SAFe, having support from every department, top to bottom, and implementing ever-evolving, complex, automated CI pipelines with our dedicated DevOps and QA teams. However, Kubernetes and the ecosystem surrounding it, really showed us how to get all those pieces working together. Another large benefit from K8s was how much just worked out of the box which was able to fill in the gaps of our pipelines until we could build something more mature and custom for ourselves. We even still have some of the original bash scripts which clobbers our helm files together and lets us reduce some of the YAML required.

Because Kubernetes brought in the right kind of complexity, we could no longer ignore the problems in our processes. Because the problems were complex, the only reasonable solution was to increase automation. Because we increased automation for delivery, we increased automation for testing. Because we increased automation for testing, we improved code quality, and increased the rate of deployments, making our customers happier.

So how does this help you? It's not just enough to know that you can get started with an imperfect culture, but it's also important to realize that you can help cultivate the culture you want by slowly introducing the tools and processes which make that desired culture the most obvious one to work towards. Map out the culture you want to have and find the tools that will help others in your organization have the aha moments, which will get them asking for that culture as well.

Avi Kessner, Software Architect

Avi started as a Flash Animator in 1999, moving into programming as Action Script evolved. He’s worked as a Software Architect at OnceHub since September 2019, where he focuses on Cloud Native Automated workflows and improving team practices. In his free time, he enjoys playing games, Dungeons & Dragons, Brazilian jujitsu, and historical European sword fighting.