6 min read

Adventures in Kubernetes (Part 3)

Featured Image

Making the Least Of Kubernetes

Why it's ok to start with Kubernetes even if you can't gain all the benefits.

You've looked at the map, you know where you are, and where you want to go. So why Kubernetes?

Kubernetes (K8S) offers a lot of great benefits:

  1. Safely pack multiple services into less hardware.
  2. Service discoverability using internal DNS.
  3. Infrastructure as code with "simple" YAML files.
  4. Automatic scaling of both pods and clusters to handle increases and decreases in load.
  5. Isolated namespaces for teams or using production for testing.
  6. Deploying multiple microservices together as a pod.
  7. Offloading common tools like log scrappers or network rules in a language agnostic way.
  8. Polyglot systems which communicate over the network.
  9. Easier operations and maintenance when using a cloud service.

As we wrote about in our first article in this series, that's quite a lot of hype... how much of it is true? How much of it is even needed? In honesty, this isn't even the full list of potential benefits.

At OnceHub, we started with Kubernetes hoping to mostly take advantage of benefits 1 and 4. We wanted our services to scale more and cost less. Everything else was just icing on the cake. As we wrote in part 1, auto scaling for us wasn't the great benefit we thought it would be, however, the ability to safely pack multiple services into fewer VMs was a huge cost savings for us. Just that one feature was enough of a benefit for the cost of the complexity of learning Kubernetes. Focusing on that one benefit also allowed us to start small with our YAML files, keeping them simple, and allowing them to grow in complexity as we learned about Kubernetes and could take advantage of more features.

Remember, that every technology choice comes with tradeoffs. Nothing is for free. So let's look at each of these benefits and see which ones are worth the cost of learning a whole new ecosystem and system of working.

  1. Safely pack multiple services into less hardware.
    For us, at OnceHub this was a major benefit. However, in hindsight we could have gotten this benefit alone by just using Docker and perhaps DockerSwarm for the deployments. Is DockerSwarm better for this simple use case than Kubernetes? Perhaps, it depends what you already know and what you need to learn. However, DockerSwarm isn't as good in other areas, so we would suggest taking the plunge, even if this is the only benefit you need.
  2. Service discoverability using internal DNS.
    If this is the only benefit you are looking for, then you already have lots of microservices, and you seem to be doing just fine. It turns out that this is a huge benefit and one of the better design decisions that the Kubernetes team made. So if you are just starting to look towards the future, definitely learn from this mechanism and take advantage of it, but I don't think this feature alone will be worth learning the complexity of creating YAML manifests.
  3. Infrastructure as code with "simple" YAML files.
    If you don't have experienced operations members on your team, this can be a huge benefit. Compared to learning how to run and maintain VMs, Load balancers, network gateways, firewalls etc., allowing Kubernetes to do it's magic is a huge time saver. You'll have the ability to learn all these things in detail as you gain more experience. However, if you already have an experienced operations team, the YAML manifests are anything but simple. Sure, they look simple, and there are an increasing number of tools making the creation of K8S manifests easier, but there is a lot of complexity here. The idea of "simple to use YAML" is more hype than reality. But in most cases, the benefits outweigh the costs. Only aim for this benefit if your team is lacking operations experts.
  4. Automatic scaling of both pods and clusters to handle increases and decreases in load.
    As we wrote about in the first part of this series, this benefit is a bust. Automatic scaling will help and does work, but only after you are able to learn a lot from your system through trial and error and a deep and complex understanding of how the K8S scheduling engine works. This is where most people think K8S will help, and I believe is also the source of most horror stories. Become more familiar with Kubernetes before you start promising scaling nirvana.
  5. Isolated namespaces for teams or using production for testing.
    Isolated namespaces for teams is a benefit we have read about but never experienced ourselves. Those who use them, swear by them. But this is more related to the company culture and business requirements than to any technical feature of Kubernetes. We've included it here in our list because cultural benefits are real benefits and should not be ignored. We'll write more about the importance of culture in part four of our series.
  6. Deploying multiple microservices together as a pod.
    When we started, we thought we could group services that often call each other into a single pod and make use of the shared disk space for improvements in performance. In practice the complexity of making that work and scale was never worth the theoretical improvement. Perhaps we'll revisit this again some day, but for now, multiple services in a single pod is best reserved for something called "side cars", which is addressed in the next point. If this is the sole reason for using K8S, we'd recommend a simpler solution than Kubernetes.
  7. Offloading common tools like log scrappers or network rules in a language agnostic way.
    One of the biggest issues with a microservice architecture is all of the cross cutting concerns, like logging, or networking policy rules, or authentication which you A. want performant and B. don't want to develop separately for each service. Kubernetes solves this really nicely by enabling the "sidecar pattern". Sidecars are containers which share resources with a microservice but are written separately and possibly in a different language. This is a hard problem to solve if you aren't using Kubernetes, and K8S makes it relatively simple. Be warned, it's not as easy to pull off correctly as it looks.
  8. Polyglot systems which communicate over the network.
    One of the less obvious benefits of using Kubernetes is that you have access to a larger range of tools and software which you can insert into your system. Since all services communicate over the network, and Kubernetes makes it easy for them to communicate with each other, you aren't limited to using open source tools designed for your specific ecosystem. Did the Go community just create a better JWT token generator? Did the Java community create a better queue? Does your team want to experiment with Rust? When you are working with a cross-network set of services, communicating between them becomes easier, and Kubernetes can help you truly find the best tool for the job. However, if your system isn't designed for cross network communication, this is of no benefit to you.
  9. Easier operations and maintenance when using a cloud service.
    This last benefit can be confusing talking across each other from different perspectives. If you use Kubernetes like most people do, in a cloud hosted environment, it makes operations and maintenance significantly easier. However, there is also a very large minority of people using Kubernetes in a "bare metal" or "hybrid" environment. For these people, Kubernetes can be overly complicated, delicate, and not worth the time or investment to get it working. OnceHub is cloud native, so we can't say much about the bare metal or hybrid settings, other than the fact that bare metal and hybrid opens up many exciting opportunities, with edge computing. But if you are a smaller shop, working in containers and Kubernetes can save many hours of operation and maintenance work, and might be a good enough benefit even if you don't get any of the rest.

The biggest thing we learned from our Adventure with Kubernetes is that even if the majority of features don't apply to you, or are just hype… sometimes that one critical feature is enough to make the complexity of learning a new ecosystem "good enough" and practical enough. You can gain a lot, when getting the least out of the "OS of the cloud". On the other hand, sometimes the least isn't enough, and the complexity of learning the declarative YAML programming language, and the K8S APIs, are enough of a headache to keep on using the older, proven systems.

In our next adventure, we will talk about the cultural expectations for teams and companies using Kubernetes, and why you can still benefit, even if you don't fit the profile.


Avi Kessner, Software Architect

Avi started as a Flash Animator in 1999, moving into programming as Action Script evolved. He’s worked as a Software Architect at OnceHub since September 2019, where he focuses on Cloud Native Automated workflows and improving team practices. In his free time, he enjoys playing games, Dungeons & Dragons, Brazilian jujitsu, and historical European sword fighting.