Crossplane: Unifying platform engineering based on Kubernetes API

A framework for building cloud native control planes

We have been talking about how cloud-native software must be for over a decade. Nowadays, the requirements are quite long and to build a platform for cloud-native apps is pretty challenging. When we look closer at how platform engineering is being performed, we will see that tools, environments, and practices are pretty diverse. Platform teams provision infrastructures on different cloud providers or on-premise. Kubernetes is the de facto container orchestration tool but there are lots of tools in the CNCF ecosystem around it. Every cloud provider, service and tool comes with its own APIs, which makes the job of platform teams much harder.

Crossplane is a CNCF project that aims to unify all APIs in platform engineering based on Kubernetes API by leveraging Kubernetes controllers.

Before digging deeper, let’s take a step back and talk about k8s controllers.

Controllers: The power of Kubernetes

Kubernetes controllers are non-terminating loops that watch the system and take necessary actions to close the gap between the actual state and the desired state. Whenever there is a change in the actual state, controllers take necessary actions again, which is called “reconciliation loop”. This is a very powerful mechanism to implement resilient systems so that users can sit back with peace of mind after defining the desired state.

Many say Kubernetes architecture is so complex but it is not. etcd is for storage. api-server is the common gate to access the source of truth. And there are a bunch of controllers around the api-server. That is all. Controllers are the power of Kubernetes.

Operators: Extending Kubernetes

When people noticed the controller pattern in Kubernetes is so powerful, they started to implement their custom controllers by using CustomResourceDefinitions, which led to the birth of Kubernetes Operators.

Basically, A Kubernetes operator is a replacement for a human operator who knows how to manage a service. For example, Prometheus Operator can manage Prometheus on top of Kubernetes like a human operator. It can install, configure, upgrade, uninstall, etc. All a user needs is only to create a Prometheus CustomResource(CR) to tell the desired state.

Then people noticed Kubernetes controllers are useful for not only managing resources in Kubernetes but also external resources. Imagine an operator that can create a VPC in AWS when you create a VPC CR in Kubernetes. Doesn’t it sound handy?

Crossplane: Controllers for everything

The idea behind Crossplane is

Let’s use Kubernetes controllers to manage everything including cloud resources.

To be able to understand upcoming sections, let’s learn Crossplane terminology a little bit.

• ManagedResources: Similar to Custom Resources. The idea is that it is not just a CR, it represents a real resource somewhere like VPC or Bucket. Crossplane is more opinionated about CRDs since it provides some functionalities by using common fields in spec&status of managed resources.

• Provider: Kubernetes operators which are compatible with Crossplane by providing necessary metadata for Crossplane and by obeying patterns defined by Crossplane. For example, provider-aws, provider-gcp, provider-argocd, provider-gitlab etc.

• Packages: OCI images that contain metadata files for Crossplane including CRDs.

The most simplified flow of usage of Crossplane is as below. I used provider-aws as an example.

Crossplane manages the lifecycle of providers and the dependencies between them by using metadata in the packages. It has also compatibility checks.

Crossplane has a dream as below. Whenever you need to create something in a cloud or in a service or in k8s, all you need is to tell Crossplane the provider and to create k8s objects in your cluster.

Composites: From Bricks to Platforms

We are able to create anything in anywhere by just creating Kubernetes objects with Crossplane in theory. This is amazing! Let’s complicate the problem a little bit.

Let’s assume you are a platform engineer and you need to create 10 managed resources to provide a service to app teams. There should be a configuration interface so that app teams can provide some parameters to your service. How can you do it?

You can create a helm chart and bundle the 10 managed resources. App teams can install the chart with their parameters. Does it sound good? Yeah. But helm doesn’t have a reconciliation mechanism. When there is a problem in current state, it is not going to fix.

You can define your own CRD and implement your own operator to benefit from controller pattern but it may be costly.

Or you can use CompositeResources(XRs) of Crossplane.

By using CompositeResourceDefinition, you can define your own type. It is like CRD but more opinionated. In the CompositeResourceDefinition, you should define the configuration interface for your CompositeResource by using OpenAPIV3Schema.

And then you can create Composition as below. It contains which managed resources will be created and how they will be rendered with the composite parameters. With the same CompositeResourceDefinition, you can write multiple composition for different scenarios.

And then app team can claim a composition by providing the parameters and optionally picking a composition when there are many.

The flow is shortly

Claim → CompositeResource → Multiple ManagedResources

Let’s emphasize some points in this design

• Controller pattern is used everywhere so every unwanted change in actual state will be fixed by reconciliation loops.

• Platforms can be implemented completely in declarative way without writing imperative code like custom controllers.

• Compositions draw a line between platform teams and app teams. Platform teams can define&use them whereas app teams can only use.

• The Claim-CompositeResource model is optional but it provides some features to platform teams for different use cases. Like being able to provision CompositeResources in advance.

Closing Notes With My Personal Opinions

• Kubernetes is becoming the common API for platform engineering so Crossplane has a bright future.

• Crossplane core does two major things: lifecycle management of providers and composition. These didn’t seem big enough for me at first glance. My expectation was to encounter harder problems solved before digging deeper. On the other hand, the cloud providers in crossplane-contrib organization are loved so much by the community. I guess Crossplane gained huge amount of popularity thanks to them. Also, there are many mechanisms in the defined specs to reuse commons between ManagedResources such as provider configuration.

• The benefits Crossplane provides are directly proportional to the number of providers in the ecosystem. That is why Upbound (the company behind Crossplane) introduced Upjet, which is a tool to generate Crossplane Providers from any Terraform Provider. Upbound generated cloud-providers again with this tool and they are available in the Upbound marketplace. I hope we will see more community provider generated with Upjet soon.