kcp: Kubernetes-like control plane
Disclaimer: The paradigm of Cloud Native Simplified is explained here. This article doesn’t aim to explain everything about kcp in detail. We are going to talk about it in only 101 level.
kcp is an open-source project that tries to solve some problems of Kubernetes when you have multi-tenants and/or multi-clusters. Before jumping into the problems, let’s take a high-level look at the architecture.
kcp serves an API that is compatible with the Kubernetes API. You can run your kubectl commands against that API.
kcp also contains some controllers and schedulers. That is why the project calls itself “Kubernetes-like control plane” instead of “Kubernetes-like api-server”.
Now the core functionality is provided by only one binary (that also starts an etcd if necessary) but it may be decoupled in time. There are some other helper components in the project for some use cases.
kcp API provides basic Kubernetes types like ConfigMap, Secret, RBAC types, etc.
kcp API doesn’t provide the types for container orchestration like Node, Pod, Deployment, etc. by default.
kcp support CRDs.
kcp API provides additional built-in types to solve some problems. Multi-tenancy and multi-clusters are the major problems.
Problem 1: Multi-tenancy
Everybody knows multi-tenancy is hard in Kubernetes.
For standard tenants who run standard workloads like Deployments, it is not easy but doable. You need to use RBAC objects, Namespaces, NetworkPolicies, ResourceQuotas, PodSecurityPolicies, etc to limit&isolate the tenants.
When it comes to advanced tenants who want to run their own operators with their own CRDs, it is huge pain. Kubernetes has no mechanism to isolate tenants for CRDs. Many people use a separate cluster per tenant in this case.
Solution 1: Workspaces
kcp implements multi-tenancy through Workspaces.
Workspaces are like namespaces but provide full isolation. It is like a create a new fresh cluster when you create a workspace and switch your kubeconfig file.
kcp serves a logical cluster per workspace behind the scene. It differentiates workspaces with subdirectories in the api-server URL.
With kubectl-ws plugin, it is easy to create&use workspaces in kcp.
Here is a short demo to show workspace creation and the isolation between workspaces.
➜ kubectl ws current Current workspace is "root". ➜ kubectl apply -f ./cert-manager.crds.yaml ➜ kubectl get crd NAME CREATED AT certificaterequests.cert-manager.io 2022-11-03T18:52:52Z certificates.cert-manager.io 2022-11-03T18:52:52Z challenges.acme.cert-manager.io 2022-11-03T18:52:52Z clusterissuers.cert-manager.io 2022-11-03T18:52:52Z issuers.cert-manager.io 2022-11-03T18:52:52Z orders.acme.cert-manager.io 2022-11-03T18:52:52Z ➜ kubectl ws create blog-example Workspace "blog-example" (type root:organization) created. ➜ kubectl get ws NAME URL blog-example https://192.168.0.18:6443/clusters/root:blog-example users https://192.168.0.18:6443/clusters/root:users ➜ kubectl ws use blog-example Current workspace is "root:blog-example" (type "root:organization"). ➜ kubectl get crd No resources found ➜ kubectl ws .. Current workspace is "root". ➜ kubectl get crd NAME CREATED AT certificaterequests.cert-manager.io 2022-11-03T18:52:52Z certificates.cert-manager.io 2022-11-03T18:52:52Z challenges.acme.cert-manager.io 2022-11-03T18:52:52Z clusterissuers.cert-manager.io 2022-11-03T18:52:52Z issuers.cert-manager.io 2022-11-03T18:52:52Z orders.acme.cert-manager.io 2022-11-03T18:52:52Z
Problem 2: Multi-clusters
People need to create multiple clusters instead of one big cluster because of some reasons such as isolation, k8s limits, multi-region/multi-cloud setups, etc. SIG Multicluster is one of the special interest groups in Kubernetes community and they are trying to build some solutions for this problem like KubeFed. However, it is still huge pain for users when they try to use multiple clusters as a single compute service.
kcp has some solutions for the multi-cluster problem.
Solution 2: Syncer and Scheduler
SyncTarget is a Custom Resource of kcp. Each sync target points out a real physical Kubernetes cluster.
Location is a Custom Resource of kcp. Thanks to this CR, admins can group SyncTargets (which means physical clusters) and serve them to end users as a unit.
In each physical cluster, there is a syncer. It is responsible to watch kcp, to manage resources in the physical cluster and to report their status back to kcp.
Placement is a Custom Resource of kcp. Thanks to this CR, end users can state desired workspace/workload - location assignments. Physical clusters and SyncTargets are invisible to end users at this point.
apiVersion: scheduling.kcp.dev/v1alpha1 kind: Placement metadata: name: aws spec: locationSelectors: - matchLabels: cloud: aws namespaceSelector: matchLabels: app: foo locationWorkspace: root:default:location-ws
In kcp, there are some internal schedulers that assign&distribute workloads to the real physical clusters. The ultimate goal is to be able to use multiple clusters as just compute resources like there is a single big cluster.
kcp supports advanced deployment strategies for scenarios such as affinity/anti-affinity, geographic replication, cross-cloud replication, etc.
If you like the 101 approach in this issue, subscribe to get further ones! (Spoiler:Confidential Computing)