Kubernetes multi-tenant with Capsule

Kubernetes Capsule multi-tenant

6 min read | by Jordi Prats

Capsule implements a multi-tenant and policy-based environment in your Kubernetes cluster, leveraging only on upstream Kubernetes. It allows you to create tenants, namespaces, and users, and define policies to control the resources and access within the cluster.

Installing Capsule

To install Capsule, you can use the Helm chart provided by the project:

helm repo add projectcapsule https://projectcapsule.github.io/charts
helm install capsule projectcapsule/capsule -n capsule-system --create-namespace

Once installed you will only see one Deployment running in the capsule-system namespace:

$ kubectl get pods -n capsule-system
NAME                                          READY   STATUS    RESTARTS   AGE
capsule-controller-manager-685ddcc48b-7859r   1/1     Running   0          5s

Besides the controller manager, Capsule will a handful of Custom Resource Definitions (CRDs)`:

$ kubectl get crd
NAME                                       CREATED AT
capsuleconfigurations.capsule.clastix.io   2025-02-22T13:37:38Z
globalproxysettings.capsule.clastix.io     2025-02-22T14:24:40Z
globaltenantresources.capsule.clastix.io   2025-02-22T13:37:38Z
proxysettings.capsule.clastix.io           2025-02-22T14:24:40Z
tenantresources.capsule.clastix.io         2025-02-22T13:37:38Z
tenants.capsule.clastix.io                 2025-02-22T13:37:39Z

Authentication

In a multi-tenant environment, authentication is crucial. Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by --capsule-user-group option, which defaults to capsule.clastix.io.

Assignment to a group depends on the authentication strategy in your cluster. We can use OIDC or AWS IAM for example.

Create tenant

To create a tenant, we'll have to use the Tenant object. Here is an example of a Tenant object that defines a tenant named demo with one owner named jordi:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: demo
spec:
  owners:
  - name: jordi
    kind: User

We can create the tenant using the kubectl apply just as any other Kubernetes object:

$ kubectl apply -f demo-tenant.yaml
tenant.capsule.clastix.io/demo created
$ kubectl get tenant
NAME   STATE    NAMESPACE QUOTA   NAMESPACE COUNT   NODE SELECTOR   AGE
demo   Active                     0                                 10s

In the .spec section of the Tenant object, we can define the rules and limits for the tenant. We can control who owns the tenant, how many namespaces it can have, and what extra labels or settings they should get. We can also limit what container images are allowed, which storage types can be used, and how workloads are prioritized. This helps keep multi-tenant Kubernetes clusters organized, secure, and efficient. On the Tenant object documentation you can find all the available options.

Login as a tenant

Each tenant comes with a delegated user or group of users acting as the tenant admin (the tenant Owner). For this example, we'll use the hack/create-user.sh script provided by the Capsule project to create a kubeconfig for the user jordi within the demo tenant:

$ git clone https://github.com/projectcapsule/capsule.git
$ file capsule/hack/create-user.sh
capsule/hack/create-user.sh: Bourne-Again shell script text executable, ASCII text
$ bash capsule/hack/create-user.sh jordi demo
creating certs in TMPDIR /var/folders/hz/r62v__kj29jf__h_zmbbv0v40000gn/T/tmp.SkXrJvwxOZ
merging groups /O=projectcapsule.dev
certificatesigningrequest.certificates.k8s.io "jordi-demo" deleted
certificatesigningrequest.certificates.k8s.io/jordi-demo created
certificatesigningrequest.certificates.k8s.io/jordi-demo approved
kubeconfig file is: jordi-demo.kubeconfig
to use it as jordi export KUBECONFIG=jordi-demo.kubeconfig

If we using the generated kubeconfig file, we'll be able to create namespaces and deploy workloads within the demo tenant:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl create ns jordi-1st-ns
namespace/jordi-1st-ns created

Unlike other multi-tenant solutions, we are going to be able to create namespaces as usual. The only difference is that the namespace will be owned by the tenant:

$ kubectl get ns jordi-1st-ns -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2025-02-22T14:20:05Z"
  labels:
    capsule.clastix.io/tenant: demo
    kubernetes.io/metadata.name: jordi-1st-ns
  name: jordi-1st-ns
  ownerReferences:
  - apiVersion: capsule.clastix.io/v1beta2
    kind: Tenant
    name: demo
    uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f
  resourceVersion: "5268"
  uid: 3118715b-02fe-4dc4-91f8-19ebdde7dbb7
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Going back to the Tenant object, we can see that the NAMESPACE COUNT has been updated to 1:

$ kubectl get tenant
NAME   STATE    NAMESPACE QUOTA   NAMESPACE COUNT   NODE SELECTOR   AGE
demo   Active                     1                                 10m

Creating workfloads is going to be as usual as well:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo --image nginx -n jordi-1st-ns
pod/demo created
$ KUBECONFIG=jordi-demo.kubeconfig kubectl get pods -n jordi-1st-ns
NAME   READY   STATUS              RESTARTS   AGE
demo   0/1     ContainerCreating   0          3s

Listing namespaces

One of the problems we'll have as a tenant user is that we won't be able to list the cluster-scoped resources, such as namespaces:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns
Error from server (Forbidden): namespaces is forbidden: User "jordi" cannot list resource "namespaces" in API group "" at the cluster scope

To overcome this issue, we can use the Capsule Proxy add-on. We can install it using the Helm chart provided by the capsule project:

helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system

Once installed, we'll need to use that service to access the cluster-scoped resources:

$ kubectl get service capsule-proxy -n capsule-system
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
capsule-proxy   ClusterIP   10.96.179.132   <none>        9001/TCP   46s

On a production environment we would need to configure and expose the service properly, but for testing purposes we can use port-forward and ignore the certificate validation:

kubectl -n capsule-system port-forward service/capsule-proxy 9001:9001

Once the proxy is running, we can update the kubeconfig file. We need to:

  • Update the server to use localhost:9001
  • Remove certificate-authority-data from the cluster entry
  • Add insecure-skip-tls-verify: true to skip the certificate validation

The resulting kubeconfig file should look like this:

apiVersion: v1
clusters:
- cluster:
    server: https://127.0.0.1:9001
    insecure-skip-tls-verify: true
  name: kind-capsule
contexts:
- context:
    cluster: kind-capsule
    user: jordi
  name: jordi-demo
current-context: jordi-demo
kind: Config
preferences: {}
users:
- name: jordi
  user:
    client-certificate: jordi-demo.crt
    client-key: jordi-demo.key

Once updated, we can use it to list the namespaces, in this case we'll be able to see the list of namespaces owned by the demo tenant:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns
NAME           STATUS   AGE
jordi-1st-ns   Active   28m

Posted on 25/02/2025

Categories