6 min read | by Jordi Prats
Capsule implements a multi-tenant and policy-based environment in your Kubernetes cluster, leveraging only on upstream Kubernetes. It allows you to create tenants, namespaces, and users, and define policies to control the resources and access within the cluster.
To install Capsule, you can use the Helm chart provided by the project:
helm repo add projectcapsule https://projectcapsule.github.io/charts
helm install capsule projectcapsule/capsule -n capsule-system --create-namespace
Once installed you will only see one Deployment running in the capsule-system
namespace:
$ kubectl get pods -n capsule-system
NAME READY STATUS RESTARTS AGE
capsule-controller-manager-685ddcc48b-7859r 1/1 Running 0 5s
Besides the controller manager, Capsule will a handful of Custom Resource Definitions (CRDs)`:
$ kubectl get crd
NAME CREATED AT
capsuleconfigurations.capsule.clastix.io 2025-02-22T13:37:38Z
globalproxysettings.capsule.clastix.io 2025-02-22T14:24:40Z
globaltenantresources.capsule.clastix.io 2025-02-22T13:37:38Z
proxysettings.capsule.clastix.io 2025-02-22T14:24:40Z
tenantresources.capsule.clastix.io 2025-02-22T13:37:38Z
tenants.capsule.clastix.io 2025-02-22T13:37:39Z
In a multi-tenant environment, authentication is crucial. Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by --capsule-user-group
option, which defaults to capsule.clastix.io
.
Assignment to a group depends on the authentication strategy in your cluster. We can use OIDC or AWS IAM for example.
To create a tenant, we'll have to use the Tenant
object. Here is an example of a Tenant
object that defines a tenant named demo
with one owner named jordi
:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: demo
spec:
owners:
- name: jordi
kind: User
We can create the tenant using the kubectl apply
just as any other Kubernetes object:
$ kubectl apply -f demo-tenant.yaml
tenant.capsule.clastix.io/demo created
$ kubectl get tenant
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
demo Active 0 10s
In the .spec
section of the Tenant
object, we can define the rules and limits for the tenant. We can control who owns the tenant, how many namespaces it can have, and what extra labels or settings they should get. We can also limit what container images are allowed, which storage types can be used, and how workloads are prioritized. This helps keep multi-tenant Kubernetes clusters organized, secure, and efficient. On the Tenant object documentation you can find all the available options.
Each tenant comes with a delegated user or group of users acting as the tenant admin (the tenant Owner). For this example, we'll use the hack/create-user.sh
script provided by the Capsule project to create a kubeconfig
for the user jordi
within the demo
tenant:
$ git clone https://github.com/projectcapsule/capsule.git
$ file capsule/hack/create-user.sh
capsule/hack/create-user.sh: Bourne-Again shell script text executable, ASCII text
$ bash capsule/hack/create-user.sh jordi demo
creating certs in TMPDIR /var/folders/hz/r62v__kj29jf__h_zmbbv0v40000gn/T/tmp.SkXrJvwxOZ
merging groups /O=projectcapsule.dev
certificatesigningrequest.certificates.k8s.io "jordi-demo" deleted
certificatesigningrequest.certificates.k8s.io/jordi-demo created
certificatesigningrequest.certificates.k8s.io/jordi-demo approved
kubeconfig file is: jordi-demo.kubeconfig
to use it as jordi export KUBECONFIG=jordi-demo.kubeconfig
If we using the generated kubeconfig
file, we'll be able to create namespaces and deploy workloads within the demo
tenant:
$ KUBECONFIG=jordi-demo.kubeconfig kubectl create ns jordi-1st-ns
namespace/jordi-1st-ns created
Unlike other multi-tenant solutions, we are going to be able to create namespaces as usual. The only difference is that the namespace will be owned by the tenant:
$ kubectl get ns jordi-1st-ns -o yaml
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2025-02-22T14:20:05Z"
labels:
capsule.clastix.io/tenant: demo
kubernetes.io/metadata.name: jordi-1st-ns
name: jordi-1st-ns
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
name: demo
uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f
resourceVersion: "5268"
uid: 3118715b-02fe-4dc4-91f8-19ebdde7dbb7
spec:
finalizers:
- kubernetes
status:
phase: Active
Going back to the Tenant
object, we can see that the NAMESPACE COUNT
has been updated to 1
:
$ kubectl get tenant
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
demo Active 1 10m
Creating workfloads is going to be as usual as well:
$ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo --image nginx -n jordi-1st-ns
pod/demo created
$ KUBECONFIG=jordi-demo.kubeconfig kubectl get pods -n jordi-1st-ns
NAME READY STATUS RESTARTS AGE
demo 0/1 ContainerCreating 0 3s
One of the problems we'll have as a tenant user is that we won't be able to list the cluster-scoped resources, such as namespaces:
$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns
Error from server (Forbidden): namespaces is forbidden: User "jordi" cannot list resource "namespaces" in API group "" at the cluster scope
To overcome this issue, we can use the Capsule Proxy add-on. We can install it using the Helm chart provided by the capsule project:
helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system
Once installed, we'll need to use that service to access the cluster-scoped resources:
$ kubectl get service capsule-proxy -n capsule-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
capsule-proxy ClusterIP 10.96.179.132 <none> 9001/TCP 46s
On a production environment we would need to configure and expose the service properly, but for testing purposes we can use port-forward
and ignore the certificate validation:
kubectl -n capsule-system port-forward service/capsule-proxy 9001:9001
Once the proxy is running, we can update the kubeconfig
file. We need to:
server
to use localhost:9001
certificate-authority-data
from the cluster
entryinsecure-skip-tls-verify: true
to skip the certificate validationThe resulting kubeconfig
file should look like this:
apiVersion: v1
clusters:
- cluster:
server: https://127.0.0.1:9001
insecure-skip-tls-verify: true
name: kind-capsule
contexts:
- context:
cluster: kind-capsule
user: jordi
name: jordi-demo
current-context: jordi-demo
kind: Config
preferences: {}
users:
- name: jordi
user:
client-certificate: jordi-demo.crt
client-key: jordi-demo.key
Once updated, we can use it to list the namespaces, in this case we'll be able to see the list of namespaces owned by the demo
tenant:
$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns
NAME STATUS AGE
jordi-1st-ns Active 28m
Posted on 25/02/2025