How to configure an application to use SecurityGroups for Pods

SecurityGroups for Pods AWS EKS

4 min read | by Jordi Prats

To be able to configure SecurityGroups we will have:

This is what needs to be done considering that we are going to use an ALB to direct traffic to the application.

Pod's SecurityGroup

Since the Pod is going to be using a set of SecurityGroups we need to make sure the cluster can also reach it:

We will have to allow the workers to communicate with the Pods to be able to check it's health via probes. We can use the following terraform code to allow this traffic:

resource "aws_security_group" "demo_pod_sg" {
  name        = "demo-pod-sg"
  description = "demo sg4pods"
  vpc_id      = var.vpc_id

  lifecycle { create_before_destroy = true }
}

resource "aws_security_group_rule" "cluster_workers" {
  description              = "allow inbound communication from the cluster security group"
  security_group_id        = aws_security_group.demo_pod_sg.id
  type                     = "ingress"

  from_port                = 0
  to_port                  = 0
  protocol                 = "-1"

  source_security_group_id = var.workers_sg
}

AWS LoadBalancer Ingress

If we are using a AWS LoadBalancer controller the ingress objects without the alb.ingress.kubernetes.io/security-groups will add all the required rules, but if we are setting a set of SecurityGroups with this annotations, we will have to make sure that the Pod's SecurityGroup has a rule to allow incomming connections from the ALB suing the first SecurityGroup set on the annotation:

resource "aws_security_group_rule" "pod_alb" {
  description              = "ALB access to demo Pods"
  security_group_id        = aws_security_group.demo_pod_sg.id
  type                     = "ingress"

  source_security_group_id = aws_security_group.external_ip_ranges_sg.id
  from_port                = 0
  to_port                  = 0
  protocol                 = "tcp"
}

SecurityGroupPolicy object

To assign a SecurityGroup to a Pod we are going to use a SecurityGroupPolicy object (CRD). This object use a familiar concept to select to which pods it is going to apply the SecurityGroups it has configured: a podSelector or a serviceAccountSelector

For both we can select some specific Pods or ServiceAccounts or simply set it to empty (podSelector: {} or serviceAccountSelector: {}) to select all the Pods/ServiceAccounts in the namespace.

To set the SecurityGroup we can define the list we want to set using spec.securityGroups.groupIds

On the following example we are selecting all the pods with the label app=demo to assign the SecurityGroup sg-1234567890abcdef0 

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: demo-sg4pod-policy
spec:
  podSelector:
    matchLabels:
      app: demo
  securityGroups:
    groupIds:
      - sg-1234567890abcdef0

The same example selecting all the pods on the namespace would look like this:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: demo-sg4pod-policy
spec:
  podSelector: {}
  securityGroups:
    groupIds:
      - sg-1234567890abcdef0

This object needs to be in place before any Pod that matches is created. If we are using helm we can use a pre-install/pre-upgrade hook to accomplish this:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: {{ include "demosg4pods.fullname" . }}
  annotations:
    "helm.sh/hook": "pre-install,pre-upgrade"
    "helm.sh/hook-delete-policy": "before-hook-creation"
    "helm.sh/hook-weight": "0"
spec:
  podSelector:
    matchLabels:
      app: demo
  securityGroups:
    groupIds:
      - sg-1234567890abcdef0

To check what policies we have we can use kubectl get SecurityGroupPolicy:

$ kubectl get SecurityGroupPolicy
NAME                 SECURITY-GROUP-IDS
demo-sg4pod-policy   ["sg-1234567890abcdef0"]

Once the Pods have been created we can check it's events to check whether the Security Groups have been applied:

$ kubectl describe demo-sg4pods
(...)
Normal   SecurityGroupRequested  62s                vpc-resource-controller  Pod will get the following Security Groups [sg-1234567890abcdef0]
Normal   ResourceAllocated       62s                vpc-resource-controller  Allocated [{"eniId":"eni-0610111213ff00112","ifAddress":"06:10:11:12:13:ff","privateIp":"10.12.30.28","vlanId":2,"subnetCidr":"10.12.16.0/23"}] to the pod

It will also add an annotation with the eni it have been assigned:

$ kubectl get pod demo-sg4pods -o yaml
apiVersion: networking.k8s.io/v1
kind: Pod
metadata:
  annotations:
  vpc.amazonaws.com/pod-eni: [{"eniId":"eni-0a8989d8f82891d05","ifAddress":"06:aa:cc:a6:2f:2b","privateIp":"10.12.16.127","vlanId":1,"subnetCidr":"10.12.16.0/23"}]
(...)

Posted on 26/08/2021

Categories