Enable access logs for an Ingress using ALB
3 min read | by Jordi Prats
When configuring the ALB using an Ingress object we can enable storing access logs to an S3 bucket in the same way we can do it for any ALB using terraform
To do so we need to use the following annotation alb.ingress.kubernetes.io/load-balancer-attributes. Using this annotations we'll need to specify the actual S3 bucket an the prefix we want to use. For the following example we are going to use:
- A bucket named: access-log-bucket
- Logs prefix: demo-app
So it's annotation will look as follows:
alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app
Once applied, we can get an error message (depends on the S3 bucket policy) complaining about S3 permissions (Access Denied for bucket Please check S3bucket permission):
$ kubectl describe ingress demo-web
Name: demo-web
Namespace: demo-flask
Address: internal-k8s-demoflask-1a00b666b7-1000851519.us-west-2.elb.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
demo-app.pet2cattle.com
/ ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
/ demo-app:9000 (10.12.102.98:9000,10.12.105.61:9000)
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/group.name: demo-app
alb.ingress.kubernetes.io/listen-ports: [{"HTTP":80},{"HTTPS":443}]
alb.ingress.kubernetes.io/load-balancer-attributes:
access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: demo-app
meta.helm.sh/release-namespace: demo-flask
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedDeployModel 23m ingress Failed deploy model due to InvalidConfigurationRequest: Access Denied for bucket: access-log-bucket. Please check S3bucket permission
status code: 400, request id: 40996c42-e02c-471c-a468-bc01ef8ab5af
We'll need to add a new statement to the bucket's policy:
(...)
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::elb-account-id:root"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/prefix/AWSLogs/your-aws-account-id/*"
}
We need to set the following values to it's appropriate value
- elb-account-id: We'll need to check AWS's documentation for enabling access logs on Application Load Balancers for a table to identify the correct account number for our AZ. For example, for us-west-2 it's going to be 797873946194
- bucket-name: As state previously, for this example we are using access-log-bucket
- your-aws-account-id: Let's assume our account number is: 123456789876
We'll need to update the bucket's policy to include at least the following statement:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::797873946194:root"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::access-log-bucket/demo-app/AWSLogs/123456789876/*"
},
{
(...)
Once applied, the Ingress is going to be able to reconcile, so eventually we will get the following event:
$ kubectl describe ingress demo-web
Name: demo-web
Namespace: demo-flask
Address: internal-k8s-demoflask-1a00b666b7-1000851519.us-west-2.elb.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
demo-app.pet2cattle.com
/ ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
/ demo-app:9000 (10.12.102.98:9000,10.12.105.61:9000)
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/group.name: demo-app
alb.ingress.kubernetes.io/listen-ports: [{"HTTP":80},{"HTTPS":443}]
alb.ingress.kubernetes.io/load-balancer-attributes:
access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: demo-app
meta.helm.sh/release-namespace: demo-flask
Events:
(...)
Normal SuccessfullyReconciled 15s (x3 over 7d1h) ingress Successfully reconciled
As soon as the ALB is allowd to push data into the S3 bucket, we are going to be able to see how the prefix we have defined gets created:
$ aws s3 ls s3://access-log-bucket
PRE demo-app/
Inside this folder we'll find several files that will match our setup:
$ aws s3 ls s3://sdlc-spinnaker-pet2cattle/demo-app/ --recursive
2022-06-28 01:12:42 104 demo-app/AWSLogs/123456789876/ELBAccessLogTestFile
2022-06-28 01:15:03 2306 demo-app/AWSLogs/123456789876/elasticloadbalancing/us-west-2/2022/06/27/123456789876_elasticloadbalancing_us-west-2_app.k8s-demoflask-1a00b666b7.ef7dfbd0eea82b38_20220628T0115Z_10.12.104.169_w36ca3td.log.gz
2022-06-28 01:15:03 123606 demo-app/AWSLogs/123456789876/elasticloadbalancing/us-west-2/2022/06/27/123456789876_elasticloadbalancing_us-west-2_app.k8s-demoflask-1a00b666b7.ef7dfbd0eea82b38_20220628T0115Z_10.12.106.222_da5mavql.log.gz
(...)
Posted on 28/06/2022