Terraform: Using the external data source

2 min read | by Jordi Prats

To be able to execute an arbitrary command to retrieve some data and being able to use it as a variable in terraform we can use the external data-source

To be able to read the data the command will have to output the data using JSON. For example, if we want to use the following data in terraform:

$ kubectl get deployment pet2cattle -n pet2cattle -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name == "MINIO_BUCKET")].value}'
pet2cattle

We will have to transform it somehow so it is in JSON, for example by simulating a map with a single key-value pair:

$ kubectl get deployment pet2cattle -n pet2cattle -o jsonpath='{"{"}"minio-bucket": "{.spec.template.spec.containers[0].env[?(@.name == "MINIO_BUCKET")].value}"}'
{"minio-bucket": "pet2cattle"}

Once the command we want to run output is a JSON, it is ready to be used using the external data-source in terraform. To do so we might have to scape some characters but the is going to be pretty much the same:

data "external" "minio_bucket" {
  program = ["sh", "-c", "kubectl get deployment pet2cattle -n pet2cattle -o jsonpath='{\"{\"}\"minio-bucket\": \"{.spec.template.spec.containers[0].env[?(@.name == \"MINIO_BUCKET\")].value}\"}'"]
}

If we output the result using:

output "minio_bucket" {
  value = data.external.minio_bucket.result
}

We will able to see that the result of the external data source it's going to be the JSON output structure:

$ terraform apply -auto-approve
data.external.minio_bucket: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

minio_bucket = {
  "minio-bucket" = "pet2cattle"
}

On the pet2cattle github you'll find the terraform-external-datasource repository with the Kubernetes resources and the terraform code to be able to test this code: Clone the repository and create the Kubernetes resources using kubectl apply:

$ kubectl apply -f k8s_resources.yaml 
namespace/pet2cattle created
deployment.apps/pet2cattle created

Once the resources are present on the cluster you just need to run terraform init and then terraform apply to see the described output value

Posted on 17/08/2021

Categories