Troubleshooting Spinnaker: Rosco Bakes

spinnaker bake stage rosco Troubleshooting

2 min read | by Jordi Prats

If we have a baking stage failing that we cannot get our heads around, it might help to be able to retieve some extra information about it. We can get more details about connecting to the redis instance that rosco uses.

From redis we will be able to retrieve all the information that have been persisted that might help us understand better any issues we face.

First, we will have to get the job's uuid, to do so we just need to go to the build stage and click on the "View Bakery Details" link. It will lead us to a page with it's logs using an url similar to:


https://api.spinnaker.pet2cattle.com/bakery/logs/eu-west-1/4ff3d49b-3110-4230-b237-cc188a1c5e3c


Since this URL contains the uuid to make queries to redis, we can now connect to it using redis-cli:

# redis-cli -h rosco-redis.int.pet2cattle.io
rosco-redis.int.pet2cattle.io:6379>

To retrieve all the available keys we can use the hkeys specifying the uuid we found:

rosco-redis.int.pet2cattle.io:6379> hkeys 4ff3d49b-3110-4230-b237-cc188a1c5e3c
 1) "createdTimestamp"
 2) "bakeKey"
 3) "roscoInstanceId"
 4) "bakeStatus"
 5) "command"
 6) "region"
 7) "bakeLogs"
 8) "updatedTimestamp"
 9) "bakeDetails"
10) "bakeRequest"
11) "bakeRecipe"

Using hget we can retrieve any of these key's value as follows:

For example, we can get the bake logs:

rosco-redis.int.pet2cattle.io:6379> hget 4ff3d49b-3110-4230-b237-cc188a1c5e3c bakeLogs
"{\"logsContent\":\"==> amazon-ebs: Prevalidating AMI Name: p2c-flask-all-20220125140001-p2c-base\\n    amazon-ebs: Found Image ID: ami-45bcc0072b1fc0053\\n==> amazon-ebs: Creating temporary keypair: packer_61f00263-1a2f-ce40-6c08-3d3487d25112\\n==> amazon-ebs: Creating temporary security group for this instance: packer_61f00265-f19a-2d2e-f7a2-0cb6c2751a7c\\n==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...\\n==> amazon-ebs: Launching a source AWS instance...\\n==> amazon-ebs: Adding tags to source instance\\n    amazon-ebs: Adding tag: \\\"Name\\\": \\\"Packer Builder\\\"\\n        amazon-ebs: Instance ID: i-0690a95a54d573003\\n==> amazon-ebs: Waiting for instance (i-95a540690ad573003) to become ready...\\n==> amazon-ebs: Using ssh communicator to connect: 10.12.16.142\\n...\"}"

The instance id that have been used to bake it:

rosco-redis.int.pet2cattle.io:6379> hget 4ff3d49b-3110-4230-b237-cc188a1c5e3c roscoInstanceId
"13ceb553-4a4c-4917-a4a5-51d5849b5f30@ip-10-12-16-142.eu-west-1.compute.internal"

The exact packer command it have been used:

rosco-redis.int.pet2cattle.io:6379> hget 4ff3d49b-3110-4230-b237-cc188a1c5e3c command
"packer build -color=false -var aws_region=eu-west-1 -var aws_instance_type=c4.large -var aws_source_ami=ami-45bcc0072b1fc0053 -var aws_target_ami=p2c-flask-all-20220125140001-p2c-base -var aws_ssh_username=ec2-user -var aws_associate_public_ip_address=true -var repository=https://hub.pet2cattle.io/rpm/ -var package_type=rpm -var packages=p2c-flask -var configDir=/opt/rosco/config/packer -var iam_instance_profile=defaultPackerRole -var env=dev/snapshot /opt/rosco/config/packer/./p2c/p2c-base.json"
rosco-redis.int.pet2cattle.io:6379>

Among other information that might help us troubleshoot the failing bake

Posted on 01/06/2022

Categories