2 min read | by Jordi Prats
Similarly on how we use the rsync command to copy content across the filesystem and even across servers, we can use aws s3 sync to not only sync files between the computer we are running it and an S3 bucket but also between S3 buckets
The can use the following combinations:
aws s3 sync <LocalPath> <S3Uri> [options]
aws s3 sync <S3Uri> <LocalPath> [options]
aws s3 sync <S3Uri> <S3Uri> [options]
If we want to copy some local files to an S3 bucket we can use the following syntax:
$ aws s3 sync . s3://example-bucket/test
upload: ./service.yaml to s3://example-bucket/test/service.yaml
(...)
It works in the same way to download content from S3:
$ aws s3 sync s3://example-bucket/test .
download: s3://example-bucket/test/serviceaccount.yaml to ./serviceaccount.yaml
(...)
But when the source and destination are also S3 URIs it's going to sync between S3 buckets:
$ aws s3 sync s3://k3s-staging/templates "s3://k3s-prod/templates"
copy: s3://k3s-staging/templates/web_layer_prod.json to s3://k3s-prod/templates/web_layer_prod.json
copy: s3://k3s-staging/templates/web_layer.json to s3://k3s-prod/templates/web_layer.json
(...)
When syncing content we might be want to be sure what's going to happen, with the --dryrun option we can see what's going to do without actually changing anything:
$ aws s3 sync s3://k3s-staging/templates "s3://k3s-prod/templates" --dryrun
(dryrun) copy: s3://k3s-staging/templates/web_layer.json to s3://k3s-prod/templates/web_layer.json
(dryrun) copy: s3://k3s-staging/templates/web_layer_prod.json to s3://k3s-prod/templates/web_layer_prod.json
(...)
Another set of useful options are the --exclude and --include options. With them we can filter the files we need. We need to take into account that filters that appear later in the command take precedence over filters that appear earlier in the command. So, for example, this will sync all the files ending with .txt:
$ aws s3 sync s3://example-bucket/test . --exclude "*" --include "*.txt"
But if we set the options the other way around no files will be sync
Posted on 03/05/2022