How to use aws s3 sync command

2 min read | by Jordi Prats

Similarly on how we use the rsync command to copy content across the filesystem and even across servers, we can use aws s3 sync to not only sync files between the computer we are running it and an S3 bucket but also between S3 buckets

The can use the following combinations:

aws s3 sync <LocalPath> <S3Uri> [options]
aws s3 sync <S3Uri> <LocalPath> [options]
aws s3 sync <S3Uri> <S3Uri> [options]

local path to S3

If we want to copy some local files to an S3 bucket we can use the following syntax:

$ aws s3 sync . s3://example-bucket/test
upload: ./service.yaml to s3://example-bucket/test/service.yaml

S3 to local path

It works in the same way to download content from S3:

$ aws s3 sync s3://example-bucket/test . 
download: s3://example-bucket/test/serviceaccount.yaml to ./serviceaccount.yaml

S3 to S3

But when the source and destination are also S3 URIs it's going to sync between S3 buckets:

$ aws s3 sync s3://k3s-staging/templates "s3://k3s-prod/templates"
copy: s3://k3s-staging/templates/web_layer_prod.json to s3://k3s-prod/templates/web_layer_prod.json
copy: s3://k3s-staging/templates/web_layer.json to s3://k3s-prod/templates/web_layer.json

Other options

When syncing content we might be want to be sure what's going to happen, with the --dryrun option we can see what's going to do without actually changing anything:

$ aws s3 sync s3://k3s-staging/templates "s3://k3s-prod/templates" --dryrun
(dryrun) copy: s3://k3s-staging/templates/web_layer.json to s3://k3s-prod/templates/web_layer.json
(dryrun) copy: s3://k3s-staging/templates/web_layer_prod.json to s3://k3s-prod/templates/web_layer_prod.json

Another set of useful options are the --exclude and --include options. With them we can filter the files we need. We need to take into account that filters that appear later in the command take precedence over filters that appear earlier in the command. So, for example, this will sync all the files ending with .txt:

$ aws s3 sync s3://example-bucket/test . --exclude "*" --include "*.txt"

But if we set the options the other way around no files will be sync

Posted on 03/05/2022