Performance and best practices

Tuning the AWS CLI

Parquet files reach 256 MB, and a large table’s partition passes 500 MB. The CLI default of 10 concurrent requests leaves bandwidth on the table.

~/.aws/config

[profile cloudhumans-export]
region = us-east-1
s3 =
  max_concurrent_requests = 20
  max_queue_size = 10000
  multipart_threshold = 16MB
  multipart_chunksize = 16MB

Defaults, so you know what you’re changing:

Parameter	Default	Worth changing?
`max_concurrent_requests`	10	Yes, it gives the best return. See the measurement below.
`max_queue_size`	1000	Only for syncs with tens of thousands of files.
`multipart_threshold`	8 MB	Raising it to 16 MB reduces overhead on large Parquet files.
`multipart_chunksize`	8 MB	Keep it in sync with the threshold.
`max_bandwidth`	no limit	Useful if the sync is choking the rest of your network.

We measured this by downloading a single 74 MB Parquet file from a us-east-1 bucket to a machine outside AWS, varying only concurrency:

`max_concurrency`	Time	Throughput
10 (default)	5.9 s	12.5 MB/s
20	4.8 s	15.4 MB/s
30	3.9 s	19.0 MB/s

In other words: roughly a 50% gain from tripling concurrency, in a scenario bound by AWS egress latency. Inside AWS, in the same region, the curve climbs further. Measure in your own environment — the number that matters is yours.

Performance tip: the biggest win doesn’t come from tuning — it comes from not downloading what you don’t need. Filtering by snapshot_date= cuts volume by orders of magnitude; adjusting concurrency cuts it by tens of percent. Filter first.

If you run on a high-network EC2 instance, the CRT client might be worth testing:

~/.aws/config

preferred_transfer_client = crt

In boto3, the equivalent is TransferConfig, covered in Download with boto3. And the reminder everyone needs exactly once: run the download in the same region as the bucket (us-east-1). Cross-region transfer is slower, and a job in sa-east-1 pulling from us-east-1 explains a lot of mysterious slowness.

Best practices

Three habits that make your life easier and your integration more predictable:

Filter by partition instead of syncing the entire bucket on every run. An incremental sync already does this naturally; cp --recursive doesn’t, and will re-download everything, every time.
Use Bulk as your default restore tier, and Expedited only when there’s real urgency.
Contact your Cloud Humans account team before a large historical reprocessing — restoring months of data or downloading the entire bucket. This isn’t about asking permission, it’s so we can help: above a few hundred objects, the efficient path is S3 Batch Operations, and that runs on our side.

There’s no volume limit and no quota on your access. The request exists only so we know when heavy load is coming.

Getting Started

Integrations

Data Export

Performance and best practices

Tuning the AWS CLI

Best practices

​Tuning the AWS CLI

​Best practices

Tuning the AWS CLI

Best practices