> ## Documentation Index > Fetch the complete documentation index at: https://docs.cloudhumans.com/llms.txt > Use this file to discover all available pages before exploring further. # Downloading files > Read the export with the AWS CLI, with boto3 in an orchestrator, or in place with a query engine. There are several ways to read the export. These are the ones we suggest, and which one fits depends on who is doing the reading. | Your case | Use | | --------------------------------------------------------- | --------------------------------------------------------------------------------- | | Exploring by hand, or a shell script | [The AWS CLI](#download-with-the-aws-cli) | | A pipeline or a scheduled job | [boto3](#download-with-boto3) | | Analysis, or you would rather not keep a copy of the data | [A query engine, with no download at all](#query-in-place-instead-of-downloading) | Credentials are the same question in all three: on your own machine you use the `cloudhumans-export` profile, and on AWS compute you use the role that compute already runs as, with no profile on the host. ## Download with the AWS CLI A quick inventory of what exists and how much space it takes: ```bash theme={null} aws s3 ls s3:///v1/ --recursive --human-readable --summarize \ --profile cloudhumans-export ``` The examples below partition by `snapshot_date=`. Tables delivered as a daily delta partition by `etl_date=` instead, and each table's page in the [table catalog](/data-export/catalog/overview) states which one it uses. Everything else about the commands is the same. One specific partition, the most common case day to day: ```bash theme={null} aws s3 sync \ s3:///v1/datamart_claudia//snapshot_date=/ \ ./data/
/snapshot_date=/ \ --profile cloudhumans-export ``` A whole table, incrementally. `sync` compares size and modification date, so running it again only pulls what changed: ```bash theme={null} aws s3 sync \ s3:///v1/datamart_claudia/
/ \ ./data/
/ \ --profile cloudhumans-export ``` On a table with more than 180 days of history this reaches archived files, which `sync` skips with a warning before exiting with code 2. See [storage classes](/data-export/storage-classes#restoring-archived-data). A date range, without downloading the entire history: ```bash theme={null} aws s3 sync \ s3:///v1/datamart_claudia/
/ \ ./data/
/ \ --exclude "*" \ --include "snapshot_date=2026-08-*" \ --profile cloudhumans-export ``` **Order matters:** `--exclude` and `--include` are evaluated left to right. `--exclude "*"` first, `--include` after. Reversed, you download everything. Check which storage class each file is in — useful for knowing, before you try, whether something will need a restore: ```bash theme={null} aws s3api list-objects-v2 \ --bucket \ --prefix v1/datamart_claudia/
/ \ --query 'Contents[].[StorageClass,Size,Key]' \ --output text \ --profile cloudhumans-export ``` ```text theme={null} STANDARD 991758 v1/datamart_claudia/
/snapshot_date=2026-08-05/0000_part_00.parquet STANDARD 0 v1/datamart_claudia/
/snapshot_date=2026-08-05/_SUCCESS STANDARD_IA 991758 v1/datamart_claudia/
/snapshot_date=2026-06-20/0000_part_00.parquet GLACIER_IR 991758 v1/datamart_claudia/
/snapshot_date=2026-04-10/0000_part_00.parquet GLACIER 77597818 v1/datamart_claudia/
/snapshot_date=2025-11-15/0000_part_01.parquet STANDARD 0 v1/datamart_claudia/
/snapshot_date=2025-11-15/_SUCCESS ``` Notice the `_SUCCESS` markers: they stay in `STANDARD` even in the oldest partition — [storage classes](/data-export/storage-classes#file-age-and-storage-class) explains why. ## Download with boto3 For use in an orchestrator (Airflow, Dagster, Prefect). Downloads a partition only after confirming the `_SUCCESS` marker, and handles the errors that actually happen instead of swallowing everything in a single `except Exception`. Before you process a day, read that snapshot date's manifest to confirm the delivery is complete, as described in [bucket layout](/data-export/bucket-layout). If the orchestrator runs on AWS compute, set `PROFILE = None`: the SDK then uses the role that compute already runs as, and no profile needs to exist on the host. See [local profile](/data-export/overview#local-profile). ```python download_partition.py lines expandable theme={null} """Download a partition from Cloud Humans Data Export.""" from __future__ import annotations from pathlib import Path import boto3 from boto3.s3.transfer import TransferConfig from botocore.exceptions import ClientError BUCKET = "" # None on AWS compute, so the SDK uses the role the environment already provides. PROFILE = "cloudhumans-export" # 20 threads instead of the default 10. Beyond that the gain flattens out and the risk of 503s increases. TRANSFER = TransferConfig(max_concurrency=20, multipart_threshold=16 * 1024 * 1024) class PartitionNotReady(Exception): """The partition exists but doesn't have the _SUCCESS marker yet.""" class ObjectArchived(Exception): """The object is in Glacier Flexible Retrieval and requires a restore before it can be read.""" def _client(): session = boto3.Session(profile_name=PROFILE) if PROFILE else boto3.Session() return session.client("s3") def partition_prefix(table: str, partition: str) -> str: """`partition` is the folder name, `snapshot_date=2026-08-06` or `etl_date=2026-08-06`.""" return f"v1/datamart_claudia/{table}/{partition}/" def is_ready(s3, prefix: str) -> bool: try: s3.head_object(Bucket=BUCKET, Key=f"{prefix}_SUCCESS") return True except ClientError as err: if err.response["Error"]["Code"] in ("404", "NoSuchKey"): return False raise def download_partition(table: str, partition: str, dest: Path) -> list[Path]: s3 = _client() prefix = partition_prefix(table, partition) if not is_ready(s3, prefix): raise PartitionNotReady(prefix) dest.mkdir(parents=True, exist_ok=True) written: list[Path] = [] for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=prefix): for obj in page.get("Contents", []): key = obj["Key"] if key.endswith("/") or key.endswith("_SUCCESS"): continue target = dest / Path(key).name try: s3.download_file(BUCKET, key, str(target), Config=TRANSFER) except ClientError as err: code = err.response["Error"]["Code"] if code == "InvalidObjectState": raise ObjectArchived(key) from err raise written.append(target) return written if __name__ == "__main__": files = download_partition("
", "snapshot_date=", Path("./data")) print(f"{len(files)} files downloaded") ``` `InvalidObjectState` gets its own exception because it's the only failure in this list that is **not** a configuration problem — the data is there and intact, it only needs restoring first. See [storage classes](/data-export/storage-classes#restoring-archived-data) for how. ## Query in place instead of downloading The files are Parquet in Hive-style partitions, so a query engine can read them where they are. This is the shortest path for analysis and for anything you run once: no copy to keep in sync, and the engine reads only the partitions your `WHERE` clause touches. Run the engine in the bucket's region, `us-east-1`. Reading across regions is slower and the volumes here are large enough for it to show. **Spark** infers the partition columns from the folder names, so pointing it at the table root is enough. Needs the `hadoop-aws` connector: ```python theme={null} df = spark.read.parquet("s3a:///v1/datamart_claudia/
/") ``` **Athena** needs the schema up front, since an external table has no inference. Take the columns from that table's page in the [table catalog](/data-export/catalog/overview), or let a Glue crawler write the definition for you: ```sql theme={null} CREATE EXTERNAL TABLE
( -- one line per column, from the table's catalog page ) PARTITIONED BY (snapshot_date string) STORED AS PARQUET LOCATION 's3:///v1/datamart_claudia/
/'; MSCK REPAIR TABLE
; ``` Three things to know before you point an engine at the bucket: * **The markers do not get in the way.** `_SUCCESS`, `_STATE` and the manifests start with an underscore, which is exactly the convention Hive, Spark, Athena and Presto use for files to skip. They are not read as data. * **Declare the partition column of a daily delta table as a string.** Those tables partition by `etl_date=`, and one of the values is the literal `__initial__`, so a date type fails on it. That also rules out Athena partition projection over a date range for those two tables. * **Archived files are not readable in place.** Beyond 180 days the objects move to S3 Glacier Flexible Retrieval, and an engine reading them fails the same way a download does. Restore first, as described in [storage classes](/data-export/storage-classes#restoring-archived-data).