> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudhumans.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hourly delivery in v2 (from Nov 3, 2026)

> From November 3, 2026, your data arrives every hour, under a new v2/ prefix. What changes in the bucket, when the switch happens, and what to review in your ingestion.

From **November 3, 2026**, your bucket receives a new delivery **every hour** instead of
once a day at 07:00 UTC (04:00 in Brasília). The hourly deliveries land under a new
prefix, `v2/`. From that date on, `v1/` receives nothing new. This page explains what
changes, when, and what to review on your side before the switch.

Everything else stays as it is today:

* The bucket, and the way you access it, with an AWS account or a Google account.
* The Parquet format and every column, `tenant_id` included.
* The 19 tables in the [table catalog](https://docs.cloudhumans.com/data-export/catalog/overview).
* Every folder already in `v1/`. Nothing is moved or deleted, and you can keep reading it.
* The storage rules: files still move to colder [storage classes](https://docs.cloudhumans.com/data-export/storage-classes.md) as they age, and to S3 Glacier Flexible Retrieval after 180 days.

> [!NOTE]
>
> All hours on this page are UTC, which is 3 hours ahead of Brasília. `snapshot_hour=03` is
> midnight in Brasília and `snapshot_hour=15` is noon. The same applies to `etl_hour`.

## What changes

The schema and table folders under `v2/` are the same as under `v1/`. Inside each table,
the date folder gains an hour folder, and the manifest is written once per hour instead
of once per day.

### Tables rewritten in full

These 16 tables keep arriving as a whole copy of the table on every delivery. The copy
lands under an hour folder inside the date:

```text theme={null}
s3://<YOUR_BUCKET>/v2/datamart_claudia/claudia_conversation/snapshot_date=2026-11-03/snapshot_hour=03/
```

The latest data is the most recent `snapshot_date=` and `snapshot_hour=` holding
`_SUCCESS`. Since every copy is the whole table, you don't need anything from `v1/` to
read these tables after the switch.

| Schema | Tables |
| - | - |
| `datamart_claudia` | `claudia_contents`, `claudia_conversation`, `claudia_conversation_feedback`, `claudia_csat`, `eddie_result_outcomes` |
| `datamart_cloudchat_saas` | `cloud_chat_account_user_availability_histories`, `cloud_chat_agent_metrics`, `cloud_chat_contacts`, `cloud_chat_conversations`, `cloud_chat_conversations_custom_fields`, `cloud_chat_conversations_first_reply_metrics`, `cloud_chat_conversations_tags`, `cloud_chat_csat`, `cloud_chat_outbound_events`, `cloud_chat_tags`, `cloud_chat_team_metrics` |

Every hourly copy is kept for at least the last 7 days. For older days, only the copy
from the last hour of the day stays, so there is always at least one complete copy per
day, as there is today.

### Tables delivered in parts

`claudia_messages`, `claudia_prompt_revision` and `cloud_chat_messages` keep arriving in
parts. Each hour adds a new folder under `v2/` holding everything that arrived since the
previous delivery:

```text theme={null}
s3://<YOUR_BUCKET>/v2/datamart_claudia/claudia_messages/etl_date=2026-11-03/etl_hour=03/
```

`v2/` continues exactly where `v1/` stopped, so the history is not copied: everything
delivered up to November 2 stays only in `v1/`, and everything after it arrives only in
`v2/`. To reconstruct a table, read the union of its folders in both prefixes:

* In `v1/`, `etl_date=__initial__` and the daily `etl_date=` folders. They no longer
  change after November 2, so you need to read them only once.
* In `v2/`, the `etl_date=` / `etl_hour=` folders, as they arrive.

A folder is never rewritten after it is delivered. When a row of `claudia_messages` or
`claudia_prompt_revision` changes after it was delivered, the new version arrives again
in a later hour folder, in `v2/`, even if the first version came in `v1/`. Keep the most
recent version of each row: the one in the latest folder, by `etl_date` and then
`etl_hour`, with the `v1/` folders before the `v2/` ones and `etl_date=__initial__`
before everything. The key is `message_id` on `claudia_messages` and
`prompt_revision_id` on `claudia_prompt_revision`. `cloud_chat_messages` sends each
message once.

`etl_hour` is the hour of the delivery, not of the event. To filter by when something
happened, use `send_at` on `claudia_messages`, and `created_at` on
`claudia_prompt_revision` and `cloud_chat_messages`.

### One manifest per hour

```text theme={null}
s3://<YOUR_BUCKET>/v2/_manifests/snapshot_date=<DATE>/snapshot_hour=<HOUR>/manifest.json
```

| Field | What it tells you |
| - | - |
| `snapshot_date`, `snapshot_hour` | The delivery this manifest describes. `snapshot_hour` is new |
| `complete` | `true` when every table arrived in this hour |
| `tables_delivered` | Which tables arrived, each with how to read it, as today |
| `tables_missing` | Which tables did not arrive in this hour. For those, read the previous hour |
| `tables_pending` | New. Tables delivered in parts that will be caught up in a later hour |
| `delivered_through` | New, on each table delivered in parts. Up to which moment its rows have been delivered |

`complete: false` does not mean the data is wrong. It means some table will arrive in a
later hour, and the latest hour holding `_SUCCESS` for it is still valid.

## When it arrives

Each delivery holds the data as of about the start of the hour, and is usually available
within 40 minutes after it. Check `_SUCCESS` or the manifest instead of relying on a fixed
time, as described in [bucket layout](https://docs.cloudhumans.com/data-export/bucket-layout.md#how-to-tell-if-a-partition-is-ready).

## The switch

| When (UTC) | In Brasília | What happens |
| - | - | - |
| Nov 2, 2026, 07:00 | Nov 2, 04:00 | Last daily delivery, in `v1/` |
| Nov 3, 2026, 03:00 | Nov 3, 00:00 | First hourly delivery, in `v2/`, at `snapshot_date=2026-11-03/snapshot_hour=03` |

There is no delivery between the two. The first hourly delivery of the tables delivered
in parts includes everything the last daily delivery did not, November 2 included, so
nothing is lost in the gap.

## What to review

**1. Point your ingestion at v2/**

Replace `v1/` with `v2/` in every path you read. After November 2, nothing new
arrives in `v1/`.

**2. Read only the latest hour of a table rewritten in full**

Reading a whole `snapshot_date=` folder returns up to 24 copies of the same table.
Pick the most recent `snapshot_hour=` holding `_SUCCESS`, and read only that.

**3. Add the hour to your catalog**

If you declare partitions in Athena or Glue, define the tables under `v2/` with
`snapshot_hour` as a second partition column after `snapshot_date`, and `etl_hour`
after `etl_date`. Declare both as `string`.

**4. Keep the v1/ history of the tables delivered in parts**

Read the `v1/` folders of `claudia_messages`, `claudia_prompt_revision` and
`cloud_chat_messages` once, if you don't have them yet, and union them with what
arrives in `v2/`, keeping the most recent version of each row. Don't treat one
`etl_date=` folder in `v2/` as a complete day: a day is split across up to 24 hour
folders.

**5. Decide how often to read**

You can read every hour, using the manifest of each hour as the trigger. If you keep
reading once a day, you still receive everything, as long as the steps above are in
place.

## Prepare the switch ahead of time

You don't need to change your ingestion on November 3. Deploy it any time before, with
one rule: on every run, check whether `v2/` has received a delivery, and read from it if
it has. Until November 2 the job keeps reading `v1/`, and it switches by itself on the
first hourly delivery, even if that delivery arrives a few minutes late. Checking for
`v2/` is safer than switching at a fixed time for that reason.

The example below is an AWS Glue job in PySpark. The same logic works in any engine or
orchestrator.

```python glue_job.py lines expandable theme={null}
import boto3
from awsglue.context import GlueContext
from pyspark.context import SparkContext
from pyspark.sql import Window
from pyspark.sql import functions as F

BUCKET = "<YOUR_BUCKET>"

s3 = boto3.client("s3")
spark = GlueContext(SparkContext.getOrCreate()).spark_session
spark.conf.set("spark.sql.sources.partitionColumnTypeInference.enabled", "false")


def current_version() -> str:
    found = s3.list_objects_v2(Bucket=BUCKET, Prefix="v2/_manifests/", MaxKeys=1)
    return "v2" if found.get("KeyCount") else "v1"


def latest_ready(prefix: str) -> str:
    markers = [
        obj["Key"]
        for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=prefix)
        for obj in page.get("Contents", [])
        if obj["Key"].endswith("/_SUCCESS")
    ]
    return max(markers).rsplit("/", 1)[0] + "/"


version = current_version()

conversations = spark.read.parquet(
    f"s3://{BUCKET}/" + latest_ready(f"{version}/datamart_claudia/claudia_conversation/")
)

messages = spark.read.parquet(f"s3://{BUCKET}/v1/datamart_claudia/claudia_messages/")
if version == "v2":
    messages = messages.unionByName(
        spark.read.parquet(f"s3://{BUCKET}/v2/datamart_claudia/claudia_messages/"),
        allowMissingColumns=True,
    )

delivered = (
    F.when(F.col("etl_date") == "__initial__", F.lit("0000-00-00")).otherwise(F.col("etl_date")),
    F.coalesce(F.col("etl_hour"), F.lit("")),
)
latest = Window.partitionBy("message_id").orderBy(*[c.desc() for c in delivered])
messages = (
    messages.withColumn("_rank", F.row_number().over(latest))
    .filter("_rank = 1")
    .drop("_rank")
)
```

* **Tables rewritten in full**, such as `claudia_conversation`: `latest_ready` returns
  the most recent folder holding `_SUCCESS`, which is a date in `v1/` and a date and hour
  in `v2/`. The same call works on both sides of the switch.
* **Tables delivered in parts**, such as `claudia_messages`: the history in `v1/` is
  always read, and the folders in `v2/` are added once they exist. The example reads the
  whole table on every run for clarity. In production, read the `v1/` history once and,
  from then on, only the `v2/` folders you haven't processed yet.
* **Keep the latest version of each row.** A row of `claudia_messages` or
  `claudia_prompt_revision` that changes after delivery arrives again in a later
  folder. The window at the end keeps, for each `message_id`, the row from the most
  recent folder. In an incremental pipeline, the same rule is an upsert by key.
* **Partition columns are read as strings.** `etl_date` holds the literal
  `__initial__` in `v1/`, and turning type inference off keeps `etl_date`, `etl_hour`
  and `snapshot_hour` as strings on both sides, so the union lines up.
* **Permissions don't change.** The access you already have covers the whole bucket,
  `v2/` included.

Questions about the switch? Contact your Cloud Humans account team, or see
[troubleshooting](https://docs.cloudhumans.com/data-export/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.