How the Detrics warehouse works under the hood

The full pipeline

While Claude designs the table groups, the video walks through how the warehouse works. The flow has four parts: the data sources (the Meta and TikTok accounts you want to store), the table groups (the data schema), the transfers (the download itself) and the destination (where the data lands in your Google Cloud).

The recurring problem: volume and images

Ad-level data is the most voluminous of all: account, campaign, ad group and ad, day by day. At that volume Google Sheets saturates: you hit the 10-million-cell limit, you can’t build more dashboards, and the ones you have turn slow. And the ad images don’t get stored: the platforms hand out URLs that expire.

How incremental deduplication works

Once the schema is designed, the transfer to BigQuery uses incremental deduplication: the most efficient way to download data, minimizing both BigQuery cost and platform API errors.

The mechanism: the history downloads once, going back in time. From then on, each day the data inside the deduplication window (two days, in the demo) gets deleted, fetched fresh, and written back along with today. Recent data gets replaced; historical data is never touched.

We call that historical data golden: it stays forever, with the images stored in your own cloud. So you can compare this year’s experiments against last year’s, or the year before, without losing anything.

Why you should set this up today

The platforms keep deleting the past. As of the video’s date (August 11, 2026), Meta only stores the last 3 years: the fourth year is being erased day by day. If you have clients older than 3 years (the case for many agencies), you’re losing the record of the work you did.

That’s why it matters to configure these transfers as soon as possible: download the available history so it becomes permanent, and let incremental deduplication store everything new from here on, for as long as the client relationship lasts.