Designing the bulk export operation — one view or many, files or tables — Jan 22, 2025

Reuse $export, or write our own?

The preliminary view was our own — bulk export carries a lot this doesn't need and lacks things it does — but design the operation first and then see how much actually overlaps.

One point held regardless of the outcome: don't invent new names for parameters that already exist in bulk export.

Atomic vs batch — again

Nikolai argued for one view, one format, one destination, then compose upward.

The pushback: a ViewDefinition covers a single resource type, and real exports cross resource types, so the atomic unit may be too small to be useful — and views exported at different moments won't line up across references.

The compromise floated: atomic operations inside a transaction bundle, plus the separate observation that a named collection of ViewDefinitions is worth having anyway, both to distribute them and to say these belong together.

The sharpest split — files vs tables

They look like two flavours of the same export, but the parameters barely overlap:

• Files — format choices (NDJSON, Parquet, CSV)
• Tables — what if it already exists, drop and recreate or append, connection details

Proposed heuristic: split when the outputs differ, combine when they overlap.

Streaming — three distinct things hiding behind one word

Streaming came up for debugging — post a ViewDefinition, get sample rows straight back. The objection was that a count or sample-N parameter does that job without dragging in streaming semantics.

Teasing it apart produced three distinct things:

• A static set of files
• URLs whose contents keep changing
• A genuine event stream — much closer to Subscription than to export

Mapping view results onto Subscription is conceptually awkward

There is no serialisation convention for a row. There's also a scaling history — implementers rejected the older subscription model precisely because they couldn't let every client register arbitrary materialised views against their database; preset views can be indexed and tuned, arbitrary ones multiply.

Consensus: start with the file-based analogue of bulk export, build it as a small JavaScript reference implementation to keep the discussion concrete, and agree the minimum parameter set before writing much code.