Topics discussed:
- DuckDB got a mixed report. Loading NDJSON is quick, the internal representation is columnar and close to what Spark and BigQuery do rather than staying JSON, on-disk size is comparable to zipped NDJSON, and plenty of operations are fast. But once the data isn't flat, performance falls apart: filters and lambdas stay quick, and switching to lateral joins degraded things by five to ten times. That was reported upstream. A separate benchmark found it slower than Spark on a single machine for non-flat Parquet.
- The deeper problem with inferring a schema from JSON is that the schema is always incomplete — inference works by sampling — so SQL written against one inference may simply not run against another, and when inference gets it wrong you are in real trouble. Both DuckDB and Spark do this, and both do it the same way.
- ClickHouse came up as the more mature columnar option — over a decade old, battle-tested behind a large analytics product, understands Parquet, has experimental JSON support. It is noticeably less pleasant to use than DuckDB, with a lot more explicit setup, but it was floated as a default for open-source work by someone who doesn't think Spark is an efficient implementation. Someone planned to repeat the DuckDB experiment against it and report back.
- The BigQuery implementation — translating a ViewDefinition into BigQuery queries, similar in approach to others in the group — hasn't moved for several months. Separately there was intent to open-source a Postgres implementation, on the view that it shouldn't stay Postgres-only and dialects could be added afterwards.
- Spec housekeeping: the canonical URL needs changing, the copyright statement needs clearing up, and the QA report still has warning-level errors. Most of those are the validation tool marking things incorrectly rather than genuine problems, so the next step agreed was to document them in the published QA page and say so on the channel, rather than chase each one.