Portable by default: open table formats in practice

Outline, not yet written. This seed article shows how Engineering notes render. Replace it in Sveltia CMS before launch.

“Yours to keep” is easy to promise and hard to check. Open table formats are one of the ways we make it checkable: the data sits in files and metadata that more than one engine can read.

What an open table format gives you

• Tables stored as open files plus open metadata, not inside one vendor’s engine.
• Schema changes and time travel without rewriting the data.
• The option to move compute without moving data.

A minimal example

A partitioned Apache Iceberg table, created from Spark SQL:

CREATE TABLE lake.sales.orders (
  order_id    BIGINT,
  customer_id BIGINT,
  amount      DECIMAL(12, 2),
  ordered_at  TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(ordered_at));

Where it fits in the pyramid

Open formats sit at the Data layer, but they matter most higher up: the knowledge graph and the models above can change platform without a migration project underneath them.

Checklist for a proposal

Question	What good looks like
Which format?	Apache Iceberg or Delta Lake, named in the design
Which catalogue?	Stated, with how to export it
Who owns the files?	The client, in the client’s cloud account