Session Outline
How fast can you modify your data collection to include a new field, make all necessary changes in data processing and storage, and then use that field in analytics or product features? For many companies, the answer is a few quarters, whereas others do it in a day. This data agility latency has a direct impact on companies’ ability to innovate with data. Schema-on-read has been a key strategy to lower that latency – as the community has shifted towards storing data outside relational databases, we no longer need to make a series of schema changes through the whole data chain, coordinated between teams to minimise operational risk. Schema-on-read comes with a cost, however. Errors that we used to catch during testing or in early test deployments can now sneak into production undetected and surface as product errors or hard-to-debug data quality problems much later than with schema-on-write solutions.
In this presentation at the Data Innovation Summit 2024, Lars Albertsson shows how we reject the trade-off between slow schema change rate and quality to achieve the best of both worlds. By using metaprogramming and versioned pipelines that are tested end-to-end, we can achieve fast schema changes with schema-on-write and the protection of static typing. We will describe the tools in our toolbox – Scalameta, Chimney, Bazel, and custom tools. We will also show how we leverage them to take static typing one step further and differentiate between domain types that share representation, e.g. EmailAddress vs ValidatedEmailAddress or kW vs kWh, while maintaining harmony with data technology ecosystems. Key talking points: The span between the least and most data agile companies is huge – 1000x latency.
A strong type system and end-to-end testing enables us to move fast with small risk. With metaprogramming, we can do better than schema-on-read – have the cake and eat it.
