How ICA have accelerated data pipeline creation in GCP – John Thomas, ICA Gruppen


Session Outline

How we created a framework that can be easily configured for ETL type work of data pipelines. This reduces the complexity and effort for new developer onboarded to the team, by more than half to deliver a pipeline.

Key Takeaways Approach:

  • We created cloud support for our existing on-prem framework allowing developers to move between environments
  • Merits of configuration based framework with example
  • Examples of Cloud specific Consideration when designing
  • Why we choose workflow/scheduler in GCP over Composer for Orchestration
  • Deployment using Dataflow Flex custom templates

Add comment