Hyperight

How ICA have accelerated data pipeline creation in GCP – John Thomas, ICA Gruppen

Bookmark

Premium content

Login or register to unlock the content

Session Outline

How we created a framework that can be easily configured for ETL type work of data pipelines . This reduces the complexity and effort for new developer onboarded to the team, by more than half to deliver a pipeline.

Key Takeaways Approach:

  • We created cloud support for our existing on-prem framework allowing developers to move between environments
  • Merits of configuration based framework with example
  • Examples of Cloud specific Consideration when designing
  • Why we choose workflow/ scheduler in GCP over Composer for Orchestration
  • Deployment using Dataflow Flex custom templates

Add comment