Cloud Native Data Pipelines • Sid Anand • GOTO 2017

From the creator

This presentation was recorded at GOTO Chicago 2017. #GOTOcon #GOTOchgo http://gotochgo.com Sid Anand - Data Architect at Agari ABSTRACT Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, network topology, and compute and storage placement, they need to closely manage their core [...] TIMECODES 0:00 Introduction 0:16 About Me 1:06 Agari : What We Do 4:55 Cloud Native Data Pipelines 6:26 Desirable Qualities of a Resilient Data Pipeline 10:23 Use-Case : Message Scoring 12:58 Architectural Components 16:55 Tackling Cost 23:09 ASG : Queue-based 24:40 ASG - Build & Deploy 27:33 Tackling Operability : Requirements 28:27 Apache Airflow - Authoring DAGS 28:56 Apache Airflow - Perf. Insights 31:15 Apache Airflow - Alerting 33:29 Stream Processing Architecture 39:10 What is Avro? 40:38 Why is Avro Useful? 43:23 Avro Schema Example 43:53 Avro Schema Data File Example 44:26 Avro Schema Streaming Example 45:01 Avro Schema Registry Download slides and read the full abstract here: https://gotochgo.com/2017/sessions/80 RECOMMENDED BOOKS Liz Rice • Container Security • https://amzn.to/3oU4iJe Liz Rice • Kubernetes Security • https://www.oreilly.com/library/view/kubernetes-security/9781492039075 Brendan Burns, Joe Beda & Kelsey Hightower • Kubernetes: Up and Running • https://amz

Published 2017-05-31 · Watch on YouTube ↗
Availability and playback restrictions are controlled by the original channel.