Table of Contents

Introduction to Data Ingestion

Principals of Data Access – Accessing your Data

Data Discovery – Understanding Our Data Before Ingesting It

Reading CSV and JSON Files and Solving Problems

Ingesting Data from Structured and Unstructured Databases

Using PySpark with Deﬁned and Non-Deﬁned Schemas

Ingesting Analytical Data

Designing Monitored Data Workflows

Putting Everything Together with Airﬂow

Logging and Monitoring Your Data Ingest in Airflow

Automating Your Data Ingestion Pipelines

Using Data Observability for Debugging, Error Handling, and Preventing Downtime

About the Author

Gláucia Esppenchutz is a data engineer with expertise in managing data pipelines and vast amounts of data using cloud and on-premises technologies. She worked in companies such as Globo, BMW Group, and Cloudera. Currently, she works at AiFi, specializing in the field of data operations for autonomous systems.
She comes from the biomedical field and shifted her career ten years ago to chase the dream of working closely with technology and data. She is in constant contact with the open source community, mentoring people and helping to manage projects, and has collaborated with the Apache, PyLadies group, FreeCodeCamp, Udacity, and MentorColor communities.