February 7, 2024

Learning and Sharing in Public with Dagster Lab's Pedram Navid

How to Subscribe
Share In

In this episode Sean is joined by Pedram Navid, Head of Data Engineering at Dagster Labs. They discuss the unique challenges and opportunities in the realm of data engineering, particularly the culture of learning and sharing within the field.

Pedram discusses the traditionally guarded nature of data engineering, contrasting it with the more open-source approach in software engineering. He highlights the potential downsides of this secrecy, such as the difficulty in learning best practices and innovating. The discussion also touches on the balance companies must strike between contributing to communal knowledge and protecting valuable data and intellectual property.

Pedram shares insights from his experiences at Dagster Labs, including the development of the Dagster Open Platform and its impact on fostering a culture of openness in data engineering. Additionally, they explore the future of collaboration in the field, considering emerging technologies and methodologies that could further encourage sharing and innovation over the next 5-10 years.

Links:

Other Podcast

September 11, 2024

Pseudoanonymization of Data with Jack Godau

In this episode, Sean sat down with Jack Godau to dive deep into the world of pseudoanonymization. Jack shared how pseudoanonymization differs from anonymization, explaining its value for maintaining data utility while complying with stringent regulations like GDPR.

August 28, 2024

The Evolution of Certificate Management with Anchor Security's Ben Burkert

In this episode we explore how certificates and TLS function, the inherent difficulties in managing internal TLS certificates, and why nearly every engineer has a horror story related to it.

August 14, 2024

What is a Data Lakehouse with Upsolver's Ori Rafael

In this episode, we sit down with Ori Rafael, CEO and Co-founder of Upsolver, to explore the rise of the lakehouse architecture and its significance in modern data management.