In this episode Sean is joined by Pedram Navid, Head of Data Engineering at Dagster Labs. They discuss the unique challenges and opportunities in the realm of data engineering, particularly the culture of learning and sharing within the field.
Pedram discusses the traditionally guarded nature of data engineering, contrasting it with the more open-source approach in software engineering. He highlights the potential downsides of this secrecy, such as the difficulty in learning best practices and innovating. The discussion also touches on the balance companies must strike between contributing to communal knowledge and protecting valuable data and intellectual property.
Pedram shares insights from his experiences at Dagster Labs, including the development of the Dagster Open Platform and its impact on fostering a culture of openness in data engineering. Additionally, they explore the future of collaboration in the field, considering emerging technologies and methodologies that could further encourage sharing and innovation over the next 5-10 years.
Links:
In this episode, Kirk Marple, CEO and Co-founder of Graphlit, joins the show. Sean and Kirk dive into the world of unstructured data management, discussing the evolution and current challenges in the field.
In this episode, Jake Moshenko, CEO and co-founder of AuthZed, joins the show to explore the world of user permissions at scale. Inspired by Google's Zanzibar, AuthZed aims to tackle the challenges of authorization - a less common focus compared to authentication in the tech industry.
Aaron Painter, CEO of Nametag, joins the show to explore the evolving threat of AI deepfakes. They discuss the increasing sophistication of deepfake technology, highlighted by the significant rise in incidents such as the Retool hack, and how these technologies can manipulate public perception and security.