You have 4 summaries left

DataFramed

#209 Effective Data Engineering with Liya Aizenberg, Director of Data Engineering at Away

Thu May 23 2024
data engineeringhigh-performing teamsbusiness alignmentmeasuring successAItrends

Description

This episode covers insights on building high-performing data engineering teams, aligning data engineering initiatives with business goals, measuring success and driving value in data engineering projects, and the role of data engineering in the age of AI.

Insights

Creating a supportive atmosphere for data team members

Creating a supportive atmosphere where data team members can experiment with new ideas while focusing on those that benefit the business is crucial.

Traits valued in data engineering teams

Cultural traits such as ownership, communication skills, adaptability to change, and being result-driven are valued in data engineering teams.

Technical skills required for early data engineering hires

Technical skills required for early data engineering hires include knowledge of Python, relational and non-relational databases, visualization tools like Tableau, virtualization tools like Lucur and DBT for building SQL models.

Prioritizing projects aligned with business goals

Focus on delivering value by aligning data engineering initiatives with business goals. Prioritize projects that address critical business needs and contribute to revenue generation or cost reduction.

Measuring success in data engineering projects

Quantifying ROI for projects like building a data platform can be based on metrics like conversion rate increase. Success of data engineering features dependent on other teams using them can be measured by empowering business users through education and collaboration.

Being iterative and agile as a data engineering team

Being iterative and agile as a data engineering team involves flexibility, adaptability, teamwork, and prioritizing value-driven projects. Approaching massive projects incrementally by starting with an MVP and adding features over time can help drive value effectively.

The role of data engineering in the age of AI

Data engineering is crucial in bringing data from various sources together for generative AI models to work effectively. Skills of data engineering teams need to evolve to handle large volumes of data as generative AI becomes more prevalent.

Trends in the data engineering space

Trends in the data engineering space include gradual adoption of AI technologies, experimentation, learning from mistakes, and collaboration within the team.

Chapters

  1. Building High-Performing Data Engineering Teams
  2. Aligning Data Engineering Initiatives with Business Goals
  3. Measuring Success and Driving Value in Data Engineering Projects
  4. The Role of Data Engineering in the Age of AI
Summary
Transcript

Building High-Performing Data Engineering Teams

00:00 - 07:05

  • Creating a supportive atmosphere where data team members can experiment with new ideas while focusing on those that benefit the business is crucial.
  • Data engineers play a vital role in enabling proper data access for all who need it.
  • Leah Eisenberg, Director of DataEngineering@away.com, shared insights on building high-performing data engineering teams aligned with business value.
  • Traits like passion for data, willingness to learn, and result-driven mindset are essential when hiring for early data engineering roles.
  • Cultural traits such as ownership, communication skills, adaptability to change, and being result-driven are valued in data engineering teams.
  • Technical skills required for early data engineering hires include knowledge of Python, relational and non-relational databases, visualization tools like Tableau, virtualization tools like Lucur and DBT for building SQL models.
  • Must-have technical knowledge includes SQL proficiency and familiarity with integration tools like IPaaS solutions, airflow, and cloud platforms (AWS, GCP, Azure).
  • Empowering existing team members to grow into specialist roles is preferred before seeking external specialists.

Aligning Data Engineering Initiatives with Business Goals

06:44 - 13:56

  • Consider internal team specialists first before seeking external expertise.
  • Pair up external experts with internal team members to facilitate knowledge transfer.
  • Identify team members' interests and strengths to assign them suitable projects for growth.
  • Select tech tools based on future-proofing, existing expertise, and budget constraints.
  • Avoid using multiple overlapping tools to ensure simplicity and cost-effectiveness.
  • Focus on delivering value by aligning data engineering initiatives with business goals.
  • Prioritize projects that address critical business needs and contribute to revenue generation or cost reduction.

Measuring Success and Driving Value in Data Engineering Projects

13:38 - 21:01

  • Identifying low-hanging fruits for quick wins is beneficial in data engineering projects.
  • Quantifying ROI for projects like building a data platform can be based on metrics like conversion rate increase.
  • Some projects, like providing order visibility, may have intangible benefits that are hard to measure but still valuable.
  • Success of data engineering features dependent on other teams using them can be measured by empowering business users through education and collaboration.
  • Being iterative and agile as a data engineering team involves flexibility, adaptability, teamwork, and prioritizing value-driven projects.
  • Approaching massive projects incrementally by starting with an MVP and adding features over time can help drive value effectively.
  • Defining the MVP together with stakeholders and focusing on small wins while avoiding overhauling everything at once is key in project success.

The Role of Data Engineering in the Age of AI

20:35 - 25:20

  • Data engineering teams should focus on projects that bring value to the business and avoid pursuing shiny but not useful tools.
  • Data engineering is crucial in bringing data from various sources together for generative AI models to work effectively.
  • Skills of data engineering teams need to evolve to handle large volumes of data as generative AI becomes more prevalent.
  • Trends in the data engineering space include gradual adoption of AI technologies, experimentation, learning from mistakes, and collaboration within the team.
1