Spark at Scale: Engineering Strategies for Data Science Workflows

September 4, 2023 | 10:43 am
Neil McCulloch, Data Science Engineer at dunnhumby

Please complete the form to watch the webinar recording


    You can watch the video below. Let us know if you learn something new or useful by tweeting us @DataIdols.

    This is a talk by Neil McCulloch, Data Science Engineer at dunnhumby.

    Improving the performance of problematic PySpark applications can often seem like a daunting task. In this talk, I will outline a strategy for tackling these projects, delving into a case study on the performance our in-store availability reporting science, and how we have slashed runtimes in half.

    This session was part of the Data Science Festival Summer School in 2023. Find out more at…

    The Data Science Festival is the place for data-driven people to come together, share cutting-edge ideas and solve real-world problems. We run monthly events, meetups and the biggest free-to-attend data festivals in the UK. Join the community at

    Mmm 🍪cookies!

    We use cookies to make your experience on this website better, and we have a variety to choose from. Use the toggles below to customise your selection or click 'Save my cookies' to get straight to the content.