Spark at Scale: Engineering Strategies for Data Science Workflows

September 4, 2023 | 10:43 am
Neil McCulloch, Data Science Engineer at dunnhumby

This is a talk by Neil McCulloch, Data Science Engineer at dunnhumby.

Improving the performance of problematic PySpark applications can often seem like a daunting task. In this talk, I will outline a strategy for tackling these projects, delving into a case study on the performance our in-store availability reporting science, and how we have slashed runtimes in half.

This session was part of the Data Science Festival Summer School in 2023. Find out more at…

The Data Science Festival is the place for data-driven people to come together, share cutting-edge ideas and solve real-world problems. We run monthly events, meetups and the biggest free-to-attend data festivals in the UK. Join the community at


Leave us a message using our contact form and we’ll get back to you straight away.

If you’re eager to get started, give us a call now on 01908 465 570


for reaching out, 🙏

A member of our team will be in touch shortly to arrange our chat.

Mmm 🍪cookies!

We use cookies to make your experience on this website better, and we have a variety to choose from. Use the toggles below to customise your selection or click 'Save my cookies' to get straight to the content.