I had the pleasure of speaking at DATA:SCOTLAND, a free data and analytics conference in Glasgow that brought together nearly a thousand data enthusiasts.
My session, titled ‘Delta Lake Optimisation’, focused on the stuff that actually matters when your lakehouse starts dragging:
• Why and when partitioning helps (until it doesn’t)
• How clustering fixes what partitioning can’t
• Why small files are silent query killers
• How to use OPTIMIZE, VACUUM, and Azure Storage Explorer to see what’s really going on under the hood
• And yes, I demoed it all live on 297M rows, with real performance gains
Big thanks to the organisation for having me, and to the crowd that showed up with great questions.
Slides and notebook recap are coming soon to my public GitHub page (/basland/presentations). Until then, keep your layouts lean and your file counts low 😉
You can read more about Partitioning and about Liquid Clustering on my blog.
During the session, I had 37 people in the session which is not too bad for such a niche, technical subject.