Data Engineering · 5 min read
Merging Dataframes in PySpark
Today I’m going to teach you how the Python merge statement works, and we use it a lot in Microsoft Fabric.
Tag
spark
Posts tagged spark, newest first.
Data Engineering · 5 min read
Case Sensitivity in Microsoft Fabric Spark
I recently ran into a sneaky issue while migrating a customer from SQL Server to Microsoft Fabric. We were doing what every lazy developer does—copy-pasting code—and suddenly our joins stopped…
Data Engineering · 2 min read
Keep Spark Sessions Alive in Microsoft Fabric
Perhaps I’m somewhat late to the party, however, last Saturday I found something that really helps me as a data engineer working with Microsoft Fabric.
Data Engineering · 7 min read
VS Code Notebooks to Improve Your Microsoft Fabric Experience
When you’re using PySpark notebooks in Microsoft Fabric data engineering, you can develop straight from the web browser. While that is interesting, a browser is usually not the most perfect software…
Data Engineering · 7 min read
Delta Lake Liquid Clustering vs Partitioning
Learn how Liquid Clustering compares to Partitioning in Delta Lake. Boost MS Fabric performance with the right optimisation strategy.
Data Engineering · 10 min read
Delta Lake Partitioning for Microsoft Fabric
Learn how Delta Lake partitioning boosts query performance and reduces costs in Microsoft Fabric. Optimise your data lake today!
Data Engineering · 10 min read
Implementing the DRY Principle in Microsoft Fabric
Implementing the DRY principle in Microsoft Fabric makes developing more efficient and easier to maintain!
Data Engineering · 5 min read
Notebook Orchestration in Microsoft Fabric
In this post we will look at notebook orchestration, using PySpark notebooks with runMultiple in Microsoft Fabric.
Data Engineering · 7 min read
Spark Dataframes are Views, not Tables
In Microsoft Fabric data engineering, we use Spark to apply transformations to our data. Just like you would write T-SQL to transform data in SQL Server, you would write PySpark or SparkSQL to…