VS Code Notebooks to Improve Your Microsoft Fabric Experience

When you’re using PySpark notebooks in Microsoft Fabric data engineering, you can develop straight from the web browser. While that is interesting, a browser is usually not the most perfect software development environment. In this article I will show you how you can use VS Code Notebooks to develop for Microsoft Fabric.

Why Use VS Code notebooks for Microsoft Fabric?

If you’re spending a lot of time working with Microsoft Fabric notebooks, switching to Visual Studio Code can make your life a bit easier. While editing notebooks in the browser works fine for quick edits or small tasks, VS Code gives you more control and flexibility. You get helpful features like autocomplete, linting, and easy file navigation all in a familiar, lightweight editor. It’s great for keeping projects organised, especially if you’re working on bigger projects or if you need to collaborate with colleagues using Git.

Version control is a big win. With Git built into VS Code, tracking changes or rolling back mistakes becomes simple. You can also work offline (in certain cases, not always with Fabric!), use extensions to suit your workflow, and write code just like you would in any other project. It’s not about ditching the browser entirely, rather it’s about using the right tool for the job when things get more complex.

How To Enable Fabric Notebooks in Visual Studio Code

Creating Notebooks from VS Code

Now if you want to start developing from inside Visual Studio Code, you can go to the Fabric Data Engineering plugin. Mind you, the icon is still the old Synapse Analytics icon so don’t try searching for the Fabric “F” 😉

The images above show the process of creating a new notebook in an existing Microsoft Fabric workspace. When you create the new notebook, VS Code automatically opens the notebook’s folder for you. In that folder make sure to click the <notebook name>.ipynb file to actually be able to create Markdown and code cells in your notebook.

Editing Existing Notebooks in Visual Studio Code

Of course, if you have already existing notebooks in Fabric you can edit them using Visual Studio Code. Doing so is easy (but I had to search for a bit the first time, hopefully this helps you!).

First, you need to navigate to one of your workspaces. Then, hover your mouse over the notebook you want to edit. Click on the following icon. This is (at the time of writing) the first icon that pops up when you hover over a notebook. Its caption reads “Open Notebook Folder”.

After doing so, you will see the same screen as in the New Notebook example before.

Running Code Against Fabric Compute

You can run the notebook’s code against your Fabric Compute from within Visual Studio Code. That is quite nice, because it means you don’t just have a proper software development environment for coding your notebooks, you can also test and run them against live data.

In order to do so, you need to specify which kernel you want the code to run against. With PySpark notebooks you will have to choose (9/10 times) between using a (local) Python kernel or an (online) Microsoft Fabric kernel. You can choose the former when you only run Python (not Spark) in your notebook cells and you want to run your code locally against a Python installation on your machine. You will choose the latter when you want to run code against Spark and against your Fabric or OneLake objects such as delta tables on a lakehouse. I will demonstrate both cases below.

Read more about Fabric compute and costs here.

Running Python Against Local Kernel

If you only run Python you can do so locally (but you can also run against an online Fabric capacity). Just select the proper kernel:

Choose for Python environments and select one:

Now running a cell might trigger the message below. If you select “install” the proper packages will be installed and your cell can run against your local Python instance.

Running PySpark Against Microsoft Fabric

After creating some awesome PySpark code like the one below, just hit the play button and the notebook cell will run.

However, it appears that if you run Spark code against your local Python kernel that won’t work. So you need to change to the proper Fabric kernel:

Now that you’ve connected to the Fabric Runtime you can run code against the compute of your Fabric capacity.

This works for regular Python cells as you can see below:

But the real super power here is when you’re interacting with your OneLake storage. For example, if you want to run queries against Delta Tables like so:

In order for this to work, the notebook must be connected to the lakehouses you’re interacting with. I have not found yet whether it’s possible to do so from VS Code, so I make sure to either publish to Fabric and do it there, or create the notebook in Fabric and then import into VS Code.

Publishing Notebooks

When you’re working with the notebooks locally, and you’re done developing, you need to publish the notebooks in order to have them appear in the Fabric workspace again.

Publishing your VS Code notebooks is fairly simple. You need to click back to the workspace view (click on the Fabric Data Engineering button). Then click the second button (Update Notebook) if available. If there are merge conflicts (updates both in remote and local definitions) you will see the screen below asking you how to merge the changes. I chose Accept Local Changes for now, but in practice you have to figure out how to resolve the conflict.

Now when you look at the notebook, the second button is called “Publish Notebook”. After hitting that button, the notebook in your Fabric workspace will have been updated.

Bonus

And the best part of using VS Code is… of course… using VS Code Pets! Who in their right mind would be developing Microsoft Fabric notebooks without occasionally throwing the ball for your loyal dog?