Let the Notebook Write Itself (Sorta): My Experience Using Supermaven in Fabric

By now, everyone and their mother has heard about AI. It is the greatest technological revolution of our times. And we can do so much with AI, it’s amazing! Today, I want to demonstrate how I use AI copilots for Microsoft Fabric to help me write better code faster.

Note: none of the links in this article are sponsored. I write about my experiences and have no commercial benefits whatsoever from linking to products.

Note (2): In this article I assume you’re already using Visual Studio Code for Microsoft Fabric notebooks. If you’re not, please read my article on using VS Code for Fabric first.

The AI Copilot for Fabric Landscape in VS Code

Using AI in Visual Studio Code has become much easier since the introduction of all kinds of AI copilots. The name Copilot has been claimed by Microsoft and is widely used for their AI agents across many products, one of them being GitHub.

GitHub Copilot is a great solution to begin with. However, in my search for a free, copilot (note the lower case), I quickly came to the conclusion that GitHub Copilot only gives you 2,000 code completions per month. That is not little, but you’ll burn through it quicker than you might think and it’s annoying if your completions suddenly stop.

Another tool I’ve come across, which in my mind wins this battle, is Supermaven. Supermaven is a copilot built for developers and can get quite large context windows, ideal for use in larger code bases to get more accurate code completions.

The free tier of Supermaven is limited by the context window we get and the model that’s used, not by limiting the number of completions like GitHub Copilot. That means the free tier is actually usable, albeit slightly slower, less intelligent than the paid version. That’s totally fair I think! The free tier has a context window of 300,000 tokens, the paid tier is 1,000,000 tokens! A token can be roughly translated to about 4 characters. Go figure…

This article assumes using the Supermaven free tier, however, if you can, the paid tier might be worth it depending on your requirements.

How to Install Supermaven for Visual Studio Code

Why AI Copilots Shine in Microsoft Fabric Notebooks

If you’re not using AI copilots yet in Microsoft Fabric, you need to keep on reading this article. Let’s dive into some of the use cases of AI copilots and why they are awesome for you Microsoft Fabric Notebooks.

Use Case 1: Boilerplates

Gone are the days where you started out your project with an empty screen, and had to start building from scratch. Using AI you can now skip those long and boring hours of setting up boilerplates, foundations, or scaffolding.

You can ask your AI copilot such as Supermaven to create DataFrame definitions, merge logic, or even parameter cells.

You start writing, and 80% of the time the AI will correctly assume the ending of those lines. Because the boilerplates are generic, the AI is very good at predicting them. From there you start doing what you are good at: developing solutions for your users.

Use Case 2: SQL Magic inside PySpark notebooks

Even when running the %%sql magic command in a notebook, the AI will understand that you are now writing SQL instead of Python, and therefore autocomplete SQL just fine.

With a bit of nudging from you, it will understand and copy your style of SQL. It’s like having a colleague working with you, but a colleague that has memorised Microsoft Learn and the SQL documentation.

Use Case 3: Speed Up Notebook Prototyping

With AI it’s easy to create test data on the fly. By context chaining, it will also remember what you’ve written in the previous cells. So if you created a nice code snippet to do slowly changing dimension merges in PySpark, it will understand that. Then, start writing a test dataset and it will help you autocomplete it in order to properly test your function.

After it works, and you confirmed the functionality, you can easily discard the notebook cell (or comment it out). But you no longer need to create your test data manually. And that’s a win.

But… Beware!

Alright. All of this is awesome. But before you become best friends with your new AI buddy and he stabs you in the back, consider the following downsides.

Hallucinations

Hallucinations are what happens when AI LLMs starting making shit up. I mean that literally and most likely you have experienced this first hand using AI chatbots such as ChatGPT. They love telling you things that are not true.

The same happens with AI copilots, essentially they are the same as the LLMs you use to write code, texts, and other creative stuff.

The danger is when the copilots start making stuff up, such as table names that don’t exist. Once, I was working on a Fabric project for a company that uses Odoo ERP. Supermaven correctly guessed the ‘contacts’ table in Odoo (res_partner) but then completely tripped on another table name for ‘products’, by stating it would be res_products where actually it’s products_products.

Occasionally, check the outputs of your copilot. After all, they are just the copilot. You are the captain, and you are responsible for everything they do.

Context Drift

Tools like Supermaven are great at keeping context, but it can get fuzzy fast. If you’re often hopping between notebooks, or if you have a lot of code in one notebook, it could start slipping up its context. That means it might be feeding you autocompletions that are not that accurate anymore within the current context of your code.

With Supermaven’s large context window, this is not a problem for smaller projects, but it’s fair to give this warning.

Confident Garbage

The confidence with which AI gives you suggestions is the same for everything. If you spar with a human developer, they might say things like “it’s definitely XYZ”, or “I think it’s A or B but I’m not sure”. The conversation gives an insight into how confident your colleague is.

With AI, everything is presented using the same level of confidence. Whether the AI is absolutely sure or not, you don’t know.

Don’t just accept completions blindly. Test before you use it…

It Doesn’t Know YOU (yet)

AI copilots are trained on general code patterns and on libraries, not on your specific lakehouse naming conventions or on your specific style of SQL (for example, I absolutely hate that most SQL is written in upper case, I always write lower case SQL queries).

However, standards happen for a reason. It is absolutely important not to deviate too much from using generally accepted standards when writing code. When doing so, not only will your code be better (likely), but also your AI copilot such as Supermaven or GitHub Copilot will be much more accurate.

Leaving “breadcrumbs” in your code to steer the copilot towards what you’re trying to do is an optimisation technique you could use here. More on that in the next paragraph.

Pro Tips for Better AI Copiloting

As we left of the last paragraph with a teaser for better AI copiloting, I want to share a few tips that I found very helpful.

Comment Your Intent

The first tip is directly related to where we left the previous paragraph. By commenting intent, not the obvious, you cannot only help others make your code more readable, it also helps the AI to understand your thought process better.

Copilots are a bit like Little Red Riding Hood from the old story books. They love breadcrumbs 😉

By commenting things like “# Loading product data from the Silver lakehouse” instead of “# load dataframe” you will tell the AI not exactly what you’re trying to achieve. This helps the AI in giving you better code.

The more context you give, the better the outcome will be. So use comments in your code (# with Python, — or /* and */ with SQL) or even doc strings (for Python functions). That will really help your copilot.

I guarantee you that when you leave comments in your code like “# Create a slowly changing dimension type 2 (SCD2) merge, from silver_df to gold_df, using business_keys list and scd2_columns list” you will get quite a good first time code completion. That will save you some time, and after all, that’s where the money is.

Use It For Prototyping, Not Just Code Completion

By telling the AI what you want to do (almost prompting) you can get quite good results. Try to write cells in a notebook like this:

“# write a function to parse nested JSON into a flattened Spark dataframe
def flatten_json( … << Let the AI do its magic from here

Completing not just your line, but probably the entire function, will be the result. 9 out of 10 times it will also be of value. Maybe not 100% right the first iteration, but at least it gives you a boiler plate solution to begin refining.

Give It Patterns

This one is a bit more sneaky. LLMs love patterns, after all, they are really good in pattern recognition and prediction. By giving it examples of how you usually read data from a Delta table into a Spark dataframe, it will repeat the next time in your style.

Conclusion

Start using copilots in your workloads today. Why wouldn’t you? You can automate away the busy work, get insights and results faster than before, and probably with better code as well.

And please don’t be afraid you will lose your job to AI. I think it’s much more likely to lose your job to a developer who is using AI if you are not, because they will be better and faster than you.

Let me know in the comments below what your experiences are using AI copilots for Microsoft Fabric!