A couple of weeks ago, I showed you how we can use DuckDB as a Python library in Microsoft Fabric to do Python notebooks on Delta Lake instead of using Spark notebooks.
The benefits of using Python over Spark: Python is more lightweight, your session startup times are much lower, and these notebooks use less compute resources than a Spark notebook. So for smaller datasets, it’s actually very nice to use DuckDB in Microsoft Fabric.
However, what I didn’t show you—and what I found out the other day and am showing you today—is how we can actually change the settings: the number of compute power, the vcores and memory that we can get inside a Python notebook in Microsoft Fabric.
The CPU and Memory Monitor
Here we are in a Python notebook. We can see that we’re running in Python language—Python is ticked, not PySpark. That’s how we can see visually that we’re looking at Python.
We have the very simple demo notebook that I showed you the other day where I was demoing DuckDB. We’re still doing the same thing: import DuckDB, connect DuckDB, attach a table from the lakehouse to DuckDB, and then do a count.
What I want to show you is right here on the bottom of the screen. It’s maybe a little bit hard to read, but what we see is that we have CPU: 2 vcores and RAM: 16 GB.
We’re only using 2.6% of our CPU and only 14% of our memory. When we click on this button, we get a graph where we can see our CPU usage over time and our memory usage over time.
What I want to show you today is how we can not only find out bottlenecks in our notebooks using this graph, but also how we can increase the compute power we get with our Python notebooks.
Testing with a Larger Dataset
I’ve created a table that contains about 74-75 million rows, and I’ll be loading that into DuckDB and then doing a simple count on that table.
Let’s run all and see what’s going on.
As you can see on the right side, my CPU usage is already increasing quite a bit. We’re hitting 100% CPU utilization at the moment.
In my testing, I was expecting about 40 to 50 seconds to actually load this table into DuckDB while using almost all of our memory as well. We’re coming from less than 2 GB consumed, and we’re increasing now percentage-wise almost to 100%.
Now we can actually see that we’re done. Right before this cell completed, we saw the CPU usage dropping quite a bit. Memory usage still at an all-time high.
Indeed, in 40 seconds we’ve created the DuckDB connection. We’re now running a query to find the row count in this table.
If we scroll down, we find that we have 74-75 million lines in this table in a total of about a minute—40 seconds to do the connection and then 16 seconds to run the query.
You can see there was a large bump in our CPU usage, and our memory usage keeps on going.
Increasing Compute Power with Magic Commands
How can we actually increase the hardware—the performance—that we get in our Python notebooks?
What we can do is put in a cell that’s a configuration cell using the magic command:
%%configure -f
{"vcores": 64}We type “vcores” and put in 64—that’s the highest we can go, I think. According to the documentation, we can go to 64.
Let’s run this cell and… we get an error.
The Capacity Limitation
Why do we get an error? We’re basically asking to create a Spark node (a server), and yes, we’re not running Spark, but we are getting a single-node server to run Python in.
We’re asking to do that in the extra-extra-large node size on our small little Fabric F2. And this F2, of course, is way too small to get this much hardware.
Looking at our concurrency limits, we’re working with an F2 today. You can see we get a maximum of four Spark cores.
With an F32, we can get 64 Spark cores. Maybe we can go higher all the way to 4,096 Spark cores on an F2048.
Upgrading the Capacity
Let’s see if we can actually get an F32 with 64 vcores. Let’s go to my portal, go to “Scale,” and change size on my demo capacity. Put in the F32 right here.
Now we go back to the notebook and try to run this again.
Apparently we’re still on an F2. I think we need a little bit of time for the engine to catch up that we’re now actually upgraded to an F32.
I tried it again, and since the cell is now running for a bit longer than I expected (we’re not getting an error), I think we’re now actually getting a larger node size.
Understanding Memory Allocation
We’ll also be getting more memory. What Fabric will do: we’ll be getting 8 GB of memory for every vcore that we have in our system.
With the two vcores, we got 16 GB. With 64 vcores, I expect to get 512 GB of memory for my Python session.
It does take a while though. I’m not sure why this takes so long.
Here we are. The session is now started, and we do have additional vcores. On the right side, we still see two vcores and 16 GB. But on the bottom, we see 64 vcores and 512 GB already.
Maybe we need to refresh. Now indeed we can see that we have way more cores than we had before.
Testing the Performance Improvement
Let’s start the connection and the query again and see what’s going to happen.
DuckDB is not defined. I need to run this again because I think the kernel got restarted. Let’s import DuckDB again and then run the connection and the mapping again.
The CPU usage is growing. Memory usage is growing as well.
Let’s see if we’re actually going faster. And yes, we are going much, much faster.
We’re now at 15 seconds as opposed to consuming 40 seconds. So we’re almost three times as fast.
The query itself was not that much faster. Maybe that’s a single-threaded process—I don’t know. But at least we’re getting an answer in total that’s way faster.
Key Takeaways
Do we have a way to make our processes in Python faster? Yes, we can. We can allocate more hardware—basically more resources—to our Python notebooks. That will speed up our process.
However, we need to keep in mind that looking at our concurrency limits, we only get so many vcores for a given Fabric SKU. Always check this table if you want to find out how many vcores you can actually attach to your Python notebook.
The Configuration Command
To summarize, here’s how to configure your Python notebook for more compute power:
%%configure -f
{"vcores": 64} # Adjust based on your capacity SKURemember:
- You get 8 GB of RAM per vcore
- Your capacity SKU limits the maximum vcores available
- Larger configurations take longer to start up
- Performance improvements vary depending on the workload
Are you using Python notebooks in Fabric? Have you tried configuring custom vcore settings? Let me know in the comments how much performance improvement you’ve seen!