How to profile memory usage in Python
- select the contributor at the end of the page -
One of the ways Python makes development fast (not to mention easier than languages like C and C++ ) is memory management. In Python it's simple because the language handles memory management for you. However, this doesn't mean memory should be forgotten. Good developers will want to track the memory usage of their application and look to lower memory usage. This post will explain common tools for doing this.
Profiling size of individual objects
The lowest layer of memory profiling involves looking at a single object in memory. You can do this by opening up a shell and doing something like the following:
Profiling a single function or method
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
This is extremely useful if you're wanting to profile a section of memory-intensive code, but it won't help much if you have no idea where the biggest memory usage is. In that case, a higher-level approach of profiling is needed first.
Profiling an entire application
This will print a nice table of usage grouped by object type. Here's an example of an PyQt4 application I've been working on:
This type of profiling can be difficult if you have a large application using a relatively small number of object types.
Finally, I recently discovered memory-profiler comes with a script called mprof, which can show you memory usage over the lifetime of your application. This can be useful if you want to see if your memory is getting cleaned up and released periodically.
Even better, using mprof is easy; just run mprof run script script_args in your shell of choice. mprof will automatically create a graph of your script's memory usage over time, which you can view by running mprof plot. It’s important to note here that plotting requires matplotlib .
Of course, I'm sure there are other approaches to profiling memory usage in Python, and I’m curious to know yours. Go ahead and take a minute to let me know your recommendations in the comments.