Author avatar

Luke Lee

Be Careful What You Import

Luke Lee

  • Apr 17, 2019
  • 5 Min read
  • 4,392 Views
  • Apr 17, 2019
  • 5 Min read
  • 4,392 Views
Data Science
Python
Introduction

Introduction

I learned a valuable lesson recently about importing into a Python application. Typically, I just find the module I need, import it, and use it without any issues. However, I found out that an unnecessary import can sometimes lead to a big waste of memory.

Searching Imports Dynamically

I encountered this problem while writing a tool that I intended to search a Python module and/or package for a given object. The script is interesting and worth another post all on its own. The gist: you provide the script with two arguments, the module/package to search, and a term to search for. The script should then give you a listing of all the places it found an object containing your search term.

For example, the documentation for PyQt4 can be pretty tough to search. I tend to use the standard Qt documentation and then translate any examples I find into code for the Python PyQt4 wrapper.

Duplicates?

I was testing my script with various terms I tend to use in PyQt4 applications such as MinimumExpanding and NoEditTriggers:

1    python import_searcher.py -s MinimumExpanding
2
3    PyQt4.Qt.QSizePolicy.MinimumExpanding = 3
4    PyQt4.QtGui.QSizePolicy.MinimumExpanding = 3
5    PyQt4.Qwt5.qplt.QSizePolicy.MinimumExpanding = 3
python

Notice anything odd about the above output? Looks like MinimumExpanding shows up in two almost identical locations -- PyQt4.Qt and PyQt.QtGui. Naturally, I thought there was a bug in my script, but after some debugging and reading I found the following jewel on the PyQt4 Wikipedia page:

The Qt module consolidates the classes contained in all of the modules described above into a single module. This has the advantage that you don't have to worry about which underlying module contains a particular class. It has the disadvantage that it loads the whole of the Qt framework, thereby increasing the memory footprint of an application. Whether you use this consolidated module or the individual component modules is down to personal taste.

So, using the PyQt4.Qt module is actually redundant and only for convenience. As a result, we actually import a lot of bulky and extraneous modules, such as QtDesigner, QtWebKit, and QtHelp.

For example, assume that your application is only using the PyQt4.QtGui module, and you mistakenly decide that you need something from PyQt4.Qt. This single import could essentially double your memory usage [1]:

1Line # Mem Usage Increment Line Contents
2================================================
3    1 @profile
4    2 def import_qt_module_by_module():
5    3 #from PyQt4 import QtCore
6    4 #from PyQt4 import QtDBus
7    5 6.953 MB 0.000 MB #from PyQt4 import QtDeclarative
8    6 12.844 MB 5.891 MB from PyQt4 import QtGui
9    7 #from PyQt4 import QtHelp
10    8 #from PyQt4 import QtMultimedia
11    9 #from PyQt4 import QtNetwork
12    10 #from PyQt4 import QtOpenGL
13    11 #from PyQt4 import QtScript
14    12 #from PyQt4 import QtScriptTools
15    13 #from PyQt4 import QtSql
16    14 #from PyQt4 import QtSvg
17    15 #from PyQt4 import QtTest
18    16 #from PyQt4 import QtWebKit
19    17 #from PyQt4 import QtXml
20    18 #from PyQt4 import QtXmlPatterns
21    19 #from PyQt4 import phonon
22    20 #from PyQt4 import QtAssitant
23    21 #from PyQt4 import QtDesigner
24    22 #from PyQt4 import QtAxContainer
25    23 19.387 MB 6.543 MB from PyQt4 import Qt

Note that just importing 'PyQt4.Qt' increased the application memory usage by 6.543 MB. This could be costly depending on how much memory your application was already using and how much your system has available. While this situation deals with smaller figures, multiple unnecessary imports can easily compound, leading to more significant memory usage issues.

Moral Takeaways

  1. Be careful of what you import. It could be redundant, costly, or both.

  2. Read the documentation carefully. The main PyQt4 documentation alludes to this redundant PyQt4.Qt module by describing it with the following, "Consolidates all other modules into a single module for ease of use at the expense of memory."

  3. Remember to try stuff, play around, and have some fun coding. A one-off script can lead to a nice, useful discovery.
LEARN MORE