Installing in Windows

For Windows, PyWin32 is a prerequisite. Follow that link and carefully read the instructions; there is a different version of PyWin32 for each version of Python. With this done you can continue to install pip using the MS-DOS command line.

However it is particularly easy to use pip-Win, which sets up both pip and virtualenv and makes it simple to install packages and to switch between different Python interpreters. (For more on the uses of virtualenv, see Supporting Multiple Platforms below.)

When pip-Win is working, enter this command in its Command field and click Run:

venv -c -i  pyi-env-name

This creates a new virtual environment rooted at C:\Python\pyi-env-name and makes it the current environment. A new command shell window opens in which you can run commands within this environment. Enter the command

pip install PyInstaller

Whenever you want to use PyInstaller,

• Start pip-Win
• In the Command field enter venv pyi-env-name
• Click Run

Then you have a command shell window in which commands execute in that environment.

General Options

What to generate

What to bundle, where to search

How to generate

Windows and Mac OS X specific options

Windows specific options

Mac OS X specific options

Setting a Custom Icon

The minimal plist provided by PyInstaller designates the icon file for the app as the icon-windowed.icns file in Resources. This is the PyInstaller logo in icns format. For now you can apply your own icon after the app is built in several ways:

• Prepare another .icns file with your own graphic, save it as icon-windowed.icns replacing the default one in Resources.
• Prepare an .icns file with your own graphic, place it in Resources and edit the Info.plist to name it.
• Prepare an .icns file with your own graphic; open in it Preview.app; select-all and copy; in the Finder, Get Info on your app; click the icon in the info display and paste.

The following programs are capable of creating .icns files from JPEG or PNG images:

• GraphicConverter ($$)
• makeicns (MIT License)
• png2icns (GPL)

Setting the Supported Document Types

You can also edit the Info.plist file to tell the Mac OS X Launcher what document types your application supports. Refer to the Mac OS developer documentation for these keywords.

Getting the Opened Document Names

When a user double-clicks a document of a type your application supports, or when a user drags a document icon and drops it on your application's icon, Mac OS X launches your application and provides the name(s) of the opened document(s) in the form of an OpenDocument AppleEvent. This AppleEvent is received by the bootloader before your code has started executing.

The bootloader gets the names of opened documents from the OpenDocument event and encodes them into the argv string before starting your code. Thus your code can query sys.argv to get the names of documents that should be opened at startup.

OpenDocument is the only AppleEvent the bootloader handles. If you want to handle other events, or events that are delivered after the program has launched, you must set up the appropriate handlers.

Supporting Multiple Python Environments

When you need to bundle your application within one OS but for different versions of Python and support libraries -- for example, a Python 3 version and a Python 2.7 version; or a supported version that uses Qt4 and a development version that uses Qt5 -- we recommend you use virtualenv. With virtualenv you can maintain different combinations of Python and installed packages, and switch from one combination to another easily.

• Use virtualenv to create as many different development environments as you need, each with its own combination of Python and installed packages.
• Install PyInstaller in each environment.
• Use PyInstaller to build your application in each environment.

Note that when using virtualenv, the path to the PyInstaller commands is:

• Windows: ENV_ROOT\Scripts
• Others: ENV_ROOT/bin

Under Windows, the pip-Win package installs virtualenv and makes it especially easy to set up different environments and switch between them. Under Linux and Mac OS, you switch environments at the command line.

Supporting Multiple Operating Systems

If you need to distribute your application for more than one OS, for example both Windows and Mac OS X, you must install PyInstaller on each platform and bundle your app separately on each.

If you share the same home directory on multiple platforms, for example Linux and OS X, you will need to set the PYINSTALLER_CONFIG_DIR environment variable to different values on each platform otherwise PyInstaller may cache files for one platform and use them on the other platform, as by default it uses a subdirectory of your home directory as its cache location.

You can do this from a single machine using virtualization. The free virtualBox or the paid VMWare and Parallels allow you to run another complete operating system as a "guest". You set up a virtual machine for each "guest" OS. In it you install Python, the support packages your application needs, and PyInstaller.

The Dropbox system is useful with virtual machines. Install a Dropbox client in each virtual machine, all linked to your Dropbox account. Keep a single copy of your script(s) in a Dropbox folder. Then on any virtual machine you can run PyInstaller thus:

cd ~/Dropbox/project_folder/src # Linux, Mac -- Windows similar
pyinstaller --workpath=path-to-local-temp-folder  \
            --distpath=path-to-local-dist-folder  \
            ...other options as required...       \
            ./myscript.py

Your bundled app is in path-to-local-dist-folder on the virtual machine's local disk. After testing it, you can compress the app to a zip file and copy it to the Public folder of your Dropbox. Your users can download it from there. (Pro tip: Do not shut down the virtual machine until Dropbox has completely uploaded the .zip to the cloud.)

It is claimed to be possible to cross-develop for Windows under Linux using the free Wine environment. Further details are needed, see How to Contribute.

TOC Class (Table of Contents)

The TOC (Table Of Contents) class is used by all of the target classes. For example, the scripts member of an Analysis object is a TOC containing a list of scripts; the pure member is a TOC with a list of modules, and so on.

Basically a TOC object contains a list of tuples of the form

(name,path,typecode)

In fact, it acts as an ordered set of tuples; that is, it contains no duplicates (where uniqueness is based on the name element of each tuple). Within this constraint, a TOC preserves the order of tuples added to it.

A TOC behaves like a list object and supports the same methods (appending, indexing, etc). A TOC also supports taking differences and intersections like a set. For these operations a simple list of tuples can be used as one argument. This makes excluding modules quite easy. For example,

a.binaries - [('badmodule', None, None)]

is an expression that yields a TOC from which any tuple named badmodule has been removed.

The right-hand argument to the subtraction operator is a list that contains one tuple in which name is badmodule and the path and typecode elements are None. (Because set membership is based on the name element of a tuple only, it is not necessary to give accurate path and typecode elements when subtracting.) So, if you modify this line in a one-folder spec file:

dist = COLLECT(..., a.binaries - [('badmodule', None, None)], ...)

or this line in a one-file spec:

exe = EXE(..., a.binaries - [('badmodule', None, None)], ...)

you remove badmodule from the output executable.

In order to add files to a TOC, you need to know the typecode values and their related path values. A typecode is a one-word string. PyInstaller uses a number of typecode values internally, but for the normal case you need to know only three:

The Tree Class

The Tree class is a way of creating a TOC that describes some or all of the files within a directory:

Tree(root, prefix=run-time-folder, excludes=match)

• The root argument is a path string to a directory. It may be absolute or relative to the build directory.
• The prefix argument, if given, is a name for a subfolder within the run-time folder to contain the tree files. If you omit prefix or give None, the tree files will be at the top level of the run-time folder.
• The excludes argument, if given, is a list of one or more strings that match files in the root that should be omitted from the Tree. An item in the list can be either:
  • a name, which causes files or folders with this basename to be excluded
  • *.ext, which causes files with this extension to be excluded

For example:

extra_tree = Tree('../src/extras', prefix='extras', excludes=['tmp'])

This creates extra_tree as a TOC object that lists all files from the relative path ../src/extras, omitting those that have the basename (or are in a folder named) tmp.

Each tuple in this TOC has:

• A typecode of DATA,
• A path consisting of a complete, absolute path to one file in the root folder,
• A name consisting of the filename of this file, or, if you specify a prefix, the name is prefix/filename.

Build-time Messages

When an Analysis step runs, it produces error and warning messages. These display after the command line if the --log-level option allows it. Analysis also puts messages in a warnings file named warn<name>.txt in the spec file's directory.

An error message appears if Analysis detects an unconditional import and the module it names cannot be found. An unconditional import is one that appears at the top level of the script, so it is certain to be executed.

A warning is given if the module named in an import cannot be found, but the import itself is conditional. An import is conditional when it appears in a function definition or in an if statement. There is a reasonable chance that such an import will not be executed, so it will not matter that the module cannot be found.

For example, os.py (which is cross-platform) works by figuring out what platform it is on, then importing and rebinding names from the appropriate platform-specific module. If your script imports os or os.path, the warning file will have lines like:

WARNING: no module named dos (conditional import by os)
WARNING: no module named ce (conditional import by os)
WARNING: no module named os2 (conditional import by os)

The analysis has detected that the import is within a conditional block (an if statement). You will know that in this system, os will never need to import the os2 module, for example, so that warning can be ignored.

Warnings may also be produced when a class or function is declared in a package (an __init__.py module), and the import specifies package.name. In this case, the analysis can't tell if name is supposed to refer to a submodule or package.

Warnings are also produced when an __import__, exec or eval statement is encountered. Either exec and eval could be used to implement a dynamic import, but normally they are used for something else. However, an __import__ warning should certainly be investigated. It probably represents a place where the script is importing code that PyInstaller cannot see.

Problems detected through these messages can be corrected; see Listing Hidden Imports below for how to do it.

Build-Time Python Errors

PyInstaller sometimes terminates by raising a Python exception. In most cases the reason is clear from the exception message, for example "Your system is not supported", or "Pyinstaller requires at least Python 2.4". Others clearly indicate a bug that should be reported.

One of these errors can be puzzling, however: IOError("Python library not found!") PyInstaller needs to bundle the Python library, which is the main part of the Python interpreter, linked as a dynamic load library. The name and location of this file varies depending on the platform in use. Some Python installations do not include a dynamic Python library by default (a static-linked one may be present but cannot be used). You may need to install a development package of some kind. Or, the library may exist but is not in a folder where PyInstaller is searching.

The places where PyInstaller looks for the python library are different in different operating systems, but /lib and /usr/lib are checked in most systems. If you cannot put the python library there, try setting the correct path in the environment variable LD_LIBRARY_PATH in Linux or DYLD_LIBRARY_PATH in OS X.

Getting Debug Messages

Giving the --debug option causes the bundled executable itself to write progress messages when it runs. This can be useful during development of a complex package, or when your app doesn't seem to be starting, or just to learn how the runtime works.

Normally the debug progress messages go to standard output. If the --windowed option is used when bundling a Windows app, they are displayed as MessageBoxes. For a --windowed Mac OS app they are not displayed.

Remember to bundle without --debug for your production version. Users would find the messages annoying.

Getting Python's Verbose Imports

You can also pass a -v (verbose imports) flag to the embedded Python interpreter (see Giving Run-time Python Options above). This can be extremely useful. It can be informative even with apps that are apparently working, to make sure that they are getting all imports from the bundle, and not leaking out to the local installed Python.

Python verbose and warning messages always go to standard output and are not visible when the --windowed option is used. Remember to not use this in the distributed program.

Extending the Path

If Analysis recognizes that a module is needed, but cannot find that module, it is often because the script is manipulating sys.path. The easiest thing to do in this case is to use the --paths= option to list all the other places that the script might be searching for imports:

pyi-makespec --paths=/path/to/thisdir \
             --paths=/path/to/otherdir myscript.py

These paths will be added to the current sys.path during analysis.

Listing Hidden Imports

If Analysis thinks it has found all the imports, but the app fails with an import error, the problem is a hidden import; that is, an import that is not visible to the analysis phase.

Hidden imports can occur when the code is using __import__ or perhaps exec or eval. You get warnings of these (see Build-time Messages).

Hidden imports can also occur when an extension module uses the Python/C API to do an import. When this occurs, Analysis can detect nothing. There will be no warnings, only a crash at run-time.

To find these hidden imports, set the -v flag (Getting Python's Verbose Imports above).

Once you know what they are, you add the needed modules to the bundle using the --hidden-import= command option, by editing the spec file, or with a hook file (see Using Hook Files below).

Extending a Package's __path__

Python allows a script to extend the search path used for imports through the __path__ mechanism. Normally, the __path__ of an imported module has only one entry, the directory in which the __init__.py was found. But __init__.py is free to extend its __path__ to include other directories. For example, the win32com.shell.shell module actually resolves to win32com/win32comext/shell/shell.pyd. This is because win32com/__init__.py appends ../win32comext to its __path__.

Because the __init__.py of an imported module is not actually executed during analysis, changes it makes to __path__ are not seen by PyInstaller. We fix the problem with the same hook mechanism we use for hidden imports, with some additional logic; see Using Hook Files below.

Note that manipulations of __path__ hooked in this way apply only to the analysis. That is, at runtime win32com.shell is resolved the same way as win32com.anythingelse, and win32com.__path__ knows nothing of ../win32comext.

Once in a while, that's not enough.

Changing Runtime Behavior

More bizarre situations can be accomodated with runtime hooks. These are small scripts that manipulate the environment before your main script runs, effectively providing additional top-level code to your script.

There are two ways of providing runtime hooks. You can name them with the option --runtime-hook=path-to-script.

Second, some runtime hooks are provided. At the end of an analysis, the names in the module list produced by the Analysis phase are looked up in loader/rthooks.dat in the PyInstaller install folder. This text file is the string representation of a Python dictionary. The key is the module name, and the value is a list of hook-script pathnames. If there is a match, those scripts are included in the bundled app and will be called before your main script starts.

Hooks you name with the option are executed in the order given, and before any installed runtime hooks. If you specify --runtime-hook=file1.py --runtime-hook=file2.py then the execution order at runtime will be:

1. Code of file1.py.
2. Code of file2.py.
3. Any hook specified for an included module that is found in rthooks/rthooks.dat.
4. Your main script.

Hooks called in this way, while they need to be careful of what they import, are free to do almost anything. One reason to write a run-time hook is to override some functions or variables from some modules. A good example of this is the Django runtime hook (see loader/rthooks/pyi_rth_django.py in the PyInstaller folder). Django imports some modules dynamically and it is looking for some .py files. However .py files are not available in the one-file bundle. We need to override the function django.core.management.find_commands in a way that will just return a list of values. The runtime hook does this as follows:

import django.core.management
def _find_commands(_):
    return """cleanup shell runfcgi runserver""".split()
django.core.management.find_commands = _find_commands

Bootloader

The bootloader prepares everything for running Python code. It begins the setup and then returns itself in another process. This approach of using two processes allows a lot of flexibility and is used in all bundles except one-folder mode in Windows. So do not be surprised if you will see your bundled app as two processes in your system task manager.

What happens during execution of bootloader:

1. First process: bootloader starts.

   1. If one-file mode, extract bundled files to temppath_MEIxxxxxx
   2. Set/unset various environment variables, e.g. override LD_LIBRARY_PATH on Linux or LIBPATH on AIX; unset DYLD_LIBRARY_PATH on OSX.
   3. Set up to handle signals for both processes.
   4. Run the child process.
   5. Wait for the child process to finish.
   6. If one-file mode, delete temppath_MEIxxxxxx.

2. Second process: bootloader itself started as a child process.

   1. On Windows set the activation context.
   2. Load the Python dynamic library. The name of the dynamic library is embedded in the executable file.
   3. Initialize Python interpreter: set PYTHONPATH, PYTHONHOME.
   4. Run python code.

Running Python code

Running Python code consists of several steps:

1. Run Python initialization code which prepares everything for running the user's main script. The initialization code can use only the Python built-in modules because the general import mechanism is not yet available. It sets up the python import mechanism to load modules from archives embedded in the executable. It also adds the attributes frozen and _MEIPASS to the sys built-in module.
2. Execute run run-time hooks: first those specified by the user, then any standard ones.
3. Install python "egg" files. When a module is part of a zip file (.egg), it has been bundled into the ./eggs directory. Installing means appending .egg file names to sys.path. Python automatically detects whether an item in sys.path is a zip file or a directory.
4. Run the main script.

Python imports in a bundled app

PyInstaller embeds compiled python code (.pyc files) within the executable. PyInstaller injects its code into the normal Python import mechanism. Python allows this; the support is described in PEP 302 "New Import Hooks".

PyInstaller implements the PEP 302 specification for importing built-in modules, importing "frozen" modules (compiled python code bundled with the app) and for C-extensions. The code can be read in ./PyInstaller/loader/pyi_importers.py.

At runtime the PyInstaller PEP 302 hooks are appended to the variable sys.meta_path. When trying to import modules the interpreter will first try PEP 302 hooks in sys.meta_path before searching in sys.path. As a result, the Python interpreter loads imported python modules from the archive embedded in the bundled executable.

This is the resolution order of import statements in a bundled app:

1. Is it a built-in module? A list of built-in modules is in variable sys.builtin_module_names.
2. Is it a module embedded in the executable? Then load it from embedded archive.
3. Is it a C-extension? The app will try to find a file with name package.subpackage.module.pyd or package.subpackage.module.so
4. Next examine paths in the sys.path (PYTHONPATH). There could be any additional location with python modules or .egg filenames.
5. If the module was not found then raise ImportError.

ZlibArchive

A ZlibArchive contains compressed .pyc or .pyo files. The PYZ class invocation in a spec file creates a ZlibArchive.

The table of contents in a ZlibArchive is a Python dictionary that associates a key, which is a member's name as given in an import statement, with a seek position and a length in the ZlibArchive. All parts of a ZlibArchive are stored in the marshalled format and so are platform-independent.

A ZlibArchive is used at run-time to import bundled python modules. Even with maximum compression this works faster than the normal import. Instead of searching sys.path, there's a lookup in the dictionary. There are no directory operations and no file to open (the file is already open). There's just a seek, a read and a decompress.

A Python error trace will point to the source file from which the archive entry was created (the __file__ attribute from the time the .pyc was compiled, captured and saved in the archive). This will not tell your user anything useful, but if they send you a Python error trace, you can make sense of it.

CArchive

A CArchive can contain any kind of file. It's very much like a .zip file. They are easy to create in Python and easy to unpack from C code. A CArchive can be appended to another file, such as an ELF and COFF executable. To allow this, the archive is made with its table of contents at the end of the file, followed only by a cookie that tells where the table of contents starts and where the archive itself starts.

A CArchive can be embedded within another CArchive. An inner archive can be opened and used in place, without having to extract it.

Each table of contents entry has variable length. The first field in the entry gives the length of the entry. The last field is the name of the corresponding packed file. The name is null terminated. Compression is optional for each member.

There is also a type code associated with each member. The type codes are used by the self-extracting executables. If you're using a CArchive as a .zip file, you don't need to worry about the code.

The ELF executable format (Windows, Linux and some others) allows arbitrary data to be concatenated to the end of the executable without disturbing its functionality. For this reason, a CArchive's Table of Contents is at the end of the archive. The executable can open itself as a binary file, seek to the end and 'open' the CArchive.

Using pyi-archive_viewer

Use the pyi-archive_viewer command to inspect any type of archive:

pyi-archive_viewer archivefile

With this command you can examine the contents of any archive built with PyInstaller (a PYZ or PKG), or any executable (.exe file or an ELF or COFF binary). The archive can be navigated using these commands:

O name

Open the embedded archive name (will prompt if omitted). For example when looking in a one-file executable, you can open the outPYZ.pyz archive inside it.

U

Go up one level (back to viewing the containing archive).

X name

Extract name (will prompt if omitted). Prompts for an output filename. If none given, the member is extracted to stdout.

Q

Quit.

The pyi-archive_viewer command has these options:

MERGE Function

A custom spec file for a multipackage bundle contains one call to the MERGE function:

MERGE(*args)

MERGE is used after the analysis phase and before EXE and COLLECT. Its variable-length list of arguments consists of a list of tuples, each tuple having three elements:

• The first element is an Analysis object, an instance of class Analysis.
• The second element is the script name (without the .py extension).
• The third element is the name for the executable (usually the same as the script).

MERGE examines the Analysis objects to learn the dependencies of each script. It modifies the total list to avoid duplication of libraries and modules. As a result the packages generated will be connected.

Example MERGE spec file

One way to construct a spec file for a multipackage bundle is to first build a spec file for each app in the package. Suppose you have a product that comprises three apps named (because we have no imagination) foo, bar and zap:

pyi-makespec options as appropriate... foo.py
pyi-makespec options as appropriate... bar.py
pyi-makespec options as appropriate... zap.py

Check for warnings and test the apps individually. Deal with any hidden imports and other problems. When all three work correctly, edit the three files foo.spec, bar.spec and zap.spec and combine them as follows. First copy the Analysis statements from each, changing them to give each Analysis object a unique name:

foo_a = Analysis(['foo.py'],
        pathex=['/the/path/to/foo'],
        hiddenimports=[],
        hookspath=None)

bar_a = Analysis(['bar.py'], etc., etc...

zap_a = Analysis(['zap.py'], etc., etc...

Now code the call to MERGE to process the three Analysis objects:

MERGE( (foo_a, 'foo', 'foo'), (bar_a, 'bar', 'bar'), (zap_a, 'zap', 'zap') )

Following this you can copy the PYZ, EXE and COLLECT statements from the original three spec files, substituting the unique names of the Analysis objects where the original spec files have a., for example:

foo_pyz = PYZ(foo_a.pure)
foo_exe = EXE(foo_pyz, foo_a.scripts, ... etc.

Save the merged spec file as foobarzap.spec and then build it:

pyi-build foobarzap.spec

There are several multipackage examples in the tests/multipackage folder of the PyInstaller distribution folder.

Remember that a spec file is executable Python. You can use all the Python facilities (for and with and the members of sys and io) in creating the Analysis objects and performing the PYZ, EXE and COLLECT statements.

Development tools

On Debian/Ubuntu systems, you can run the following to install everything required:

sudo apt-get install build-essential

On Fedora/RHEL and derivates, you can run the following:

su
yum groupinstall "Development Tools"

On Mac OS X you can get gcc by installing Xcode. It is a suite of tools for developing software for Mac OS X. It can be also installed from your Mac OS X Install DVD. It is not necessary to install the version 4 of Xcode.

On Solaris and AIX the bootloader is tested with gcc.

On Windows you can use the Visual Studio C++ compiler (Visual Studio 2008 is recommended). A free version you can download is Visual Studio Express.

Note: There is no connection between the Visual Studio version used to compile the bootloader and the Visual Studio version used to compile Python. The bootloader is a self-contained static executable that imposes no restrictions on the version of Python being used. So you can use any Visual Studio version you have around.

You can download and install or unpack MinGW distribution from one of the following locations:

• MinGW - stable and mature, uses gcc 3.4 as its base
• MinGW-w64 - more recent, uses gcc 4.4 and up.
• TDM-GCC - MinGW and MinGW-w64 installers

Building

On Windows, when using MinGW, it is needed to add PATH_TO_MINGW\bin to your system PATH. variable. In command prompt before building bootloader run for example:

set PATH=C:\MinGW\bin;%PATH%

Change to the bootloader subdirectory. Run:

python ./waf configure build install

This will produce

• ./PyInstaller/bootloader/YOUR_OS/run,
• ./PyInstaller/bootloader/YOUR_OS/run_d
• ./PyInstaller/bootloader/YOUR_OS/runw and
• ./PyInstaller/bootloader/YOUR_OS/runw_d

which are the bootloaders.

On Windows this will produce in the ./PyInstaller/bootloader/YOUR_OS directory: run*.exe (bootloader for regular programs), and inprocsrvr*.dll (bootloader for in-process COM servers).

Note: If you have multiple versions of Python, the Python you use to run waf is the one whose configuration is used.

Note: On AIX the bootloader builds with gcc and is tested with gcc 4.2.0 on AIX 6.1.

Linux Standard Base (LSB) binary

By default, the bootloaders on Linux are LSB binaries.

LSB is a set of open standards that should increase compatibility among Linux distributions. PyInstaller produces a bootloader as an LSB binary in order to increase compatibility for packaged applications among distributions.

Note: LSB version 4.0 is required for successfull building of bootloader.

On Debian- and Ubuntu-based distros, you can install LSB 4.0 tools by adding the following repository to the sources.list file:

deb http://ftp.linux-foundation.org/pub/lsb/repositories/debian lsb-4.0 main

then after having update the apt repository:

sudo apt-get update

you can install LSB 4.0:

sudo apt-get install lsb lsb-build-cc

Most other distributions contain only LSB 3.0 in their software repositories and thus LSB build tools 4.0 must be downloaded by hand. From Linux Foundation download LSB sdk 4.0 for your architecture.

Unpack it by:

tar -xvzf lsb-sdk-4.0.3-1.ia32.tar.gz

To install it run:

cd lsb-sdk
./install.sh

After having installed the LSB tools, you can follow the standard building instructions.

NOTE: if for some reason you want to avoid LSB compilation, you can do so by specifying --no-lsb on the waf command line, as follows:

python waf configure --no-lsb build install

This will also produce support/loader/YOUR_OS/run, support/loader/YOUR_OS/run_d, support/loader/YOUR_OS/runw and support/loader/YOUR_OS/runw_d, but they will not be LSB binaries.

ImportTracker

ImportTracker can be called in two ways: analyze_one(name, importername=None) or analyze_r(name, importername=None). The second method does what modulefinder does - it recursively finds all the module names that importing name would cause to appear in sys.modules. The first method is non-recursive. This is useful, because it is the only way of answering the question "Who imports name?" But since it is somewhat unrealistic (very few real imports do not involve recursion), it deserves some explanation.

analyze_one()

When a name is imported, there are structural and dynamic effects. The dynamic effects are due to the execution of the top-level code in the module (or modules) that get imported. The structural effects have to do with whether the import is relative or absolute, and whether the name is a dotted name (if there are N dots in the name, then N+1 modules will be imported even without any code running).

The analyze_one method determines the structural effects, and defers the dynamic effects. For example, analyze_one("B.C", "A") could return ["B", "B.C"] or ["A.B", "A.B.C"] depending on whether the import turns out to be relative or absolute. In addition, ImportTracker's modules dict will have Module instances for them.

Module Classes

There are Module subclasses for builtins, extensions, packages and (normal) modules. Besides the normal module object attributes, they have an attribute imports. For packages and normal modules, imports is a list populated by scanning the code object (and therefor, the names in this list may be relative or absolute names - we don't know until they have been analyzed).

The highly astute will notice that there is a hole in analyze_one() here. The first thing that happens when B.C is being imported is that B is imported and its top-level code executed. That top-level code can do various things so that when the import of B.C finally occurs, something completely different happens (from what a structural analysis would predict). But mf can handle this through its hooks mechanism.

code scanning

Like modulefinder, ImportTracker scans the byte code of a module, looking for imports. In addition it will pick out a module's __all__ attribute, if it is built as a list of constant names. This means that if a package declares an __all__ list as a list of names, ImportTracker will track those names if asked to analyze package.*. The code scan also notes the occurance of __import__, exec and eval, and can issue warnings when they are found.

The code scanning also keeps track (as well as it can) of the context of an import. It recognizes when imports are found at the top-level, and when they are found inside definitions (deferred imports). Within that, it also tracks whether the import is inside a condition (conditional imports).

Hooks

In modulefinder, scanning the code takes the place of executing the code object. ExtensionModules, of course, don't get scanned, so there needs to be a way of recording any imports they do.

Please read Listing Hidden Imports for more information.

ImportTracker goes further and allows a module to be hooked (after it has been scanned, but before analyze_one is done with it).

Warnings

ImportTracker has a getwarnings() method that returns all the warnings accumulated by the instance, and by the Module instances in its modules dict. Generally, it is ImportTracker who will accumulate the warnings generated during the structural phase, and Modules that will get the warnings generated during the code scan.

Note that by using a hook module, you can silence some particularly tiresome warnings, but not all of them.

Cross Reference

Once a full analysis (that is, an analyze_r call) has been done, you can get a cross reference by using getxref(). This returns a list of tuples. Each tuple is (modulename, importers), where importers is a list of the (fully qualified) names of the modules importing modulename. Both the returned list and the importers list are sorted.

ImportManager

ImportManager formalizes the concept of a metapath. This concept implicitly exists in native Python in that builtins and frozen modules are searched before sys.path, (on Windows there's also a search of the registry, while on Mac, resources may be searched). This metapath is a list populated with ImportDirector instances. There are ImportDirector subclasses for builtins, frozen modules, (on Windows) modules found through the registry and a PathImportDirector for handling sys.path. For a top-level import (that is, not an import of a module in a package), ImportManager tries each director on its metapath until one succeeds.

ImportManager hides the semantic complexity of an import from the directors. It is up to the ImportManager to decide if an import is relative or absolute; to see if the module has already been imported; to keep sys.modules up to date; to handle the fromlist and return the correct module object.

ImportDirector

An ImportDirector just needs to respond to getmod(name) by returning a module object or None. As you will see, an ImportDirector can consider name to be atomic - it has no need to examine name to see if it is dotted.

To see how this works, we need to examine the PathImportDirector.

PathImportDirector

The PathImportDirector subclass manages a list of names, most notably sys.path. To do so, it maintains a shadowpath, a dictionary mapping the names on its pathlist (eg, sys.path) to their associated Owners. (It could do this directly, but the assumption that sys.path is occupied solely by strings seems ineradicable.) Owners of the appropriate kind are created as needed (if all your imports are satisfied by the first two elements of sys.path, the PathImportDirector's shadowpath will only have two entries).

Owner

An Owner is much like an ImportDirector but manages a much more concrete piece of turf. For example, a DirOwner manages one directory. Since there are no other officially recognized filesystem-like namespaces for importing, that's all that's included in iu, but it's easy to imagine Owner``s for zip files (and I have one for my own ``.pyz archive format) or even URLs.

As with ImportDirectors, an Owner just needs to respond to getmod(name) by returning a module object or None, and it can consider name to be atomic.

So structurally, we have a tree, rooted at the ImportManager. At the next level, we have a set of ImportDirectors. At least one of those directors, the PathImportDirector in charge of sys.path, has another level beneath it, consisting of Owners. This much of the tree covers the entire top-level import namespace.

The rest of the import namespace is covered by treelets, each rooted in a package module (an __init__.py).

Packages

To make this work, Owners need to recognize when a module is a package. For a DirOwner, this means that name is a subdirectory which contains an __init__.py. The __init__ module is loaded and its __path__ is initialized with the subdirectory. Then, a PathImportDirector is created to manage this __path__. Finally the new PathImportDirector's getmod is assigned to the package's __importsub__ function.

When a module within the package is imported, the request is routed (by the ImportManager) directly to the package's __importsub__. In a hierarchical namespace (like a filesystem), this means that __importsub__ (which is really the bound getmod method of a PathImportDirector instance) needs only the module name, not the package name or the fully qualified name. And that's exactly what it gets. (In a flat namespace - like most archives - it is perfectly easy to route the request back up the package tree to the archive Owner, qualifying the name at each step.)

Possibilities

Let's say we want to import from zip files. So, we subclass Owner. The __init__ method should take a filename, and raise a ValueError if the file is not an acceptable .zip file. (When a new name is encountered on sys.path or a package's __path__, registered Owners are tried until one accepts the name.) The getmod method would check the zip file's contents and return None if the name is not found. Otherwise, it would extract the marshalled code object from the zip, create a new module object and perform a bit of initialization (12 lines of code all told for my own archive format, including initializing a package with its __subimporter__).

Once the new Owner class is registered with iu, you can put a zip file on sys.path. A package could even put a zip file on its __path__.

Compatibility

This code has been tested with the PyXML, mxBase and Win32 packages, covering over a dozen import hacks from manipulations of __path__ to replacing a module in sys.modules with a different one. Emulation of Python's native import is nearly exact, including the names recorded in sys.modules and module attributes (packages imported through iu have an extra attribute - __importsub__).

Performance

In most cases, iu is slower than builtin import (by 15 to 20%) but faster than imputil (by 15 to 20%). By inserting archives at the front of sys.path containing the standard lib and the package being tested, this can be reduced to 5 to 10% slower (or, on my 1.52 box, 10% faster!) than builtin import. A bit more can be shaved off by manipulating the ImportManager's metapath.

Limitations

This module makes no attempt to facilitate policy import hacks. It is easy to implement certain kinds of policies within a particular domain, but fundamentally iu works by dividing up the import namespace into independent domains.

Quite simply, I think cross-domain import hacks are a very bad idea. As author of the original package on which PyInstaller is based, McMillan worked with import hacks for many years. Many of them are highly fragile; they often rely on undocumented (maybe even accidental) features of implementation. A cross-domain import hack is not likely to work with PyXML, for example.

That rant aside, you can modify ImportManger to implement different policies. For example, a version that implements three import primitives: absolute import, relative import and recursive-relative import. No idea what the Python syntax for those should be, but __aimport__, __rimport__ and __rrimport__ were easy to implement.

iu Usage

Here's a simple example of using iu as a builtin import replacement.

>>> import iu
>>> iu.ImportManager().install()
>>>
>>> import DateTime
>>> DateTime.__importsub__
<method PathImportDirector.getmod
  of PathImportDirector instance at 825900>
>>>