Python Scripting within a Python Script

UPDATE: In order to use reload in Python 3, it must be imported from the imp built-in module. Thus, some of the examples below will require the line from imp import reload in order to work in Python 3.

I’ve been mulling over an idea for a little while now, but didn’t have the chance to do anything with it until today. Turns it it was way easier than I thought it would be. For reasons I won’t go into here, I wanted to have a Python program which could write and run other Python programs. Clearly Python can write programs (since it can save text as a .py file), and it can easily run programs just by importing a module. The trouble was editing those files and running the updated code in the master program without having to restart the master program. Here’s how I did it.

The Problem

When the Python interpreter runs a script, it compiles the code into Python bytecode to run on the Python virtual machine. It also compiles all the imported modules. Thus, you cannot change the imported modules at any point and see the changes without stopping the original program and starting it back up again. To demonstrate, try the following. Write a simple script that prints something to the screen.

print("Test")

(This is the standard syntax for Python 3, which works in Python 2.5 as well.)

Save this as “test.py”. Start Python’s interactive mode and do the following.

>>> import test
Test

While interactive mode is still running, edit the source file to print something else, for instance, “Cahoots”, then save it. Go back to interactive mode and import ‘test’ again. It will still print out ‘Test’.

The reason this happens is because Python makes a dictionary of all the modules that are currently loaded. Since you have already imported the test module, it does not re-import it even when you invoke the import statement.

The simple solution to this is to use the reload function. In this case:

>>> reload(test)
Cahoots

The Deeper Problem and Its Solution

My problem was that the user gets to specify which program to run. I cannot specify a particular import statement at design time. Fortunately Python has a built-in __import__ function which can take a module name as a string and import it. For instance,

>>> __import__('test')

This will load the test module just like import test does. But if I change test, I can’t use this statement a second time just like I can’t use the regular ‘import’ a second time.

Fortunately, __import__ returns the module it imports. That means we can say something like this:

>>> x = __import__('test')

This allows us to use the reload statement. When we make a change, all we have to do is invoke ‘reload(x)’.

This will let the user import a module at runtime, then make a change to it and re-import it.

Addendum

Suppose, however, that one wants to reload a user-specified module without having to store it as a variable. Never fear. Python’s system module has a dictionary of all the modules that are currently loaded. Just do the following in interactive mode (or a script file, if you like):

>>> __import__('test')
Test

Then change what prints out from ‘Test’ to ‘Cahoots’.

>>> __import__('test')
Test
>>> import sys
>>> test_module = sys.modules['test']
>>> reload(test_module)
Cahoots

sys.modules is a dictionary of all the currently loaded modules. If you want, you can call del sys.modules['test'], then simply re-import the test module, but that is a little dangerous and not as transparent as simply reloading the module.