Python Subprocess

Suprocess

Although subprosses in Python are incredible useful and I use them often to interact with command-line applications, I often fail to know exactly which of the subprocess methods to use in each situation. I'll try to make it more understandable to myself and hopefully to you in this extended recipe.

A lot of the information collected here comes from http://jimmyg.org/blog/2009/working-with-python-subprocess.html , which offers a great introduction to both command-line and suprocess module. As well as from Python 2.7 official documentation on subprocess, from which some chunks of text were copied from.

To start working with the subprocesses import the module. import subprocess

Suprocess can be used in two ways: * using convenience functions: call, check_call and check_output * or Popen interface, which allows a great customization, being able to replicate the behavior of any of the convenience functions.

subprocess args

In a subprocess args are the heart of the call. They are essential the command you'd run in shell, but now you are using Python to launch them.

Args can either be a string of a sequence of arguments. A sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).

To facilitate the creation of a sequence of arguments from a string, you can import the module shlex to help with that task.

    
>>> import shlex
>>> shlex.split('echo "Hello Process!" ')
['echo', 'Hello Process!']

Convenience Functions

subprocess.call

Runs a command with arguments, wait for it to complete, then returns the returncode.

    
>>> subprocess.call(firefox') #launch firefox
>>> subprocess.call(['touch', 'foo'], shell=False) #touch file foo in the current dir
0
# same call could performed with Popen
>>> subprocess.Popen(firefox')
>>> subprocess.Popen(['touch', 'foo'], shell=False)

check_output

Run command with arguments and return its output as a string.

>>> cmd = shlex.split('figlet "hello process"') #ascii art
>>> p=subprocess.check_output(cmd)
> " _          _ _                                           \n| |__   ___| | | ___    _ __  _ __ ___   ___ ___  ___ ___ \n| '_ \\ / _ \\ | |/ _ \\  | '_ \\| '__/ _ \\ / __/ _ \\/ __/ __|\n| | | |  __/ | | (_) | | |_) | | | (_) | (_|  __/\\__ \\__ \\\n|_| |_|\\___|_|_|\\___/  | .__/|_|  \\___/ \\___\\___||___/___/\n                       |_|                                \n"

Popen

The Popen class offers the flexibility to handle less common cases not covered by the convenience functions.

Using Popen

Execute process

>>> cmd = shlex.split('ls -trl')
>>> p=subprocess.Popen(cmd)

getting subprocess stdout value

    
>>> p=subprocess.Popen('ls ~/Downloads', shell=True, stdout=subprocess.PIPE) 
>>> p1.communicate()
("7reader_Wikipedia.pdf\nbin-bash_004.png\ncomputerlib.jpg\nDeutsch-Gothic\nDeutsch-Gothic.zip\n, None)

piping stdout to another process

# in this example will be piping the stdout of echo to the stdin of sed
# the goal is to replace every 'e' resulting from echo stdout by an '3'
# and getting the output
>>> p=subprocess.Popen("echo Hello e | sed 's/e/3/g'", shell=True, stdout=subprocess.PIPE)
>>> p.communicate()
('H3llo 3\n', None)

>>> cmd = shlex.split('echo "hello process"')
>>> cmd_sed = shlex.split("sed 's/e/3/g'")
>>> p1 = subprocess.Popen(cmd, stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd_sed, stdin=p1.stdout,stdout=subprocess.PIPE)
>>> p2.communicate()
('h3llo proc3ss\n', None)

• stdout=subprocess.PIPE - open pipe to stdout of p1
• stdin=p1.stdout - the stdout from p1 subprocess is the stdin of p2
• stdout=subprocess.PIPE- open pipe to stdout of p2

Popen options - explained

stdin, stdout and stderr

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively.

shell True

If shell is True, the specified command will be executed through the shell. It offers access to shell features such as shell pipes, filename wildcards, environment variable expansion

# echo "hi" pipe it to figlet and write the result to foo.txt file
>>> subprocess.Popen("echo hi | figlet > foo.txt", shell=True)
Out[75]: <subprocess.Popen at 0x7fb5f797d810>

On Unix-based system subprocess uses /bin/sh shell. To change the default shell you can set the argument shell=True to define the shell with the executable argument

>>> subprocess.Popen('echo "Hello world!"', shell=True, executable="/bin/bash")

Hello world! <subprocess.Popen object at 0x...>

stdout

stdout is the output that is written by a shell program to the terminal emulator. That is why you are able to see the outcome of the ran commands. If you want to read the output inside your Python script, and maybe store it in a variable, you need to set the stdout argument in the initial call to Popen, specifying that a pipe to the standard stream should be opened: stdout=subprocess.PIPE

    
>>> p=subprocess.Popen("echo 'Hello ee'|sed 's/e/3/g'", shell=True, stdout=subprocess.PIPE)
>>> p.communicate() #read the output from the pipe w/ the communicate() method

('H3llo 33\n', None) The communicate() method returns a tuple with two values, containing the data from stdout and the data from stderr two values.

stdin

stdin is the stream of data (often text) that is written to the program; As "Hello World" is written to echo in echo "Hello World".
In order to write to shell program you have to open a pipe to stdin. stdin=subprocess.PIPE

     
>>> p = subprocess.Popen('figlet', stdin=subprocess.PIPE)
>>> myvar = "foo"
>>> p.communicate(myvar)

_ _ | |__ (_) | '_ \| | | | | | | |_| |_|_| (None, None)

Both stdin and stdout pipes can be open in the same subprocess

    
>>> p = subprocess.Popen(shlex.split("sed 's/e/3/g'"), stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> c = p.communicate("bee")
>>> print c

('b33', None)

stderr

stderr consist on the stream of error messages from a program.

    
>>> p=subprocess.Popen("echo 'Hello ee'|sedx 's/e/3/g'", shell=True, stdout=subprocess.PIPE, >>> stderr=subprocess.PIPE) #will create error: there is no sedx application
>>> p.communicate()

(, '/bin/sh: 1: sedx: not found\n')

wait and poll

When you launch a subprocess, the parent process - the python script - doesn't wait for the child processes to finish execution.

If you want the script to wait for the termination of the subprocess, you need to rely on poll() or wait(), to return the value 0, indicating the exit status.

Popen.poll() Check if child process has terminated. Popen.wait() Wait for child process to terminate.

Examples

If for instances you want to run a child process that takes time, such adding a book to Calibre's library, the parent process (script) does not wait for the child process to ends, as in the example below.

#!/usr/bin/env python
import subprocess, shlex 
book = ("/home/andre/Downloads/XML-Wrox.epub")

cmd_addbook = shlex.split('calibredb add --library=/home/andre/CalibreLibrary {var_book}'.format(var_book=book))
addbook = subprocess.Popen(cmd_addbook) #add book to calibre library

ps = subprocess.Popen('ps', stdout=subprocess.PIPE) #list the running processes
ps_output = ps.communicate()[0]
print ps_output

$ python addbook.py

PID TTY          TIME CMD
16246 pts/11   00:00:00 bash
18667 pts/11   00:00:00 python
18668 pts/11   00:00:00 calibredb
18669 pts/11   00:00:00 ps

Backing up metadata
Added book ids: 1188
Notifying calibre of the change

Notice that ps_output is printed before calibredb has finnished its process - adding the book and calibredb;

If you want ps to wait until the previous subprocess is finnished , you can use Popen.wait().

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import subprocess, shlex 
book = ("/home/andre/Downloads/XML-Wrox.epub")

cmd_addbook = shlex.split('calibredb add --library=/home/andre/CalibreLibrary {var_book}'.format(var_book=book))
addbook = subprocess.Popen(cmd_addbook) #add book to calibre library

addbook.wait() # WAIT for addbook subprocess to be completed

ps = subprocess.Popen('ps', stdout=subprocess.PIPE) #list the running processes
ps_output = ps.communicate()[0]
print ps_output

$ python addbook.py

Backing up metadata
Added book ids: 1189
Notifying calibre of the change
PID TTY          TIME CMD
16246 pts/11   00:00:00 bash
19557 pts/11   00:00:00 python
19570 pts/11   00:00:00 ps

Notice the addbook.wait() forced the script to wait until the addbook subprocess was completed.