Jeff Maxim

My experience at Recurse Center

subprocess.Popen: Using Python's Subprocess Module to Pipe Stdin and Stdout Between Processes

| Comments

Let’s say you have two programs that need to communicate with eachother via stdin and stdout. Pipes is one way to accomplish this. A pipe is a connection between two processes such that the stdout of one program becomes the stdin of the other.

With python, this piping process is streamlined with the subprocess module.

Let’s look at an example of how this plays out in python. Take our first program, foo.py:

foo.py:

import time
import sys

for i in range(0,10):
    print "Hello world " + str(i)
    sys.stdout.flush()
    time.sleep(1)

All foo.py does is print “Hello World” with an iterator number, and it does this every second. This is our sample output. Now, let’s imagine a second program that is trying to catch the stdout from foo.py as stdin. Let’s make that second program called bar.py, and it will be based on python’s subprocess module documentation:

bar.py:

import subprocess

proc = subprocess.Popen(['python','foo.py'],stdin=subprocess.PIPE)

proc.communicate()

Running bar.py produces our desired result: it catches the stdout from foo.py, uses it as stdin, and prints “Hello world” plus an iterator to the console every second.

Now, let’s say you want to catch each line of output from foo.py as a variable in bar.py, and maybe save that output to a variable in bar.py. Let’s write baz.py, based on this post from stackoverflow, to accomplish that goal:

baz.py:

import subprocess

proc = subprocess.Popen(['python','foo.py'],stdout=subprocess.PIPE)

while True:
  line = proc.stdout.readline()
  if line != '':
    print "test: ", line.rstrip()
  else:
    break

You’ll notice two key differences between bar.py and baz.py. First, baz.py uses a “stdout” arg in it’s call to subprocess.Popen, where bar.py uses a “stdin” arg here. Second, baz.py replaces bar.py’s “proc.communicate()” call with a while-block. This while-block in baz.py accomplishes our goal of collecting each line of output from foo.py in a variable. With baz.py, we just print that variable out. However, you could presumably append that line to a list and save all your line variables in this way.

Any thoughts, questions, comments?

Comments