Python Subprocess Grep

Question

I am trying to use the grep command in a python script using the subprocess module.

Here's what I have:

userid = 'foo12'
p = subprocess.Popen(['grep', "%s *.log"%userid], stdout=subprocess.PIPE)

And it returns nothing. I am not entirely sure what I am doing wrong so can someone please explain. The current method that I am using that works is by adding the shell=true which makes it output the correct output but as the help pages have pointed out it is unsafe. I need help trying to make this work so that my script isn't unsafe.

Answer 1

I think you're running up against two problems:

1. This call:

   p = subprocess.Popen(['grep', "%s *.log"%userid]...
   

   will not work as expected without shell=True because the list of arguments are being passed directly to os.execvp, which requires each item to be a single string representing an argument. You've squished two separate arguments together into a single string (in other words, grep is interpreting "foo12 *.log" as the pattern to search, and not pattern+file list).

   You can fix this by saying:

   p = subprocess.Popen(['grep', userid, '*.log']...)
   

2. The second issue is that, again without shell=True, execvp doesn't know what you mean by *.log and passes it directly along to grep, without going through the shell's wildcard expansion mechanism. If you don't want to use shell=True, you can instead do something like:

   import glob
   args = ['grep', userid]
   args.extend(glob.glob('*.log')
   p = subprocess.Popen(args, ...)