Python Subprocess: Too Many Open Files

Question

I am using subprocess to call another program and save its return values to a variable. This process is repeated in a loop, and after a few thousands times the program crashed with the following error:

Traceback (most recent call last):
  File "./extract_pcgls.py", line 96, in <module>
    SelfE.append( CalSelfEnergy(i) )
  File "./extract_pcgls.py", line 59, in CalSelfEnergy
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
  File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1166, in _execute_child
    errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 24] Too many open files

Any idea how to solve this issue is much appreciated!

Code supplied from comments:

cmd = "enerCHARMM.pl -parram=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()

Answer 1

In Mac OSX (El Capitan) See current configuration:

#ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

Set open files value to 10K :

#ulimit -Sn 10000

Verify results:

#ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10000
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited