How to write to a file using non blocking IO?

Question

I want to write to a file using a non-blocking method in Python. On some googling, I found that the language supports fcntl in order to do so, but the method to implement the same is not very clear to me.

This is the code snippet (I don't know where I am going wrong):

import os, fcntl
nf = fcntl.fcntl(0,fcntl.F_UNCLK)
fcntl.fcntl(0,fcntl.F_SETFL , nf | os.O_NONBLOCK )
nf = open ("test.txt", 'a') 
nf.write ( " sample text \n")

Is this the correct way to perform a non-blocking IO operation on a file? I doubt it. Also, could you suggest any other modules in Python which allow me to do so?

Answer 1

This is how you turn non-blocking mode on for a file in UNIX:

fd = os.open("filename", os.O_CREAT | os.O_WRONLY | os.O_NONBLOCK)
os.write(fd, "data")
os.close(fd)

On UNIX, however, turning on non-blocking mode has no visible effect for regular files! Even though the file is in non-blocking mode, the os.write call won't return immediately, it will sleep until the write is complete. To prove it to yourself experimentally, try this:

import os
import datetime

data = "\n".join("testing\n" * 10 for x in xrange(10000000))
print("Size of data is %d bytes" % len(data))

print("open at %s" % str(datetime.datetime.now()))
fd = os.open("filename", os.O_CREAT | os.O_WRONLY | os.O_NONBLOCK)
print("write at %s" % str(datetime.datetime.now()))
os.write(fd, data)
print("close at %s" % str(datetime.datetime.now()))
os.close(fd)
print("end at %s" % str(datetime.datetime.now()))

You'll notice that the os.write call does take several seconds. Even though the call is non-blocking (technically, it's not blocking, it's sleeping), the call is not asynchronous.

---

AFAIK, there is no way to write to a file asynchronously on Linux or on Windows. You can simulate it, however, using threads. Twisted has a method named deferToThread for this purpose. Here's how you use it:

from twisted.internet import threads, reactor

data = "\n".join("testing\n" * 10 for x in xrange(10000000))
print("Size of data is %d bytes" % len(data))

def blocking_write():
    print("Starting blocking_write")
    f = open("testing", "w")
    f.write(data)
    f.close()
    print("End of blocking_write")

def test_callback():
    print("Running test_callback, just for kicks")

d = threads.deferToThread(blocking_code)
reactor.callWhenRunning(cc)
reactor.run()