How does Python's Twisted Reactor work?

Question

Recently, I've been diving into the Twisted docs. From what I gathered, the basis of Twisted's functionality is the result of it's event loop called the "Reactor". The reactor listens for certain events and dispatches them to registered callback functions that have been designed to handle these events. In the book, there is some pseudo code describing what the Reactor does but I'm having trouble understanding it, it just doesn't make any sense to me.

 while True:
     timeout = time_until_next_timed_event()
     events = wait_for_events(timeout)
     events += timed_events_until(now())
     for event in events:
         event.process()

What does this mean?

Answer 1

In case it's not obvious, It's called the reactor because it reacts to things. The loop is how it reacts.

One line at a time:

while True:

It's not actually while True; it's more like while not loop.stopped. You can call reactor.stop() to stop the loop, and (after performing some shut-down logic) the loop will in fact exit. But it is portrayed in the example as while True because when you're writing a long-lived program (as you often are with Twisted) it's best to assume that your program will either crash or run forever, and that "cleanly exiting" is not really an option.

     timeout = time_until_next_timed_event()

If we were to expand this calculation a bit, it might make more sense:

def time_until_next_timed_event():
    now = time.time()
    timed_events.sort(key=lambda event: event.desired_time)
    soonest_event = timed_events[0]
    return soonest_event.desired_time - now

timed_events is the list of events scheduled with reactor.callLater; i.e. the functions that the application has asked for Twisted to run at a particular time.

     events = wait_for_events(timeout)

This line here is the "magic" part of Twisted. I can't expand wait_for_events in a general way, because its implementation depends on exactly how the operating system makes the desired events available. And, given that operating systems are complex and tricky beasts, I can't expand on it in a specific way while keeping it simple enough for an answer to your question.

What this function is intended to mean is, ask the operating system, or a Python wrapper around it, to block, until one or more of the objects previously registered with it - at a minimum, stuff like listening ports and established connections, but also possibly things like buttons that might get clicked on - is "ready for work". The work might be reading some bytes out of a socket when they arrive from the network. The work might be writing bytes to the network when a buffer empties out sufficiently to do so. It might be accepting a new connection or disposing of a closed one. Each of these possible events are functions that the reactor might call on your objects: dataReceived, buildProtocol, resumeProducing, etc, that you will learn about if you go through the full Twisted tutorial.

Once we've got our list of hypothetical "event" objects, each of which has an imaginary "process" method (the exact names of the methods are different in the reactor just due to accidents of history), we then go back to dealing with time:

     events += timed_events_until(now())

First, this is assuming events is simply a list of an abstract Event class, which has a process method that each specific type of event needs to fill out.

At this point, the loop has "woken up", because wait_for_events, stopped blocking. However, we don't know how many timed events we might need to execute based on how long it was "asleep" for. We might have slept for the full timeout if nothign was going on, but if lots of connections were active we might have slept for effectively no time at all. So we check the current time ("now()"), and we add to the list of events we need to process, every timed event with a desired_time that is at, or before, the present time.

Finally,

     for event in events:
         event.process()

This just means that Twisted goes through the list of things that it has to do and does them. In reality of course it handles exceptions around each event, and the concrete implementation of the reactor often just calls straight into an event handler rather than creating an Event-like object to record the work that needs to be done first, but conceptually this is just what happens. event.process here might mean calling socket.recv() and then yourProtocol.dataReceived with the result, for example.

I hope this expanded explanation helps you get your head around it. If you'd like to learn more about Twisted by working on it, I'd encourage you to join the mailing list, hop on to the IRC channel, #twisted to talk about applications or #twisted-dev to work on Twisted itself, both on Freenode.