Support some degree of live python scripting

[amorenoz]

After chatting with @igsilya and others at OVS/OVN Conference, some feedback we got is:
1 - It'd be nice to be able to run python scripts at collect time
2 - It'b be nice to be able to ease the execution of python scripts by just using UNIX pipes

Doing 1 by embedding the python executor in the collect main loop (which runs to completion) can introduce latency that may lead to increased event loss. So we'd need to split "printing/processing" and collection into different threads.

However, 2 actually makes it possible to do 1 easier by making collect and python two different processes, "just" paying the cost of {,un}marshaling events with minimal changes to the core event loop. Let's imagine we could:

retis collect [...] --format json | retis python myscript.py

[igsilya]

Hmm. Is there a real difference though in terms of potential for event dropping? The pipe writes are blocking once the buffer is filled up. And if you make them non-blocking then you'll still drop events, unless they are buffered somewhere else.

[amorenoz]

Having multi-threaded loop to avoid blocking on I/O is something that we'll need to tackle eventually but it's probably a relatively big work. This problem exists today as well, of course, whether it's writing to terminal, file or both, it can delay event processing.

Embedding the python interpreter in the event processing loop adds a potentially unbound delay with no buffering unless we implement it internally. Using a pipe gives us a buffer and parallel event processing "for free" with minimal changes in retis.

I'm not saying we won't embed the python interpreter in the event loop, we've wanted to add custom event formatting for some time, but right now enabling piping is a very cheap way making this "work". Note "some degree of" in the title :-)

[westurner]

• NFQUEUE ("nfq")
  • https://wiki.nftables.org/wiki-nftables/index.php/Queueing_to_userspace
  • https://github.com/chifflier/nfqueue-bindings :

    nfqueue-bindings was written to provide an interface in high-level languages such as Perl or Python to libnetfilter_queue.