Critique my async
mikael last edited by
While this is not Pythonista specific, I use it with Pythonista, and would be happy to get some feedback on it. Please find the file on Github if you want to play with it.
Note that while this is working code that I use in my (simple) projects, it is work in progress, and is mainly intended to act as a concrete expression of my need for simpler async coding.
yieldas a synchronization point
Functions and generators with
genrdecorator are executed in a new thread, with a special handling of the
yields. For example, in the following:
@genr def example1(): task1() task2() yield task3()
Tasks 1 and 2, also decorated with
genr, are executed in parallel.
yieldwaits until they are both complete, after which task 3 gets to run.
End of the function also acts as a collection point, i.e. we wait for all called functions to complete before returning. Thus, in the example above, there is little practical difference whether
task3is executed in a separate thread or not.
yieldto get results
Decorated functions return a
Future, which can be used in the usual way, but for more readable code, we use a Tornado-inspired way of
yielding the result. The classical example below demonstrates retrieving several web pages in parallel:
@genr def example2(): return yield [ fetch(url) for url in ( 'https://python.org', 'https://pypi.org' )] @genr def fetch(url): return requests.get(url).text
Ad hoc threads
Being a decorator,
genrcan be used to start parallel execution of individual functions inline, with the following syntax:
genr(func_name)(param1, param2, ...)
While this avoids having to create named one-line functions just for the sake of parallel execution, the syntax is not especially readable, and quickly becomes untenable if you need to add timeouts, exception management etc.
genrplays nice with being
KeyboardInterrupted. To wait for a
genrthread to complete in the main thread, you can simply wait for the result:
- Minimum cognitive load, no need to even understand the concept of a future.
- Move quickly from need to implementation without needing to understand asyncio.
- For relatively easy cases, functional need for parallel execution translates into a clear story in a single main function.
- One decorator for the different use cases:
- Launching a single parallel function
- Multiple parallel threads
- Collecting results
- Likely to suffer from the same challenge that drove the creation of asyncio: it is not very visible whether a function is being called in a separate thread or not, which can lead to hard-to-see bugs.
- Overloads generator semantics in a way that is not Pythonic.
mikael last edited by
One thing that I am currently trying to understand is how exactly is this different from the deprecated
asyncio.coroutine, and why was it deprecated in the first place.
mikael last edited by mikael
Hmm, so, no interest in this topic. Would be very interesting to hear whether it is because:
a) you find existing thread or asyncio support straightforward to use in a way that fills your needs,
b) you do not really need async in your projects,
c) you deliberately avoid async because it always leads to problems, or
d) other, please specify.
cvp last edited by cvp
@mikael I've never used asyncio, I really don't know what it is, thus I'm not able to criticize anything. I often use threading, is that far from asyncio?
pulbrich last edited by
@mikael Thanks for this, you awoke my interest and I started reading up on the topic. But I cannot (yet) contribute to the discussion.
cvp last edited by
To not die silly, I read this, interesting
mithrendal last edited by
I like the intuitive and simplistic API approach as always when mikael comes out with a new thing ;-) Next time I have a need for parallel execution in my code I will definitly come back to genr ...