Async DBI
A big part of my work is doing non-blocking servers using the AnyEvent Perl module (I used to prefer POE, but AnyEvent beats POE both in terms of performance and simplicity).
One recurring problem is using DBI inside a non-blocking server. The modules that you can find on CPAN that deal with this problem use slave processes to run the DBI code, and then a lightweight protocol to send the requests back to the master process. This way the DBI can block as much as it wants without causing problems to the non-blocking master process.
This works of course, but I would be lighter to skip all this process mumbo-jumbo if we could have a non-blocking DBI.
I browsed the DBI code and sketched a small API to allow you to execute_nb a statement and receive the results via a callback, but I slammed into problems with the DBD's.
For example, DBD::mysql uses the C-based mysqlclient. But this library is pretty much useless in terms of building a non-blocking client. It just doesn't have the required functions to allow that kind of usage.
One workaround, at least for DBD::mysql, would be to code a small threaded engine. We would use C-level threads, one for each connection, and the master Perl/XS module would communicate with them to simulate non-blocking behavior. This would not be something new, IO::AIO uses this trick with much success (see the impressive performance that you can squeeze out of projects like qpsmtpd, Perlbal, or DJabberd, all built on top of IO::AIO via Danga::Socket).
But we are back to the master-slaves approach of the current CPAN solutions, only now in-process so the communication between threads should be pretty fast and simple. And its specific to DBD::mysql. Its better but still not good enough.
I'll keep poking at this from time to time. I would like to have a solution that would only require DBI for this. My next item on the "things to try"-list is reading the code for DBD::Gofer. In itself, I don't see it as a solution, but I'm hoping to learn something that triggers a possible approach for this.
So for now, I'll keep using the master-slaves approach. While not perfect, it's good enough for now.
Update: also to check, the callback mechanism of DBI, as explained by Tim Bunce.
