Notes

From the quickies department, the Stubby exception generators (search for stubby) made it to Perl 5.

Posted by Pedro Melo at 10:07 AM | Permalink | |

I was checking something in the Wordpress site:

melo@MrTray:melo $ lwp-request -U -s -S -e  -m HEAD http://www.wordpress.com/
HEAD http://wordpress.com/
User-Agent: lwp-request/2.08

HEAD http://www.wordpress.com/ --> 301 Moved Permanently
HEAD http://wordpress.com/ --> 200 OK
Connection: close
Date: Thu, 08 May 2008 10:48:28 GMT
Server: nginx/0.6.29
Vary: Cookie
Content-Type: text/html; charset=UTF-8
Client-Date: Thu, 08 May 2008 10:48:36 GMT
Client-Peer: 72.232.101.43:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
X-Hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
X-Pingback: http://wordpress.com/xmlrpc.php

Nice way to do job hunting.

Posted by Pedro Melo at 11:51 AM | Permalink | |

A big part of my work is doing non-blocking servers using the AnyEvent Perl module (I used to prefer POE, but AnyEvent beats POE both in terms of performance and simplicity).

One recurring problem is using DBI inside a non-blocking server. The modules that you can find on CPAN that deal with this problem use slave processes to run the DBI code, and then a lightweight protocol to send the requests back to the master process. This way the DBI can block as much as it wants without causing problems to the non-blocking master process.

This works of course, but I would be lighter to skip all this process mumbo-jumbo if we could have a non-blocking DBI.

I browsed the DBI code and sketched a small API to allow you to execute_nb a statement and receive the results via a callback, but I slammed into problems with the DBD's.

For example, DBD::mysql uses the C-based mysqlclient. But this library is pretty much useless in terms of building a non-blocking client. It just doesn't have the required functions to allow that kind of usage.

One workaround, at least for DBD::mysql, would be to code a small threaded engine. We would use C-level threads, one for each connection, and the master Perl/XS module would communicate with them to simulate non-blocking behavior. This would not be something new, IO::AIO uses this trick with much success (see the impressive performance that you can squeeze out of projects like qpsmtpd, Perlbal, or DJabberd, all built on top of IO::AIO via Danga::Socket).

But we are back to the master-slaves approach of the current CPAN solutions, only now in-process so the communication between threads should be pretty fast and simple. And its specific to DBD::mysql. Its better but still not good enough.

I'll keep poking at this from time to time. I would like to have a solution that would only require DBI for this. My next item on the "things to try"-list is reading the code for DBD::Gofer. In itself, I don't see it as a solution, but I'm hoping to learn something that triggers a possible approach for this.

So for now, I'll keep using the master-slaves approach. While not perfect, it's good enough for now.

Update: also to check, the callback mechanism of DBI, as explained by Tim Bunce.

Posted by Pedro Melo at 03:58 PM | Permalink | |

Got to love science:

Men who said they had sex twice a week had a risk of dying half that of the less passionate participants who said they had sex once a month, Dr. Davey-Smith’s team said.

via Justin Mason.

So, print a couple of copies of the study before you leave for the pub/bar tonight. You got science on your side now.

Posted by Pedro Melo at 02:13 PM | Permalink | |

Yesterday, Rui pointed me to two applications/services that immediately found a place in my Applications folder.

The first is Yuuguu. Its a screen sharing application that just works. You install the client (Windows, Mac. You can also use the web server), create a free account and add your friends (by email address). That's it. You can now share you entire desktop with any of your friends. Essential for those support calls you get from your family members.

The second one is PortMap, a free application from the CodingMonkeys (of SubEthaEdit fame). It allows you to open services running on your desktop when you are sitting behind a NAT gateway. It will use NAT-PMP or UPNP to map a public port to your private address.

PortMap is the end user application. They also made available the Framework they created around the code that does this magic. I should point out that Leopard has this functionality built-in (point from João Pavão, the Cocoa guru at hand).

Posted by Pedro Melo at 03:36 PM | Permalink | |

I'm a big fan of DevonThink. I've used the Pro version for two or three years now, to store my web clippings, my own personal Google.

Yesterday, I read through the Evernote review at AppleInsider and it looks like it could be event better. My favorite feature is the bookmarklet that lets you select part of a web page inside Safari, and send to the Evernote server.

I love this combination: a web service to store all this data, and a companion desktop application to ingrate perfectly with my environment.

Posted by Pedro Melo at 03:26 PM | Permalink | |

I'm not a expert on MySQL, but I spent a better part of this past night optimizing a server, and I've collected some notes. This is mostly targeted at InnoDB-based tables.

To change this settings, you should edit your my.cnf and update them in the mysqld section. See the MySQL System Variables manual page for more information. Please remember to keep a good working backup of your previous my.cnf. Better yet, include your my.cnf in your source control tree. I assume that you already have tested backups of your data...

So, do this steps:

• run the tunning-primer.sh script to see how well is your server doing (or not);
• read the MySQL Performance blog articles, specially the article about choosing innodb_buffer_pool_size;
• make sure you don't double-cache with innodb_flush_method=O_DIRECT;
• make sure Linux is not swapping out your memory "just because":
  • echo 0 > /proc/sys/vm/swappiness
  • echo 'vm.swappiness = 0' >> /etc/sysctl.conf

These are the basic settings that I pay attention to. Of course there is much much more settings that you can tune (transaction-isolation and innodb_flush_log_at_trx_commit come to mind), but the things above will cover most of what you need.

After this, you should also use the mysqlreport tool. It requires more investment on your part, understanding the data, but its very thorough.

Finally, install Maatkit and get used to the tools it provides. They are essential if you are using replication.

Posted by Pedro Melo at 04:12 PM | Permalink | |

My favorite XEPs of last year where PIP (XEP-0223) and PEP (XEP-0163). These two made Publish-Subscribe (XEP-0060) trivial to implement on XMPP clients, and pave the way for rich-presence clients.

One problem was that open-source servers did not implement PIP/PEP. Recently this has changed, with both Openfire and ejabberd having decent implementations. Other servers, though, are still MIA.

Also of note, GTalk does not supported it yet. This is a bit of a problem due to his visibility in the XMPP community.

So if you do implement it in your client (and I think you should, really), it seems that you have to keep using the current protocols for avatar (for better or worse XEP-0054 plus XEP-0153) and server-side storage (XEP-0049).

There is something that servers could do to improve this situation though. There is a small change that would give client authors even more good reasons to implement PIP/PEP today.

If private storage and vCards are implemented on top of PIP and PEP, we could have the best of both worlds.

From the point-of-view of current clients, nothing will change: they will still use XEP-0049 to set and get their private storage nodes, and they will still use XEP-0054 to publish and retrieve their vCard. But at the same time, each private storage item would also be available in a PIP node, and the user vCard would be available in a PEP node.

This would allow PIP/PEP-capable clients to announce support for vcard-temp+nofity namespace in their disco response, and get instant notifications of vCard changes. Also, announce storage:bookmarks+notify and clients that only support version 1.0 of XEP-0048 will immediately support version 1.1.

This could be a easy way to increase supply of nodes, to see if we can reach critical mass sooner, in the move to a pubsub-world.

Posted by Pedro Melo at 11:41 AM | Permalink | |

In case you missed it, check your Software Update, there is a important Safari upgrade in there.

I classify this as "important" not because of the security fixes included, but because Mail.app is now working properly again.

With the last Safari 3.1, I stopped being able to paste plain text inside a mail message. I would copy something from TextMate or a Terminal window, and when I pasted it into a Mail.app message window, the line breaks would be lost.

This is most likely related to the same bug that Gruber complained about.

Anyway, Safari 3.1.1 fixes it, so I would strongly recommend that you upgrade to it.

Posted by Pedro Melo at 01:09 PM | Permalink | |

A friend of mine was complaining that he could only upload a file to my server at 2Mbytes/sec with his FIOS link at home. Some people have interesting "problems".

Anyway, I send him my /etc/sysctl.conf that I have on my Mac for quite some time without any problems, and with significant gains in network performance on my local LAN (specially for other computers with Gigabit ethernet).

net.inet.tcp.mssdflt=1440
kern.ipc.maxsockbuf=800000
net.inet.tcp.sendspace=400000
net.inet.tcp.recvspace=400000

Stick those lines in your /etc/sysctl.conf and they will be active every time your Mac reboots. Make sure you place them on all your Macs. To activate them right now, run this:

sudo sysctl -w net.inet.tcp.mssdflt=1440
sudo sysctl -w kern.ipc.maxsockbuf=800000
sudo sysctl -w net.inet.tcp.sendspace=400000
sudo sysctl -w net.inet.tcp.recvspace=400000

You should see network a better network throughput immediately.

There is a fifth setting:

net.inet.tcp.sockthreshold=0

but I admit not to fully understand the implications of his use. Apparently without it, the sendspace/recvspace settings would not work, but my tests tell me the oposite. I'll update this when I do know if/when to use the one.

For reference, a local FTP transfer to a 100Mbit server was not using all of the bandwidth. With just the first line, I was able to saturate the server link. With all the lines, over a gigabit ethernet link, I was able to reach 35Mbytes/sec between two Macs. Without: a meager 6Mbyte/sec.

All the gory details are available in a study by the Pittsburgh Computing Center.

Update: sorry, miss-typed the values, remove the extra zero.

Posted by Pedro Melo at 12:40 PM | Permalink | |

I've been a bit under the weather this week. Something inside not quite right, nothing major.

But after seeing the stupid internal Microsoft Vista SP1 video, I feel the need to rush to the bathroom and throw up. I don't believe that anybody really expect this sort of "inspirational" video to pump up your sales team, do they? It's like the monkey dance all over again.

I'm a fan of Bruce Springsteen (I was a very impressionable teenager in the 80's, so sue me) and I fell that some sort of lawsuit is in order here.

Posted by Pedro Melo at 11:31 AM | Permalink | |

iCal has an option to extract birthday dates from the Address Book and create a calendar.

Maybe I'm missing something but given that there seems to be no way to set a default alarm on those items (like 1 day before or on the day itself), isn't this a very stupid feature? I mean, I don't live inside my iCal, it might work for people who do, but I don't.

I guess I need to publish the calendar to some service that lets me add the alarms... Oh well...

Posted by Pedro Melo at 04:06 PM | Permalink | |

Andy Lester wrote an article a couple of days back about rethinking the CPAN interface. The key part of the argument is:

We don't want to "make CPAN easier to search." What we're really trying to do is help with the selection process. We want to help the user find and select the best tool for the job

I though a bit about this and my own CPAN usage over the years. I've started with Perl around 91 or 92 so I've used it a bit.

I don't know the answer for this one, but I would start with the Perl module version of the iusethis site: each person could select the list of modules that they use.

This first approach would start shuffling modules to the top.

You could then ask each person to classify why are they using the module. This is the hard part because you would need to come up with a classification scheme, like the one we still have for CPAN module, actually. It will never be perfect but I prefer to have 10 or 20 common things like "Parsing XML", or "Sending MIME email" than nothing.

An optional improvement would be to have a 4 or 5 level rank, inside your own module list, to allow you to say that, DBI is much more important than CPAN::Mini.

Another layer would be some sort of social angle. This is not to make the site look hype and fresh, but to help find new modules. I could watch a couple of persons I trust and see what are the modules they are using and the most recently added. Instead of looking through 50+ modules updates per day, I just see a filtered list of potential targets.

A final twist - a aging digg-like system: I can "This saved my bacon today"-vote on any module, but my vote would only be counted for a month, giving rise the the Bacon Savers list.

But as always, you need a carrot to make all those perl programmers compile their module list. There are two immediate carrots that I can see:

• selective CPAN announcements: if a module on your list gets an update, you would get a notification (daily email, weekly email, or personalized RSS feed);
• a automatic Bundle module, or Task module to install all your modules: I would not upload all this modules to CPAN, but we could create a perl script that downloads my Bundle or Task and calls the classic cpan to do the heavy lifting.

Anyhow, this are my €.02 to the conversation.

Posted by Pedro Melo at 08:00 AM | Permalink | |

Well, the kid is out of the bag.

Congratulations to all the the Carmo house (or at what is left of the house, thanks to the wonderful species that are contractors).

Anyway, after two kids myself, I enjoy seeing new parents, in a sadistic kind-of-way.

Unfortunately Rui has already set some pretty harsh rules, ruling out any kind of Perl adventures. Thats just not fair, he could learn so much with Uncle Melo (and now thats an even scarier picture in Rui's mind...).

Sleep, that's for kids.

Posted by Pedro Melo at 06:08 PM | Permalink | |

Remember when Plazes appears a couple of years ago? Quickly the got a lot of mac addresses geo-referenced.

Now you can go through all that work again, but this time to improve the service that Skyhook provides to his customers, like Apple with the "Locate me" button. Just fill this form and be happy.

Speaking of Plazes and mac addresses, now you can map you location using Plazes with a mac address or any other unique network identifier, like a bluetooth ID, or the mac address of an appliance or office server, or a GSM tower ID. Its all due to the love of the new plazes.net site.

I wonder if Apple will add a "Improve my location" button to the Maps application. You could tell the service, "No, I'm actually here", the iPhone/iPod Touch would scan the near by mac addresses, GSM IDs and send them to Plazes and Skyhook.

This would be an interesting App to develop with the new SDK, if anyone is looking for an idea.

Posted by Pedro Melo at 04:40 PM | Permalink | |