Notes

Tarpipe is a wonderful service to execute a workflow of web services. You create your workflow visually, and then you send an email or POST to a API end-point to run it.

For example, this is the workflow I use to post to Twitter, Jaiku, and Pownce:

Usually I would send an email and my subject would be posted, but I also have an IM client always available.

So I wrote this XMPP-to-Tarpipe gateway. Its simple to use:

• use Tarpipe to create an workflow: you must use the REST Decoder;
• When you save the workflow, you'll be given a API token, a 32 char hex-string;
• subscribe <the API token>@tarpipe.simplicidade.org. You can subscribe from multiple JIDs.

That's it: you now have a new buddy on your roster. Any message sent to him (apart from help) will trigger the Tarpipe workflow.

Have fun.

Posted by Pedro Melo at 12:36 PM | Permalink | Comments (view)

New location: the How-to is now live at http://www.simplicidade.org/how-to/mysql-master-master/.

In the past, I needed to use a MySQL master/master setup for a client. At the time, I got it to work, but forgot to take notes about the process.

In the last days, a friend asked my for help to do a setup like that again, to use with a Tigase XMPP server.

So I wrote this how-to, complete with configuration files, sample data and schemas, that walks you step-by-step through the process of setting up a pair MySQL servers in a master/master configuration.

You can fetch the entire how-to with git:

git clone git://github.com/melo/mysql-master-master-how-to.git

The how-to is written in MultiMarkDown, but a simple Markdown should also format it correctly.

A tarball with everything is also available.

Please let me know if you find any mistakes. Pull requests are the preferred form of collaboration :).

Update: included some tweaks from dbr, and added a HTML version with proper CSS. Download again, and look at how-to.html.

Posted by Pedro Melo at 03:11 AM | Permalink | Comments (view)

cheat is a very nice command line cheat sheet system.

chit is a git-powered version, that even supports private cheats or project level cheats distributed using the git system.

I've added a bash completion script to chit. It's very simple but it works for me. I've also sent a pull request to robin, so you might see it in the main chit repo in the future.

Update: you can find the bash completion script in the master chit repo now.

Posted by Pedro Melo at 12:54 PM | Permalink | Comments (view)

I've moved the Search in Project With ack TextMate command to GitHub.

In the process I've included a change by Corey Jewett to respect the selected files in the project drawer, and I've also added support for the TM_ACK_COMMAND_PATH variable, in case your ack copy is somewhere strange.

There is an open TODO (catch popen errors if the ack command is not found), but I need to freshen up my ruby skill. It should be trivial, push requests welcome.

This is a very simple command. If you are looking for something more complete check out the ack TextMate bundle by Trevor Squires.

Posted by Pedro Melo at 11:32 AM | Permalink | Comments (view)

I've started to publish my ~/bin script stash at GitHub.

Not all of them are written by me. For example, I've included Gruber's TitleCase.pl script under a new name.

Before you ask about the strange long names, my memory sucks, so I use tab completion a lot. I name all my script with a x- prefix so that I can just type x-<TAB> and see my entire script repository. I use long and (I hope) self describing names.

Common scripts that I use every day get single line shortcuts using shell aliases.

To use, just clone the repository, and add the bin/ to your PATH.

Patches welcome. Recommended workflow:

1. sign up at GitHub;
2. fork my project (small fork icon);
3. clone your fork to your local workstation;
4. hack, hack, hack;
5. git-push
6. hop onto GitHub and send me a pull§ request.

Posted by Pedro Melo at 07:52 AM | Permalink | Comments (view)

A big part of my work is doing non-blocking servers using the AnyEvent Perl module (I used to prefer POE, but AnyEvent beats POE both in terms of performance and simplicity).

One recurring problem is using DBI inside a non-blocking server. The modules that you can find on CPAN that deal with this problem use slave processes to run the DBI code, and then a lightweight protocol to send the requests back to the master process. This way the DBI can block as much as it wants without causing problems to the non-blocking master process.

This works of course, but I would be lighter to skip all this process mumbo-jumbo if we could have a non-blocking DBI.

I browsed the DBI code and sketched a small API to allow you to execute_nb a statement and receive the results via a callback, but I slammed into problems with the DBD's.

For example, DBD::mysql uses the C-based mysqlclient. But this library is pretty much useless in terms of building a non-blocking client. It just doesn't have the required functions to allow that kind of usage.

One workaround, at least for DBD::mysql, would be to code a small threaded engine. We would use C-level threads, one for each connection, and the master Perl/XS module would communicate with them to simulate non-blocking behavior. This would not be something new, IO::AIO uses this trick with much success (see the impressive performance that you can squeeze out of projects like qpsmtpd, Perlbal, or DJabberd, all built on top of IO::AIO via Danga::Socket).

But we are back to the master-slaves approach of the current CPAN solutions, only now in-process so the communication between threads should be pretty fast and simple. And its specific to DBD::mysql. Its better but still not good enough.

I'll keep poking at this from time to time. I would like to have a solution that would only require DBI for this. My next item on the "things to try"-list is reading the code for DBD::Gofer. In itself, I don't see it as a solution, but I'm hoping to learn something that triggers a possible approach for this.

So for now, I'll keep using the master-slaves approach. While not perfect, it's good enough for now.

Update: also to check, the callback mechanism of DBI, as explained by Tim Bunce.

Posted by Pedro Melo at 03:58 PM | Permalink | Comments (view)

Last February, Google's Android Team shocked the XMPP world by having the gall of saying that they where moving to a Binary encoding of their GTalk API in the M5 release.

Of course this sent shock waves through the XMPP community, not because they where totally wrong, but because they didn't have the vision to solve it.

We of course understand that they have their hands full (of vapor, some bad mouths would say...) getting the platform in shape for the late 2008 release, and the XMPP.. err, sorry, GTalk protocol is not a top priority.

Well, today Android users around the world can rejoice because once again, the XMPP Standards Foundation has risen to the challenge and unveils to the world the solution to the "verboseness" problem, by introducing the breakthrough Binary XMPP protocol.

This has been in development for quite some time, taking long hours of (sometimes) heated discussion. The goal is to have something that nobody could ever accuse of being verbose (the symbol count is extremely low, and the meaning is clear from the start), but at the same time remain compressible for those extreme cases where bandwidth is at a premium (we tested, and we have code to prove it: a Binary XMPP stream can be compressed to 2% of its original size). We know of clients that live in five-sides-shaped-buildings that, with their hard 9600 baud limits, will be the first to use this.

Its been a great ride, and it feels me with immense joy to be able to put my small signature as a co-author of this spec. Of course, my part is extremely small when you have the documentation genius of Peter Saint-Andre, and Fabio Forno strict guidelines and requirements, as co-authors. Its been a pleasure to work with both on this. And I should also thanks the important Kevin Smith contributions.

And now time to rest. SRV records are up at simplicidade.org domain, and Fabio's CM is running also.

PS: and yes, now you know why the prolonged silence around here.
PPS: for those who have moved on, the code is also available on GitHub.

Posted by Pedro Melo at 12:44 AM | Permalink | Comments (view)

If you want to try some agent-love, then let me introduce you to Alter Ego, your friendly pluggable agent.

Alter Ego is a XMPP agent, where everything is a plugin. You can clone the current version with git (recommended, easy to keep track of new features and fixes) or use the tarballs.

Right now, you can find it running under my own JID, so yes, I trust it enough to run it on my main account.

The only plugin I wrote is a User Location (XEP-0080) updater. With all the cool geo toys (like Fire Eagle) around, I wanted to play a bit with location-aware services. So the current UserLocation plugin does a very simple thing:

• discovers the mac address of your default gateway;
• searches the mac address in a local database;
• if found, he publish the information to your User Location PEP node.

So you do need a PEP-enabled server like ejabberd 2.0 or OpenFire.

Also, nothing prevents you from using other sources of location like a GPS. For example, I'll probably include something like this perl-GPS simple example in a future version of the plugin.

You can look at the roadmap to see what I'll be working on, or you can subscribe any of the zillions of feeds that github provides. I usually keep all the development of each plugin in a separate topic branch, so you can track a specific development easily.

If you want to pitch in, I recommend that you get an account at github and fork the project. Github has some nice workflow tools that make it very easy to track each other work. If you need a github invite, ping me via XMPP.

Posted by Pedro Melo at 12:41 PM | Permalink | Comments (view)

So there are all this ingredients floating around:

• a nice Fortuna GPSmart BT: it has a mini usb that can be used for power, but I still prefer to fetch the NMEA feed over Bluetooth;
• a Macbook with BT;
• a 3G phone with good connectivity in Portugal, also with BT;
• a ejabberd server with PEP support, so I can publish my User Location.

The glue of course will be Perl, and the first step is already working. This program will print my location once per second:

use strict;
use warnings;
use lib 'lib';
use GPS::NMEA;

my $gps = GPS::NMEA->new(
  Port => '/dev/cu.GPSmart-SPPslave-1',
  Baud => 9600,
);

while(1) {
    my($ns,$lat,$ew,$lon) = $gps->get_position;
    print "($ns,$lat,$ew,$lon)\n";
}

This is a pure copy-and-paste version of the script in the GPS::NMEA manual page, it just works (after you set the correct Port), and it was my first version.

I did get some warnings, so be sure to download and install the latest development version of perl-GPS (You must specifically tell cpan to do it).

Next steps: hack some bot with Net::XMPP2 to publish this information.

Posted by Pedro Melo at 09:18 AM | Permalink | Comments (view)

So my kids are 1.3 and 3 years old now, and the oldest one loves a computer game that the place we have breakfast at on weekends has, based on a touch screen interface.

It has a set of games, like painting black and white layouts, filling the spaces with colors, basic memory games (6 cards down, with pictures, pick two alike to make them stick), and differences between pictures games.

I just got a quote for 15" TFT touch screens. Including a fan-less Via-based computer, with 500MB ram, and 30Gb disk, and a license of Windows XP, the whole deal comes about 1300€...

Hmms... I wonder if I can make the same games using HTML and the canvas stuff? I would build a full-screen web-kit, they would have a home-page for games, photos, what not...

Posted by Pedro Melo at 11:52 AM | Permalink | Comments (view)

I work at 8 different projects now (5 consulting, 3 personal business), and between them, I use 4 different version control systems: cvs, svn, git and darcs.

To keep myself sane, I developed a set of front-end scripts to deal with the most common operations: diff, commit, add files, and update.

Starting today and during the next week I'll post all of those scripts online, and eventually I'll also repackage them in a repository that people can pull from if interested. All the scripts are licensed in the Artistic License, and the usual disclaimer applies: they work for me, if they burn down your computer or any surrounding environment (like your house, for example), you get to deal with the fire department yourself.

The base of all the other scripts is the x-vcs-discover that basically will tell you which of those systems the current directory is in.

The script takes no parameters, and will output to the stdout a single line with the VCS that was detected.

#!/bin/sh
#
# Discover the VCS that is being used in the current dir if any
#
# (c) Pedro Melo, 2007 - Artistic License, same as Perl
#

vcs=
found=

while [ -z "$found" -a -z "$vcs" ] ; do 
  cwd=`pwd`
  if [ "$cwd" == "/" -o "$cwd" == "$HOME" ] ; then
    found='give up'
  elif [ -d CVS ] ; then
    vcs=cvs
  elif [ -d .svn ] ; then
    vcs=svn
  elif [ -d .git ] ; then
    vcs=git
  elif [ -d "_darcs" ] ; then
    vcs=darcs
  else
    cd ..
  fi
done

if [ -n "$vcs" ] ; then
  echo $vcs
fi

Posted by Pedro Melo at 11:53 AM | Permalink | Comments (view)

At the last geek lunch, Nuno I talked about some script to index mail using KinoSearch.

Sometime this week, Nuno asked me for a reference to that tool, and I just could not find it. I'm now pretty much sure that I was dreaming about it.

But the idea nagged me for a couple of days, and today I broke and took Mail-tools and KinoSearch for a ride.

The result is maildir_indexer.

Right now it works. It indexes about 100 messages per second on my laptop. Searching is instantaneous. I'll index about 10Gb of mail tomorrow to have better numbers. The index size is about 1/3 of the data.

This was a quick hack and it shows. Feel free to send patches, but look at the future/todo list first.

Posted by Pedro Melo at 12:51 AM | Permalink | Comments (view)