Notes

A beginning is the time for taking the most delicate care that the balances are correct.

from Manual of Muad'Dib by the Princess Irulan

extracted from Dune, by Frank Herbert

In the next weeks I'll be publishing a series of articles about a new project that I'm starting. Actually, I started this project once already, but I didn't like the path it was taking, so I'm restarting it.

The new system will replace the current EVOLUI.COM code base. Why rewrite a system that is in production for 8/9 years and that works reasonably well? There are several reasons actually.

The first reason is code rot: the current system is spread across 3 code bases (called internally E1 through E3). E1 was the first stab (who would guess, right?), and grew organically inside the company, without much planning, and coded by several programmers. E2 added a better management interface, and E3 added a new LMS and forum system.

A lot has changed in the last 8 years. And my inexperience in certain topics shows by the lack of proper tests and documentation. Also, it has grown to 850 Perl packages. Yes, eight hundred and fifty packages. I expect that more than half of those are no longer in active use.

The second, and more important reason, is that the new business requirements require a deep restructuring of the code, but given that we don't have a decent test suite, it would be madness to change stuff in one of those 850 packages.

The way I see it, I can write the new system properly, using a test-driven methodology, making sure that all the business logic is properly tested.

I'm not starting from scratch, I plan to reuse certain small but critical parts of the old system, but going over them to add tests and documentation.

The failed attempt was called E4, so this new version will be called E5.

I plan to cover the following topics (not sure of the order yet):

• source control, and ticketing systems;
• project directory layout;
• documentation techniques;
• business layer techniques;
• databases used and ORM layers;
• web frameworks;
• template systems;
• logging;
• smoke testing;
• deployment.

Hope you enjoy the ride.

Posted by Pedro Melo at 05:04 PM | Permalink | Comments (view)

Some weeks ago, Johannes Schindelin talked about the UGFWIINI (Use Git For What It Is Not Indended) contest. You can read the initial candidates in the git mailing list thread.

A couple of months ago, I had this idea of writting a script that would constantly watch over a directory (including subdirectories) and commit the changes every time anything would change.

At the time, the tought of battling the FSEvents interface was not appealing at all, and it was just a joke anyway. But in the since then several things happened:

• Andy Grundman released Mac::FSEvents to CPAN: a working interface to the FSEvents system, with a extremely nice API;
• João Pedro, a friend at SAPO, IM'ed me asking for such a script: I pointed him to the new Perl module;
• A script, Flashbake, by Thomas Gideon made it to the contest.

So I had no chance but to write such a script.

My version is a quick hack, but already works perfectly. It watches a specific directory, and commits all the changes to a standalone Git repository.

To use just do:

cacm --repo dir_of_git_repo --watch dir_to_watch

and it will start recording your changes with git commits.

Right now, it commits at most once per 5 seconds, but you can tweak that using the command line option --latency latency_in_seconds.

I've uploaded version 0.01 to CPAN. To install just do cpan App::CMAM (it might take 24 hours to spread to your CPAN mirror).

For version 0.02 I'll clean some small details:

• ignore .git and other common directories;
• cleanup git invocation: all the output should be hidden from you.

Fun!

Posted by Pedro Melo at 05:09 PM | Permalink | Comments (view)

Uploaded to PAUSE AnyEvent::Mojo 0.6, should hit the CPAN mirrors throughout the day.

Update: uploaded 0.6001, with an attempted fix for some 100% CPU usage cases.

The major change is the renaming of AnyEvent::Mojo to AnyEvent::Mojo::Server.

There are two reasons for this change. The first is to prepare the release of AnyEvent::Mojo::Client in the next release, wrapping the Mojo::Client API.

The second is to free the AnyEvent::Mojo to be used with the new functional API.

Starting a Mojo server inside your AnyEvent apps is now very simple:

use AnyEvent;
use AnyEvent::Mojo;

my $server; $server = mojo_server undef, 4323, sub {
  my ($self, $tx) = @_;

  $tx->res->body('Cool!');  
};

$server->run;

The interface is similar to AnyEvent::Socket.

This is the first beta release. I don't expect to change the API in a backwards incompatible way until 1.0.

I do plan the following for the next releases:

• implement AnyEvent::Mojo::Client and the correspondent mojo_client function;
• implement support for 100 Continue.

Mojo is fun!

Oh, and please vote for the Perl Foundation grant to improve Mojo.

Posted by Pedro Melo at 03:52 PM | Permalink | Comments (view)

I've uploaded to PAUSE (give it a couple of hours to appear on your local CPAN mirrors) the latest release (0.5) of AnyEvent::Mojo.

The biggest change is the new resumable requests. After your handler is called, you can pause the connection, do other stuff using the AnyEvent asynchrounous nature, and when you are ready to send back the response, you can resume the request.

In pratical terms, this is now possible:

sub my_handler_cb {
    my ($self, $tx) = @_;

  # Pause the connection while we HTTP GET our information
  $tx->connection->pause;

  # This HTTP get is done asynchronously
  http_get $status_url, sub {
      my ($data) = @_;
    my $res = $tx->res;


      if (!defined $data) {
        $res->code(503);
      }
      else {
        $res->body($data);
      }

      # Ok, the response will be sent now
      $tx->connection->resume;
  }

  # We paused the connection so we can return
  return;
}

While the inner http_get is going on, the AnyEvent::Mojo server can handle other requests.

This allows you to keep several thousands of requests going on without any effort at all.

I'm still labeling AnyEvent::Mojo as alpha. The interface will change one last time for 0.6. After that it should be beta time until 1.0.

I need to find a decent WebSockets client implementation (or some sort of long-pooling client) to make a decent application with this. Any recomendations?

Posted by Pedro Melo at 11:48 AM | Permalink | Comments (view)

Apparently some things never die. In January 1994, a friend of mine, Paulo Ferreira, created an online community around a Lambda Moo server. Its called Moosaico and it still kicking.

I spent an enormous amount of time there and I still have fond memories of that time, of the people, the stories, everything.

But access to Moosaico (and other MUD's or Talkers) is usually done with Telnet or using a client like TinyFugue or Atlantis. Keeping one of those open to be connected somehow never entered my normal workflow, so I drifted away.

Its amazing that something that I was hooked on 14 years ago is still alive in me.

The friday something happened that will pull me back in.

Another friend, Marcos Marado, commented that he was looking for a XMPP interface to his talker, Selva. I had an headache and didn't feel like working on what I was supposed to, so I took the challenge.

Later that afternoon, Rasputine was born.

Rasputine (or Ras, as Corto Maltese called him) is a generic Moo/MUD/Talker-to-XMPP gateway. You add a buddy to your roster and then you can use it to connect to that world.

For example, to log in to Moosaico, add moosaico@rasputine.simplicidade.org and then type //connect. Presto, the login window of Moosaico should be readily available. The Selva talker is available at selva@rasputine.simplicidade.org.

The code is still very young and feature-less. The copy that I run on my staging server already has per-user status messages and avatars, and next I plan to implement auto-connect (probably even XEP-0100 support).

For now, I'm just enjoying getting back to Moosaico.

Update: by the way, if you want to connect your own talker/MUD/Moo, ping me via XMPP or email and I can set you up. At least to try it out.

Posted by Pedro Melo at 06:13 PM | Permalink | Comments (view)

My favorite BitTorrent client is Bitflu.

It is a head-less client that runs on a server somewhere, and you can manage it using a HTTP or telnet interface. It also supports a auto-start directory where all the torrent files you drop in it will be picked up automatically.

What it doesn't have is a way to upload a torrent file programatically.

At least, not until today. Check out my torrent-upload branch of Bitflu. It provides a new HTTP-based API to upload a single torrent file.

To use it, do this:

lwp-request -c 'application/x-bittorrent' -m POST \
  http://your.bitflu.site:4081/new_torrent  < my.torrent

The lwp-request command does not support authentication. I need to hack a Net::Bitflu module to make this easier.

The patch was sent to the author, but I don't know if it is something that he is interested on or not.

Posted by Pedro Melo at 01:33 PM | Permalink | Comments (view)

Update: uploaded version 0.04, fixes two small things: I was using pre-release 0.8 of Mojo, and apparently I don't know how to use requires in Makefile.PL, causing a stream of CPAN tester failures.

Sebastian Riedel released version 0.7 of his new HTTP stack Mojo (also includes a tiny web-framework called Mojolicious).

The code is very very nice and clean, and in the future you might be able to run Catalyst apps on top of it.

Given that I was looking for a small web framework to run inside my long-lived AnyEvent daemons, I wrote a small glue module to run Mojo-apps using the AnyEvent::Socket/AnyEvent::Handle goodness.

The module is AnyEvent::Mojo, and I've just uploaded release 0.2, that includes keep-alive support.

There are three features that I want to do before a 1.0 release:

• Support for 100 Continue handling: I don't use it but the other Mojo::Server classes have it;
• Asynchronous processing of requests: you start processing a request, but decide that you need something from a network service somewhere. You can start that request, pause the Mojo transaction, and when you receive the response from the network service, you complete the Mojo transaction;
• build a AnyEvent HTTP client using Mojo::Client.

Fun stuff, specially the second one.

Performance-wyse, I'm happy with the result so far. The most simple Mojo::HelloWorld application is capable of this numbers:

# Start a server
perl -MEV -MAnyEvent::Mojo -e 'AnyEvent::Mojo->new->run' &

# Benchmark
ab -k -c 10 -n 5000 http://127.0.0.1:3000/

# Executive summary
Concurrency Level:      10
Time taken for tests:   6.935770 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      795000 bytes
HTML transferred:       190000 bytes
Requests per second:    720.90 [#/sec] (mean)
Time per request:       13.872 [ms] (mean)
Time per request:       1.387 [ms] (mean, across all concurrent requests)
Transfer rate:          111.88 [Kbytes/sec] received

# (Quad-core 2.6Ghz, Mac OS X 10.5.4, stock perl 5.8.8)

At 700 requests per second with a 1.3 ms delay, the overhead seems to be pretty low, for a pure Perl based HTTP 1.1 compliant server.

Posted by Pedro Melo at 02:13 PM | Permalink | Comments (view)

I'm thinking on ordering one of these EFI-x dongles, to play around. I wonder how well it works.

My motherboard is supported... hmms...

Posted by Pedro Melo at 11:07 AM | Permalink | Comments (view)

For quite some years now, I have been using some compile.sh scripts to setup my baseline system for each project. If happen to work in Portugal, at one of my previous employers, you might find them somewhere in /servers.

(Historical note: the /servers nomenclature was devised between '95 and '97, either at Telenet or IP Global. It later was refined at other companies to include /servers/etc, /servers/logs, /servers/data, and /servers/workspace for specific purposes. You can still find it today at several places in Portugal, given that the *NIX/ISP environment community was pretty small at that time.)

Each compile script (I lost count of the ones I have, from Perl, Apache, mod_perl, libraries, djb-tools) downloads the version of the package that I want to use, configures, builds and installs it on a local directory. Each project has his own version of the baseline.

In the past I used to create a two user accounts for each project, named P and Pbase. The first is used to keep and run all the code for said project, and the second is used to install all the dependencies.

The system has worked pretty well but I wanted to clean it up, so I coded a basic system to replace my compile scripts.

Its pretty raw, but I plan to move all my compile.sh script to it.

I'm sure that this was already done. It has a lot of similarities with RPM or .deb packages, but this one is my own, and I can take it to whatever level I want.

If you are curious about it, you can find the code and rule sets (minimal for now) in the official repository.

Posted by Pedro Melo at 03:28 PM | Permalink | Comments (view)

Mozilla Labs Ubiquity looks very nice indeed.

I wonder how hard it would be to port it to a Fluid plugin?

(via Ajaxian)

Posted by Pedro Melo at 12:10 PM | Permalink | Comments (view)

I've updated my own Search In Project with ack TextMate command.

Changes:

• search in selected files is optional now: start your query with a : to search only in the selected files;
• fixed search for words with a initial dash in them.

You can find the current version at GitHub or download the tmCommand file directly.

This is a very simple ack interface. For a more complex and feature-full, see the ack TextMate bundle by Trevor Squires.

Posted by Pedro Melo at 01:46 PM | Permalink | Comments (view)

Tarpipe is a wonderful service to execute a workflow of web services. You create your workflow visually, and then you send an email or POST to a API end-point to run it.

For example, this is the workflow I use to post to Twitter, Jaiku, and Pownce:

Usually I would send an email and my subject would be posted, but I also have an IM client always available.

So I wrote this XMPP-to-Tarpipe gateway. Its simple to use:

• use Tarpipe to create an workflow: you must use the REST Decoder;
• When you save the workflow, you'll be given a API token, a 32 char hex-string;
• subscribe <the API token>@tarpipe.simplicidade.org. You can subscribe from multiple JIDs.

That's it: you now have a new buddy on your roster. Any message sent to him (apart from help) will trigger the Tarpipe workflow.

Have fun.

Posted by Pedro Melo at 12:36 PM | Permalink | Comments (view)

New location: the How-to is now live at http://www.simplicidade.org/how-to/mysql-master-master/.

In the past, I needed to use a MySQL master/master setup for a client. At the time, I got it to work, but forgot to take notes about the process.

In the last days, a friend asked my for help to do a setup like that again, to use with a Tigase XMPP server.

So I wrote this how-to, complete with configuration files, sample data and schemas, that walks you step-by-step through the process of setting up a pair MySQL servers in a master/master configuration.

You can fetch the entire how-to with git:

git clone git://github.com/melo/mysql-master-master-how-to.git

The how-to is written in MultiMarkDown, but a simple Markdown should also format it correctly.

A tarball with everything is also available.

Please let me know if you find any mistakes. Pull requests are the preferred form of collaboration :).

Update: included some tweaks from dbr, and added a HTML version with proper CSS. Download again, and look at how-to.html.

Posted by Pedro Melo at 03:11 AM | Permalink | Comments (view)

cheat is a very nice command line cheat sheet system.

chit is a git-powered version, that even supports private cheats or project level cheats distributed using the git system.

I've added a bash completion script to chit. It's very simple but it works for me. I've also sent a pull request to robin, so you might see it in the main chit repo in the future.

Update: you can find the bash completion script in the master chit repo now.

Posted by Pedro Melo at 12:54 PM | Permalink | Comments (view)

I've moved the Search in Project With ack TextMate command to GitHub.

In the process I've included a change by Corey Jewett to respect the selected files in the project drawer, and I've also added support for the TM_ACK_COMMAND_PATH variable, in case your ack copy is somewhere strange.

There is an open TODO (catch popen errors if the ack command is not found), but I need to freshen up my ruby skill. It should be trivial, push requests welcome.

This is a very simple command. If you are looking for something more complete check out the ack TextMate bundle by Trevor Squires.

Posted by Pedro Melo at 11:32 AM | Permalink | Comments (view)

I've started to publish my ~/bin script stash at GitHub.

Not all of them are written by me. For example, I've included Gruber's TitleCase.pl script under a new name.

Before you ask about the strange long names, my memory sucks, so I use tab completion a lot. I name all my script with a x- prefix so that I can just type x-<TAB> and see my entire script repository. I use long and (I hope) self describing names.

Common scripts that I use every day get single line shortcuts using shell aliases.

To use, just clone the repository, and add the bin/ to your PATH.

Patches welcome. Recommended workflow:

1. sign up at GitHub;
2. fork my project (small fork icon);
3. clone your fork to your local workstation;
4. hack, hack, hack;
5. git-push
6. hop onto GitHub and send me a pull§ request.

Posted by Pedro Melo at 07:52 AM | Permalink | Comments (view)

A big part of my work is doing non-blocking servers using the AnyEvent Perl module (I used to prefer POE, but AnyEvent beats POE both in terms of performance and simplicity).

One recurring problem is using DBI inside a non-blocking server. The modules that you can find on CPAN that deal with this problem use slave processes to run the DBI code, and then a lightweight protocol to send the requests back to the master process. This way the DBI can block as much as it wants without causing problems to the non-blocking master process.

This works of course, but I would be lighter to skip all this process mumbo-jumbo if we could have a non-blocking DBI.

I browsed the DBI code and sketched a small API to allow you to execute_nb a statement and receive the results via a callback, but I slammed into problems with the DBD's.

For example, DBD::mysql uses the C-based mysqlclient. But this library is pretty much useless in terms of building a non-blocking client. It just doesn't have the required functions to allow that kind of usage.

One workaround, at least for DBD::mysql, would be to code a small threaded engine. We would use C-level threads, one for each connection, and the master Perl/XS module would communicate with them to simulate non-blocking behavior. This would not be something new, IO::AIO uses this trick with much success (see the impressive performance that you can squeeze out of projects like qpsmtpd, Perlbal, or DJabberd, all built on top of IO::AIO via Danga::Socket).

But we are back to the master-slaves approach of the current CPAN solutions, only now in-process so the communication between threads should be pretty fast and simple. And its specific to DBD::mysql. Its better but still not good enough.

I'll keep poking at this from time to time. I would like to have a solution that would only require DBI for this. My next item on the "things to try"-list is reading the code for DBD::Gofer. In itself, I don't see it as a solution, but I'm hoping to learn something that triggers a possible approach for this.

So for now, I'll keep using the master-slaves approach. While not perfect, it's good enough for now.

Update: also to check, the callback mechanism of DBI, as explained by Tim Bunce.

Posted by Pedro Melo at 03:58 PM | Permalink | Comments (view)

Last February, Google's Android Team shocked the XMPP world by having the gall of saying that they where moving to a Binary encoding of their GTalk API in the M5 release.

Of course this sent shock waves through the XMPP community, not because they where totally wrong, but because they didn't have the vision to solve it.

We of course understand that they have their hands full (of vapor, some bad mouths would say...) getting the platform in shape for the late 2008 release, and the XMPP.. err, sorry, GTalk protocol is not a top priority.

Well, today Android users around the world can rejoice because once again, the XMPP Standards Foundation has risen to the challenge and unveils to the world the solution to the "verboseness" problem, by introducing the breakthrough Binary XMPP protocol.

This has been in development for quite some time, taking long hours of (sometimes) heated discussion. The goal is to have something that nobody could ever accuse of being verbose (the symbol count is extremely low, and the meaning is clear from the start), but at the same time remain compressible for those extreme cases where bandwidth is at a premium (we tested, and we have code to prove it: a Binary XMPP stream can be compressed to 2% of its original size). We know of clients that live in five-sides-shaped-buildings that, with their hard 9600 baud limits, will be the first to use this.

Its been a great ride, and it feels me with immense joy to be able to put my small signature as a co-author of this spec. Of course, my part is extremely small when you have the documentation genius of Peter Saint-Andre, and Fabio Forno strict guidelines and requirements, as co-authors. Its been a pleasure to work with both on this. And I should also thanks the important Kevin Smith contributions.

And now time to rest. SRV records are up at simplicidade.org domain, and Fabio's CM is running also.

PS: and yes, now you know why the prolonged silence around here.
PPS: for those who have moved on, the code is also available on GitHub.

Posted by Pedro Melo at 12:44 AM | Permalink | Comments (view)

If you want to try some agent-love, then let me introduce you to Alter Ego, your friendly pluggable agent.

Alter Ego is a XMPP agent, where everything is a plugin. You can clone the current version with git (recommended, easy to keep track of new features and fixes) or use the tarballs.

Right now, you can find it running under my own JID, so yes, I trust it enough to run it on my main account.

The only plugin I wrote is a User Location (XEP-0080) updater. With all the cool geo toys (like Fire Eagle) around, I wanted to play a bit with location-aware services. So the current UserLocation plugin does a very simple thing:

• discovers the mac address of your default gateway;
• searches the mac address in a local database;
• if found, he publish the information to your User Location PEP node.

So you do need a PEP-enabled server like ejabberd 2.0 or OpenFire.

Also, nothing prevents you from using other sources of location like a GPS. For example, I'll probably include something like this perl-GPS simple example in a future version of the plugin.

You can look at the roadmap to see what I'll be working on, or you can subscribe any of the zillions of feeds that github provides. I usually keep all the development of each plugin in a separate topic branch, so you can track a specific development easily.

If you want to pitch in, I recommend that you get an account at github and fork the project. Github has some nice workflow tools that make it very easy to track each other work. If you need a github invite, ping me via XMPP.

Posted by Pedro Melo at 12:41 PM | Permalink | Comments (view)

So there are all this ingredients floating around:

• a nice Fortuna GPSmart BT: it has a mini usb that can be used for power, but I still prefer to fetch the NMEA feed over Bluetooth;
• a Macbook with BT;
• a 3G phone with good connectivity in Portugal, also with BT;
• a ejabberd server with PEP support, so I can publish my User Location.

The glue of course will be Perl, and the first step is already working. This program will print my location once per second:

use strict;
use warnings;
use lib 'lib';
use GPS::NMEA;

my $gps = GPS::NMEA->new(
  Port => '/dev/cu.GPSmart-SPPslave-1',
  Baud => 9600,
);

while(1) {
    my($ns,$lat,$ew,$lon) = $gps->get_position;
    print "($ns,$lat,$ew,$lon)\n";
}

This is a pure copy-and-paste version of the script in the GPS::NMEA manual page, it just works (after you set the correct Port), and it was my first version.

I did get some warnings, so be sure to download and install the latest development version of perl-GPS (You must specifically tell cpan to do it).

Next steps: hack some bot with Net::XMPP2 to publish this information.

Posted by Pedro Melo at 09:18 AM | Permalink | Comments (view)

So my kids are 1.3 and 3 years old now, and the oldest one loves a computer game that the place we have breakfast at on weekends has, based on a touch screen interface.

It has a set of games, like painting black and white layouts, filling the spaces with colors, basic memory games (6 cards down, with pictures, pick two alike to make them stick), and differences between pictures games.

I just got a quote for 15" TFT touch screens. Including a fan-less Via-based computer, with 500MB ram, and 30Gb disk, and a license of Windows XP, the whole deal comes about 1300€...

Hmms... I wonder if I can make the same games using HTML and the canvas stuff? I would build a full-screen web-kit, they would have a home-page for games, photos, what not...

Posted by Pedro Melo at 11:52 AM | Permalink | Comments (view)

I work at 8 different projects now (5 consulting, 3 personal business), and between them, I use 4 different version control systems: cvs, svn, git and darcs.

To keep myself sane, I developed a set of front-end scripts to deal with the most common operations: diff, commit, add files, and update.

Starting today and during the next week I'll post all of those scripts online, and eventually I'll also repackage them in a repository that people can pull from if interested. All the scripts are licensed in the Artistic License, and the usual disclaimer applies: they work for me, if they burn down your computer or any surrounding environment (like your house, for example), you get to deal with the fire department yourself.

The base of all the other scripts is the x-vcs-discover that basically will tell you which of those systems the current directory is in.

The script takes no parameters, and will output to the stdout a single line with the VCS that was detected.

#!/bin/sh
#
# Discover the VCS that is being used in the current dir if any
#
# (c) Pedro Melo, 2007 - Artistic License, same as Perl
#

vcs=
found=

while [ -z "$found" -a -z "$vcs" ] ; do 
  cwd=`pwd`
  if [ "$cwd" == "/" -o "$cwd" == "$HOME" ] ; then
    found='give up'
  elif [ -d CVS ] ; then
    vcs=cvs
  elif [ -d .svn ] ; then
    vcs=svn
  elif [ -d .git ] ; then
    vcs=git
  elif [ -d "_darcs" ] ; then
    vcs=darcs
  else
    cd ..
  fi
done

if [ -n "$vcs" ] ; then
  echo $vcs
fi

Posted by Pedro Melo at 11:53 AM | Permalink | Comments (view)

At the last geek lunch, Nuno I talked about some script to index mail using KinoSearch.

Sometime this week, Nuno asked me for a reference to that tool, and I just could not find it. I'm now pretty much sure that I was dreaming about it.

But the idea nagged me for a couple of days, and today I broke and took Mail-tools and KinoSearch for a ride.

The result is maildir_indexer.

Right now it works. It indexes about 100 messages per second on my laptop. Searching is instantaneous. I'll index about 10Gb of mail tomorrow to have better numbers. The index size is about 1/3 of the data.

This was a quick hack and it shows. Feel free to send patches, but look at the future/todo list first.

Posted by Pedro Melo at 12:51 AM | Permalink | Comments (view)