Main

September 18, 2012

*Flow setups for git

Pedro Figueiredo pointed me to HubFlow, a twist on the original GitFlow setup to work more closely with Github.

I've used GitFlow for a couple of weeks when I found it for the first time, but dropped it eventually because it didn't fit my needs. I'm not dissing it, it is very well though out and it provides a nice set of tools to make it all work, but it also adds a complexity layer that may not match your environment.

There are two situations in particular where I would recommend not using these *Flow setups: single developer, and if you use a faster release train.

The first should be obvious: all those shenanigans moving code from topic branches to develop to release branches make sense if you have a team of people with defined roles, like module owners who must approve changes to their own turf, or release managers who take a set of features accepted into the develop branch and make a release out of it.

The second is more of a personal preference. I hate the release manager notion. I'm sure that it has value in a lot of projects, but I can say that I'm fortunate enough not to have worked on one of those. My preference came from a previous job were I was the de facto release manager, not because the position was official, but because I was more adept (or masochist, depends on the point of view) to merging CVS branches... Oh yes... CVS... Branches... Hell.

So I moved to keeping the trunk/master branch of my projects always deploy-able, to automate the release process, and to make sure releasing is a one command away, available to anyone on the team.

*Flow setups get in the way of those setups a bit. The develop branch sits between topic/feature branches and master, and delays the move from feature to deployment status of the code. I don't like that.

Having said that, if you have a team of developers, if you use git, and your rough setup is getting in the way, I would suggest you to try one of these *Flow setups. At least read them thoroughly and take the ideas that fit your local conditions.


For future reference, this is the setup I use for single-developer situations. It is based on git, but most of this could be applied to any SCM that has decent branch/merging semantics.

The master branch is the production version: it's always deploy-able, and it's also regularly smoke tested with two configurations:

  • one that mimics production environment as much as possible. If your DB is small enough to have a copy available for smoking, do so. Also use the same versions of the dependencies you have in production. If you are a Perl user, cartoon is very helpful here.
  • the second with the latest version of your language, plus the latest version of your dependencies: I'm using perl 5.14.2 but I smoke it with 5.16.1 and 5.17.latest, with the latest versions of my dependencies. This will catch future problems early.

All development is done on feature branches, one branch per feature. Feature branches can/should be pushed to the central repository to be picked up by the CI to be tested with a more production-like setup. When fully ready, or ready enough to be hidden from end-users behind a feature flag, merge into master and release it. I always use --no-ff when I merge, I like to know certain commits were made outside master.

If a feature is being developed for a long time (more than a month), then either rebase it regularly (my preference) or at least create a integration branch from master and merge the feature branch into it to test. It is essential to test long term branches with the latest production code.

This works for me quite well. Simple, low complexity, I can explain it in less than 3 minutes.

June 17, 2011

GitHub, CA errors and old curl's

A couple weeks back I noticed someone on Twitter having problems cloning git repos from GitHub using HTTPS. I didn't pay attention to it because I usually use git: protocol - nothing against HTTP, just habit.

But today, on a Mac OS X 10.5.8 system, I noticed something similar:

$ curl -LO http://xrl.us/cpanm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   185  100   185    0     0    301      0 --:--:-- --:--:-- --:--:--   301
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Now, you can work around it quickly if you add --insecure to that command line, but that feels dirty.

I checked on my other Mac, running 10.6.6, and I had no problems. The curl version in Leopard is just too old, and lacks some of the new certification authorities:

### 10.5.8
$ curl --version 
curl 7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3

### 10.6.6
$ curl --version 
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3

If you check curl SSL certs documentation you'll see that, yes 7.16 is very old and until 7.18.0, the bundled CA file is "severely outdated".

The solution is to update the bundled CA file. First we need to find it and curl-config --ca is your friend:

$ curl-config --ca
/usr/share/curl/curl-ca-bundle.crt

I though "I'll just copy the file from 10.6.6..." and be done with it, but no such file is present on my Snow Leopard. I assume that curl uses the system keychain in 10.6, but I don't know for sure.

So we do it the hard way. I'm just interested on accessing GitHub without problems so I checked the CA GitHub uses and downloaded the CA chain from them: you'll need both the "DigiCert High Assurance EV Root CA" and the "DigiCert High Assurance EV CA-1".

Put those file in a directory, open a terminal to it and type:

cat /usr/share/curl/curl-ca-bundle.crt \
    DigiCertHighAssuranceEVRootCA.crt \
    DigiCertHighAssunceEVCA-1.crt \
    >> curl-ca-bundle-new.crt

To test this new CA bundle you can use:

curl --cacert curl-ca-bundle-new.crt -LO http://xrl.us/cpanm

and the download should work perfectly.

To make this change more permanent you can replace the original curl-ca-bundle-new.crt with this commands:

sudo cp /usr/share/curl/curl-ca-bundle.crt /usr/share/curl/curl-ca-bundle.crt-backup
sudo cp curl-ca-bundle-new.crt /usr/share/curl/curl-ca-bundle.crt
sudo chmod 644 /usr/share/curl/curl-ca-bundle.crt
sudo chown root:wheel /usr/share/curl/curl-ca-bundle.crt

And that's it! All your HTTPS downloads from GitHub should now be CA errors free, including clones using https:// URLs.

Although I had this problem on a Mac, the solution should work as well with other operating systems, as soon as you find the location of the curl-ca-bundle.crt file.

February 15, 2011

Tip: keep the Host header via nginx proxy_pass

For the past 2 or 3 years, my favorite web server is nginx. It replaced lighttpd on most of my production sites, and soon it will take over them all.

One of the roles nginx is performing is to act as a frontend server for Perl-based Web apps. This apps run under Starman or one of the other PSGI-compatible servers.

The simplest configuration block you need to use is this:

server {
    server_name my.site.com;

    location / {
        proxy_pass http://127.0.0.1:9999/;
    }
}

This will proxy the request to your Web app running at 127.0.0.1, port 9999.

The problem is that the Host HTTP header is rewritten to 127.0.0.1. In fact, the header is rewritten to whatever value you use as the host in the URL given to the proxy_pass directive.

With a bit of /etc/hosts manipulation you can use stuff like:

proxy_pass http://my_web_app.production:9999/;

But this makes the deployment process more complex because you need to keep /etc/hosts in sync with all your apps.

A cleaner alternative is to force the original Host header to be sent to the backend server, like this:

server {
    server_name my.site.com;

    location / {
        proxy_pass http://127.0.0.1:9999/;
        proxy_set_header Host $http_host;
    }
}

With this configuration, the backend server will receive whatever Host HTTP header your browser sent that matched this server block, which is particularly useful if you use a regular expression to match host names.

Or if you have multiple server_name's but want your backend to receive a fixed host name, use:

proxy_set_header Host "my.forced.hostname";

Either way, its much simpler to keep frontend and backend servers in sync, with regard to virtual host names.

October 12, 2009

Java 1.6 on Leopard

I did the research on this a month ago and I forgot to write it down, so I just spent another hour doing it again. I should know better by now.

Anyway, you can download the Java 1.6 update for Mac OS X Leopard from the Apple Software site, but its only 64-bit.

I do have a desktop that is 64-biT, unfortunately my Macbook Pro laptop is only a 32-bit Core Duo. Hence, I cannot run the new version of Java.

If you do have a 64-bit CPU, and after you install the update, you might want to switch the default version of Java to 1.6. To do that, the proper way to do that is to run the /Applications/Utilities/Java\ Preferences.app and drag the 1.6 version to the top of the list.

I just wanted to try out Jake... Guess I wont be able to, at least not until I can get my hands on the wife's laptop... hmms... she's sleeping already...

October 08, 2009

Attaching jobs in Gearman

I've used Gearman on and off in the past but for a new project, I've decided to explore some features I rarely made use of previously. Most notably, the unique ID that clients can submit with each job.

Let me just clarify some behaviors for non-background jobs:

  • the worker will execute the job until it finishes even if the client that submitted it dies;
  • if the client dies before the job is passed on to a worker, it will be removed and will never execute.

For background jobs, the behavior is different: a client submits a background job, the gearmand daemon queues this (on stable storage if you use the persistent queues of the latest versions of the C implementation), and sends it to a worker as soon as one is available.

This is basic stuff, I'm just making sure we are all on the same page regarding Gearman behavior.

Back to the the unique parameter. You can define a unique key for your jobs. Those keys should be unique per function, so you can have the same key on different functions and they will be two different jobs.

The fun part with unique is that you can have multiple clients listening to status and completion events of the same job if you share the key between them. Say that you start a job with a key alpha. If other clients submit a job with the same key, and (and this is an important) the job is still running, this second client will attach itself to the same job.

To test this I wrote a small shell script worker (save it as slow.sh and make it executable with chmod 755 slow.sh):

#!/bin/sh

echo "Starting a slow worker..." >>/dev/fd/2
for i in 5 4 3 2 1 ; do
  echo $i >>/dev/fd/2
  sleep 1
done
echo done $PID

This worker does nothing expect print to STDERR a count, and then sends to the client a done string with the worker process ID. But it takes 5 seconds to run so it allows us some time to switch between windows.

After you start your gearmand server (I used gearmand -vv just to have some debug) you can start a worker process like this:

gearman -w -f slow ./slow.sh

This worker registers the slow function and will execute the slow.sh per job.

Now open two new terminal windows and type on each one:

gearman -f slow -u alpha -s

This will submit a non-background job for the slow function, with the unique key alpha. The -s means that we won't be sending any data.

You'll see one execution of the worker, and then both clients will output something like this:

done 57998

You can experiment, and attach the second client only half-way through the worker run. Or you can stop the worker, start both clients, and then start the worker. The result will be the same: both clients receive the output for the same job.

This is a very very cool feature, and can be used easily for slow processing inside a web request.

First, if a web request requires a slow processing phase, we store all the relevant data and submit a background job to do the processing with some random key. The user receives a "Processing page" on his browser, that includes this key.

A small javascript program using AJAX, connects to our long-pooling server and submits a non-background job to the same function and using the key. This second client request will now wait until the processing is done, and can even receive status updates from the worker and send them back to the browser.

This attach-to-job-using-key is very reliable. I've tested several combinations with and without workers running, strange order of job submission, adding multiple clients to the same job, and all of them work as expected.

The only real problem with this is that the clients need to use the same API that is used to submit a job. Instead of a new API, like ATTACH_JOB for example, you use the same SUBMIT_JOB API that you use for new jobs. This works fine, until your second client attaches with a key for a job that has already ended. The gearmand server will fail to find it, and will dutifully create a new job.

My current workaround for this is to send a specific payload on the attach requests, to signal the worker that this is a attach request and not a new job. The worker, if he detects this signal, just ends the processing. For example, if your jobs require payload data, you can use an empty data field as the flag.

This is not optimal of course, you would be waking up workers just to return immediately, but it would work.

If you have multiple gearmand servers, you need to make sure that the clients that will be attaching to a job use the same server. A solution would be to pass the server identification (IP or even better hash-of-IP) along with the key.

The traditional way of doing this is have the workers store the completed result in a database somewhere. The solution presented here does not invalidate that, it only provides a very light notification of completion to background jobs. It beats pooling the database to see if the background job is completed.

October 01, 2009

bash completion for Github gem

For some time now, I had the Github gem installed (if you want to know more, I suggest a old blog post about the Github gem). This gives you a small gh script that interface to Github APIs and make common operations like creating repositories, cloning, and fetching other repositories in the project network easy and fast.

But I'm a lazy bastard, and the lack of a bash completion script was getting on my nerves.

So here: you can get the new gh-completion.bash script from my repository. I've sent a pull request to Chris Wanstrath (defunkt)

Please note that this is still work-in-progress.

Right now, most of the command and options have completion, but I'm still trying to understand certain operations (like pull-request and track) that I never use. I don't really understand what they do, so I have no way to create a decent completion rule for them.

But using this bash completion is totally safe... Until you hit the ENTER key, that is.

September 24, 2009

Tip: use pv to monitor mysql loads

If you are fortunate enough to be able to reload your development databases from time to time with production data, this might help.

The usual command you would use is something like this:

gunzip -c db-backup.sql.gz | mysql -udevuser -ppass db_dev

If your dump is big this can take a while and you wont have a clue about what is happening.

Instead, install pipe viewer and do:

gunzip -c db-backup.sql.gz | pv | mysql -udevuser -ppass db_dev

and you get a nice speed meter. For even better results:

dump=db-backup.sql.gz
size=`gunzip --list $dump | perl -ne 'print "$1\n" if /\d+\s+(\d+)\s+\d/'`
gunzip -c $dump | pv -s $size | mysql -udevuser -ppass db_env

and you'll get a speed meter and an ETA. The second command will get the uncompressed size of the dump and use that to teach pv how much data to expect.

Pipe viewer rockz.

September 10, 2009

Log::Log4perl tip

I use Log::Log4perl for all my logging needs. Ok, I lie. I use a wrapper that deals with some stuff that I just don't like with Log::Log4perl, but that is a story for another day.

One thing that we inherited from log4j was the notion that a message can match multiple loggers in your logging hierarchy.

The logic is simple and explained in detail on a Log::Log4perl FAQ entry. If you write something like this in your logger configuration file:

log4perl.logger.Cat        = ERROR, Screen
log4perl.logger.Cat.Subcat = WARN, Screen

which define two loggers. Cat and Cat.Subcat, the second a subcategory of the first, and then use:

my $logger = get_logger("Cat.Subcat");
$logger->warn("Warning!");

you'll get a duplicate message in your log file because it matches both loggers.

I knew that and I always added a line saying:

log4perl.additivity.Cat.Subcat = 0

that prevented this behavior, but this required a line like that per logger. Pain. Not lazy, at all.

But for some reason (stupidity comes to mind) I didn't read the FAQ completely, because at the end, there is a solution. Just put this in your logger configuration file:

log4perl.oneMessagePerAppender = 1

Bliss, pure bliss.

Mind you that oneMessagePerAppender is not compatible with log4j, something that Log::Log4perl tries very hard to be, and therefore this feature is not documented at all except on this FAQ entry.

Problem solved

I keep a pad of paper between me and my keyboard at all times. I used to take down notes, keep track of what I need to do today, small brain dumps, and random scribbles.

But with my hands going about their bussiness, the corner of the paper starts to bend upwards.

Problem

The solution is not rocket science. A simple paper clip.

Solved

Problem solved.

August 30, 2009

When to upgrade to Snow Leopard

This are the rules I use to do the upgrades between major versions of Mac OS X.

You upgrade when:

  • you have half-day of free time to do a clean install;
  • all your must-have apps support the new version;
  • any of your must-have apps requires the new version.

I usually wait until the last item is true before doing the upgrade.

May 28, 2009

PGP WDE update

I mentioned last week that I had started using PGP Whole Disk Encryption on my laptop (a first generation MacBook Pro 17" btw, it has a 32bit 2.16Ghz Core Duo with 2Gb RAM).

I encrypted the external FW800 500Gb hard disk that I use exclusively for Time Machine last night. It took about 8 hours. So far so good.

When I connect it to my mac, a PGP WDE dialog pop-up shows up, asking me for the correct pass-phrase. After I enter it, it shows up as a Time Machine external disk, and the backup starts.

Be aware that, in case of hard disk disasters, encrypting your TM disk can be more awkward than encrypting your internal hard drive.

With a normal non-encripted TM drive, you can boot your Mac from a Leopard DVD, and ask the Installer to restore a TM backup directly.

If you encrypt you TM disk, thats no longer an option. You have to install a bare bones system, install PGP WDE, and then restore the TM backup.

A possible solution (untested for now) is to have a clone of your internal hard drive (even an encrypted clone should work) that you update from time to time. Then you should be able to boot from the clone and restore a TM backup to the internal disk.

As I said, I haven't tried this yet. It should work, I don't see any reason not to.

I do have a clone of my internal hard drive, thats the next one to encrypt. I'll check to see if he is still bootable afterwards, and I'll try to find an extra disk somewhere to restore a TM backup.

I'm really happy with PGP WDE, works great, no surprises so far.

Mercurial Plugin to use Git servers

The Github gang developed a Mercurial plugin, hg-git, that allows Mercurial users to push/pull from Git servers.

It seems very good (I'm not a Hg user, so I don't really know). Lossless bi-directional synchronization.

Nice.

April 21, 2009

Merging two unrelated repositories

I keep a repository for operations-style stuff. Server configurations, Puppet recipes, old CFEngine stuff, the works.

But for some idiotic reason long lost in time, my DNS setup was in a totally unrelated repository. This didn't make that much sense. Even more stupid: other stuff, unrelated to DNS, was also stored in there.

So I had:

  • a oss/ repo with the good and clean stuff to configure all the servers I manage;
  • a dns/ repo with all the DNS configuration stuff, and a bunch of other unrelated configurations.

And the goal: merge the history of the network/dns/ directory in the dns/ repo to the dnssrv/ directory of the oss/ repo.

Like this:

# create a copy of my dns/ repo
git clone dns dns_work && cd dns_work

# remove everything except the network/dns/ directory
git filter-branch --prune-empty --subdirectory-filter network/dns -- --all

# move stuff back to the dnssrv/ directory
git filter-branch -f --prune-empty --tree-filter '
   mkdir -p .dnssrv;
   mv * .dnssrv;
   mv .dnssrv dnssrv
' -- --all

# make sure we clean all the cruft
git gc --aggressive

# ok, prepare to merge
cd ../oss && git remote add dns ../dns_work && git fetch dns

# Merge...
git merge dns/master

# Remove the cruft
git remote rm dns && git gc --aggressive

And with that, all the history of my DNS work is merged back into the oss/ repo in the proper directory.

As a final step, I need to remove the network/dns commits from the dns/ repo:

# prepare...
cd ../dns

# Remove the old directory and any empty commits lying around
git filter-branch -f --prune-empty --tree-filter 'rm -rf network' HEAD

# Cleanup
git gc --aggressive

Done.

One of the common complaints I hear about git is that it allows you to rewrite the history. Although this operation can be very damaging in a repository that is heavily cloned, banning, or making history rewriting a second-class operation feels like banning hammers because you can harm yourself. Git is a tool, and it should make complex, but at times useful operations like this, easy or at least possible.

March 21, 2009

ssh client as SOCKS server

This is probably basic stuff for some of you but it was a surprise for me.

Your ssh command line client can work as a SOCKS server. Its actually very simple to use:

ssh -D 5566 remote.server

The -D 5566 will make ssh start a SOCKS (v4 and v5 supported) server at port 5566. You can now set your SOCKS server for your preferred apps as 127.0.0.1:5566.

The connections will be made by the remote.server to the destination host.

Very useful indeed.

March 19, 2009

x-test-continuous

I've pushed a new tool to my scripts repository at GitHub, x-test-continuous.

This script watches changes in the ./lib and ./t directories, and executes the command given as argument on any changes. Right now, I've hardcoded a one second delay.

My usual command while I'm writing a new Perl module is this one:

x-test-continuous prove -l -v -r t

Now, my tests are a single "Save all" away.

An even better usage, in case you are using TextMate. First, open your TextMate preferences and enable Advanced > Saving > Save files when focus is lost. Then, run this command on a tall terminal window:

x-test-continuous 'clear; prove -l -v -r t'

Now you can just click or Cmd-Tab between TextMate and your Terminal window.

This script only runs on Mac OS X 10.5.x for now. Need to learn how to use inotify on Linux.

February 10, 2009

Greatest gizmo ever

Buy it! It's fucking great!

February 08, 2009

Mental note

This should be obvious after you think of it, but it is a honest mistake and cleaning up the mess afterwards can cause you big problems, so here it is:

Running some_command --base=~/Sites/dir is very different from some_command --base ~/Sites/dir

The extra = makes all the difference. With it, the ~/ is not expanded to your $HOME by the bash shell, so if your some_command uses that path as a base directory, you'll end up with a new path like ./~/Sites/dir.

This means that in your current working directory, you'll have a subdirectory named ~, and you might be tempted to just rm -rf ~ to clean it up...

I think that we can all agree that a rm -rf ~ command would be disastrous (the correct command would be rm -rf ./~), so please be careful when using options with parameters. The = will disable shell expansion, and you could end up in a bad place.

February 03, 2009

Varnish ESI support

I have a love/hate relationship with Varnish, but partial support for ESI was added in recent builds.

ESI allows most of your site to be served from the Varnish cache, and smaller parts of the page, like personalization, still hit you application server.

Think old style server-side includes, but with decent cache control.

November 06, 2008

AnyEvent::Mojo 0.6

Uploaded to PAUSE AnyEvent::Mojo 0.6, should hit the CPAN mirrors throughout the day.

Update: uploaded 0.6001, with an attempted fix for some 100% CPU usage cases.

The major change is the renaming of AnyEvent::Mojo to AnyEvent::Mojo::Server.

There are two reasons for this change. The first is to prepare the release of AnyEvent::Mojo::Client in the next release, wrapping the Mojo::Client API.

The second is to free the AnyEvent::Mojo to be used with the new functional API.

Starting a Mojo server inside your AnyEvent apps is now very simple:

use AnyEvent;
use AnyEvent::Mojo;

my $server; $server = mojo_server undef, 4323, sub {
  my ($self, $tx) = @_;

  $tx->res->body('Cool!');  
};

$server->run;

The interface is similar to AnyEvent::Socket.

This is the first beta release. I don't expect to change the API in a backwards incompatible way until 1.0.

I do plan the following for the next releases:

  • implement AnyEvent::Mojo::Client and the correspondent mojo_client function;
  • implement support for 100 Continue.

Mojo is fun!

Oh, and please vote for the Perl Foundation grant to improve Mojo.

November 02, 2008

Other uses for Rasputine

Rasputine was born to connect Moo/MUD/talkers to the XMPP network, but in fact its a generic Telnet-to-XMPP gateway. This opens up a lot of interesting use cases.

You could add a buddy for each router that you have. Or one for each perlbal/memcached admin interface.

The internals are still a bit messy but after a small cleanup, you could even have two users, with their own Jabber ID, sharing a telnet session to a router. A kind of Pair-Configuration session.

Just think of all the devices you have with a Telnet administration console: Rasputine should be able to handle it.

October 31, 2008

Crazy QRCode tool

Crazy... A QR Code generator that outputs HTML.

To use:

cpan HTML::QRCode
perl -MHTML::QRCode -e 'print HTML::QRCode->new->plot($ARGV[0])' TEXT

Replace TEXT with the string you want to encode. For example, a QR Code for /notes/ looks like this in HTML:

Who said tables where dead?

Next step: search for a barcode-in-HTML generator.

git-gui screencast

Yesterday, I uploaded to Vimeo a screencast showing some of the features of git-gui.

With a UNIX background, and with only a brief detour to Windows NT 3.51 (the last Windows version with decent internals), I'm mostly keyboard oriented and never before I used a GUI to solve my SCM needs.

This changed with git-gui. With the Stage chunk and Stage line features, I'm much more productive with git-gui, and my commit history is cleaner and more logical to follow.

These features make it easy for me to commit an hours work into several discrete logical steps, and even do a quick commit to fix a small bug without a full cycle of git-stash/fix/commit/git-stash pop.

Anyway, enough courtship. Enjoy.


git-gui screencast from Pedro Melo on Vimeo.

October 29, 2008

Bitflu

My favorite BitTorrent client is Bitflu.

It is a head-less client that runs on a server somewhere, and you can manage it using a HTTP or telnet interface. It also supports a auto-start directory where all the torrent files you drop in it will be picked up automatically.

What it doesn't have is a way to upload a torrent file programatically.

At least, not until today. Check out my torrent-upload branch of Bitflu. It provides a new HTTP-based API to upload a single torrent file.

To use it, do this:

lwp-request -c 'application/x-bittorrent' -m POST \
  http://your.bitflu.site:4081/new_torrent  < my.torrent

The lwp-request command does not support authentication. I need to hack a Net::Bitflu module to make this easier.

The patch was sent to the author, but I don't know if it is something that he is interested on or not.

My current git setup

I've been using git for more than an year now, and I've settled on a repository organization that I like.

The foundations of this setup are:

  • a GitHub account: the public face of my repositories. A free account is enough, and although I don't have private repositories at GitHub, I'm using the Micro plan, a mixture of kudos to the GitHub owners, and my preference for HTTPS-based access;
  • gitosis: manages all my git repositories, both public and private;
  • my x-git-update-to-latest-version: I re-compile git every day with the master branch, to help catch regressions.

For each project I have, I keep the repository on all my computers in the ~/work/ directory. Those repositories are synced between work computers (desktop at the office, and laptop everywhere else) using a Unison profile.

The use of Unison gives me the possibility of leaving work half done in the desktop, and finish up at home if I feel like it, without having to commit and push from one to the other.

On each repository, I have two main remotes configured.

The origin remote points to my gitosis setup. You can use my gitosis install how-to for a painless setup. In addition, I add a Host entry to my ~/.ssh/config like this:

Host git
  HostName hostname.of.server.with.gitosis
  User git

This allows me to use a simple git:melo/repo.git as a remote URL. Short and to the point.

The second remote, named github, is only created on public repositories, and points to my GitHub account.

Manually I add a third remote named all. I copy the other two remotes urls to it. This allows me to git push all and have my changes pushed to both remotes with a single command. For example, for the AnyEvent::Mojo project, I have this .git/config file:

[remote "origin"]
    url = git:perl/anyevent-mojo.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[remote "github"]
    url = git@github.com:melo/anyevent--mojo.git
    fetch = +refs/heads/*:refs/remotes/github/*
[remote "all"]
    url = git:perl/anyevent-mojo.git
    url = git@github.com:melo/anyevent--mojo.git

This setup gives me peace of mind (all my code ends up on 4 different systems), and provides a pretty face (Github) for other to use.

Next step: create a script to make all this a one-step process.

October 25, 2008

Tip: a Macbook Pro without the battery will always hybernate

My laptop battery started expanding to new territories (the outside, really), and given that it is no longer under warranty, I ordered a new one online.

But for the last week or two, I've been working on my laptop without the battery, and in the process I learned something useful.

If I close the laptop lid, and wait for the light to start glowing, I can then unplug the power cord, move to a new location and plug back it again.

The Macbook will use the hibernation feature so I can keep all my state even without any power.

Much better than the shutdown/startup process I was using until I accidently noticed this.

October 19, 2008

Financial crisis explained

A video to explain the current financial crisis, by Marketplace Senior Editor Paddy Hirsch.

The best one I found so far.

October 18, 2008

Stitcho

There is this new network notification service called Stitcho. Think Growl (which they support) but with a HTTP-based API.

Its still in beta, but works pretty well. I've found some problems with UTF8 encoded strings and I have an open ticket on that.

I've released a Perl client library to CPAN, Net::Stitcho. It allows you to use it from your Perl programs easily. Its not feature complete yet, only the send message API is supported, but it should have the signup API today.

Its not clear to me the business model behind Stitcho, and getting this deployed widely enough to make a difference is also a challenge, but I like the concept so lets see how it goes.

Update: the Net::Stitcho module is, as of version 0.03, feature complete.

October 14, 2008

SIP is up and running

My new ADSL provider, Telepac, gives me a free SIP number, with free fixed-line calls, so I downloaded a soft-phone, X-Lite, and set to work.

The configuration is pretty simple. See the screenshot below:

Configuration for X-Lite using Telepac SIP

You can use whatever you want as a display name. The user name is the string +351 followed by the number you got. The password field should pretty obvious.

The two fields that wasted most of my time where Domain (must be voip.telepac.pt) and the Send outbound via: must be set to proxy with the string proxy.voip.telepac.pt:5070.

This configuration is enough for me to receive and place calls. There are still some things I would like to know, like if it is possible to call my number via another SIP provider.

Although I'm using Telepac VoIP service, the SAPO VoIP service is similar (if not the exact same one), so replace telepac.pt with sapo.pt and you should have a working setup too (untested).

So now I have a hard phone connected to the VoIP router at the office, and a soft phone running on my Macs. With this, I now have two VoIP accounts, Skype and SIP. Next step: get a XMPP client with Jingle.

If you want to say hi, +351 302 029 050. Use a fixed line for now, it seems to be mis-configured if you use a mobile network.

September 20, 2008

Push to multiple repositories

This I did not know. You can have several URLs per remote in Git, and git-push will update them all.

On a personal project I have this:

[remote "backup"]
    url = git@github.com:melo/perl-sapo-broker.git
    url = git@git:melo/perl-sapo-broker.git

A simple git push backup master and all is well.

Installing gitosis

Gitosis is a wonderful little system to manage Git repositories, providing access over SSH, with tight access control and using only one shell account.

The installation instructions provided with the README.rst, and the Hosting Git article by Garry Dolley provide you most of what you need to install it. But they cover the most basic installation where everything is in your system PATH.

My setup is not standard at all, so the process needs to be tweaked a bit.

Although not mentioned, Gitosis requires a recent version of Python (at least more recent than my system 2.3.4) and setuptools (also missing from my system).

I choose to compile all the dependencies. To isolate this as much as possible, I created an account gitdeps to hold all the stuff I need to run Gitosis.

I logged in as gitdeps and did:

# make sure other users can use this commands
chmod 711 $HOME
mkdir src && cd src

# Install Python
wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz
tar zxf Python-2.5.2.tgz
cd Python-2.5.2
./configure --prefix=$HOME
make
make install
cd ..
export PATH=$HOME/bin:$PATH

# Install setuptools
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

# Install Git
wget http://kernel.org/pub/software/scm/git/git-1.6.0.2.tar.gz
tar zxf git-1.6.0.2.tar.gz
cd git-1.6.0.2
./configure --prefix=$HOME
make
make install
cd ..

# Install Gitosis
git clone git://eagain.net/gitosis.git
cd gitosis
python setup.py install

You should have all the software needed to run Gitosis now.

The rest of the installation is pretty simple. You need a couple of things:

  • choose a directory to hold all the files: we will assume /home/git but you can use whatever you want;
  • a user account for the system: usually this user is git. You can have several Gitosis installations in the same server, each one using a different user;
  • the SSH public key of the user that will be the initial administrator of Gitosis.

To create the git user, you should use the proper tool for your operating system. The README.rst provides the command to run on a Debian-like system. I'm using CentOS so the command is this:

# As root
useradd \
      -s /bin/sh \
      -c 'git version control' \
      -r \
      -d /home/git \
      git
mkdir -p /home/git
chown git:git /home/git

After this, you just need to initialize the Gitosis system. Do:

# As root
PATH=/home/gitdeps/bin:$PATH
export PATH
sudo -H -u git gitosis-init < /path/to/gitosis_admin_ssh_public_key.pub

You should see two lines of output:

Initialized empty Git repository in /home/git/repositories/gitosis-admin.git/
Reinitialized existing Git repository in /home/git/repositories/gitosis-admin.git/

On a standard system, that would be it. But we have all the binaries in a non-standard directory, /home/gitdeps/bin. To make sure that they are found, we need to tweak the SSH instalation.

First, you need to create a SSH environment file with the proper PATH to use:

# as root
echo "PATH=/home/gitdeps/bin:/bin:/usr/bin:/usr/local/bin" > ~git/.ssh/environment
chown git:git ~git/.ssh/environment
chmod 400 ~git/.ssh/environment

Then you need to make sure that your sshd is configured to read the file. Edit the /etc/ssh/sshd_config file. There are two settings you must check:

  • PermitUserEnvironment: must be yes;
  • UseLogin: must be no.

If UseLogin is yes, proceed with caution. You might break ssh service for other users. One alternative (left as an exercise to the reader) is to use a separate sshd just for the git user.

Restart your sshd. And we are done.

To manage Gitosis, you clone the gitosis-admin.git repository. Inside your local copy, you'll find a gitosis.conf and a keydir/ directory with the public keys of all the users, in the format USER_ID.pub.

# on your laptop/desktop
git clone git@server.domain.here:gitosis-admin.git
cd gitosis-admin
ls -la *
-rw-rw-r--  1 melo  staff  91 Sep 20 15:44 gitosis.conf

keydir:
total 8
-rw-rw-r--  1 melo  staff  666 Sep 20 15:44 melo@simplicidade.org.pub

Have the appropriate amount of fun.

September 19, 2008

Dropbox is open

I missed the announcement a couple of days ago, but Dropbox is now open. They also released their Linux client.

Dropbox is what iDisk should have been: a simple, just works, way to share files between computers. Right now I have a couple of shared folders shared with other Macs and Windows boxes, without any problems.

Recommended.

TextMate Scratches

Scratches is a new TextMate bundle that I'm finding very useful.

Basically it allows you to take snippets of code, "scratch" them, and then reuse them on other places. A sort-of glorified multi-buffer copy&paste. I believe the inspiration came from a BBEdit feature.

The current version (see below for instructions to install) has a very nice scratch viewer. You can see how it looks with the small screencast (authored by Hans-Jörg Bibiko).

The Scratch bundle is still in the review section of the Macromates SVN. To install do:

export LC_CTYPE=en_US.UTF-8
export LC_ALL=
cd ~/Library/Application\ Support/TextMate/Bundles
svn co http://macromates.com/svn/Bundles/trunk/Review/Bundles/Scratch.tmbundle

Tell TextMate to reload its bundles (menu Bundles > Bundle Editor > Reload Bundles) and you should be ready to use it.

The cool part is that this bundle was created in a couple of days in the TextMate mailing list. If you are curious, the threads starts here.

September 16, 2008

Updated x-git-update-to-latest-version: now with man pages

I don't contribute with code to the git project, so the least I can do is use the master version daily.

As I explained previously, I have a automatic process to do that.

To keep my git up-to-date, I use my x-git-update-to-latest-version script.

This script updates my local clone of the git repo (localy at ~/work/track/git), and then configures, installs (at /usr/local/git-git describe) and updates the/usr/local/git` symlink.

This way, I can have /usr/local/git/bin in my PATH and I'm always using the latest version.

The latest version of this script also installs the man pages. You need to tweak your MANPATH to include the /usr/local/git/share/man directory.

For extra points, you can also run this script as a cronjob. Mine runs at 10:30 in the morning.

Update: Junio is on vacation so the master branch is not being updated. In the meantime you are encouraged to use the Shawn O. Pearce branch of git. I tweaked the x-git-update-to-latest-version to deal with the fact that Shawn's tree does not have tags.

September 05, 2008

Auto-save

Jack Moffitt was bitten by auto-save.

My auto-save setup is "Save when TextMate looses focus" but yesterday I was scripting something better that will be a great auto-save post-script.

When I start to work on something experimental, I would like to have a snapshot of every path I take and undo. Sometimes I write some code, and then say "naahhh, wont work ok", and undo it, without any record. And this is bad, because some of those actually were a good path after all.

Right now, the script does some git add voodoo and a git commit, so that I have a commit of the entire workspace each 2 minutes.

I'm still tweaking some details: I would like these commits to exist in a parallel repository, not my main one, so I'm playing with having a second .git control directory inside the main one.

Anyway, if you auto-save could also commit the file to a changes repository, then you would have all the changes since you started.

September 02, 2008

Pretty and useful

If you thrive to achieve a stress free life, and keep programming at the same time, I assume that you know how automated testing and test-driven development are an essential tool.

I've been using them for most (not all) of what I do in the last year or so. Basic stuff, using Test::More and friends, and more recently Test::Most and using the basic prove tool and Devel::Cover for extra peace of mind.

I was reading the latest edition of the Test Automation Tips and they mentioned Smolder and another Perl module. I knew Smolder already. It is one of those tools that I always keep on my list to install someday, but never get to do it because I expect it to be hard (without any reason to think that, mind you).

The other module is TAP::Formatter::HTML and it just blew my mind. Simple installation with cpan TAP::Formatter::HTML, a single tweak to my run_all_tests.sh script and gorgeous looking HTML reports. I don't have them online but you can look at a sample report (be sure to click around).

Strongly recommended.

August 27, 2008

Pretty print XML

In case you didn't know, you can pretty print XML with the xmllint command line tool that comes with libxml2, and its installed by default with Mac OS X.

Basic usage is

xmllint --format xml_file.xml

and the pretty version will be sent to standard output.

But working with XMPP and SAPO Broker, I'm always copy&pasting XML from one place to the other and it would be nice to format the XML snippet sitting in the clipboard.

This pipe does the trick quite nicely:

pbpaste | xmllint --format - | pbcopy

I wrapped this into a script, called x-xml-format-clipboard and now its just one command away from gratification.

Next step, create a version that I can place in the services menu. I need to find a way to access the selected text on the current Cocoa app. I hope thats not to hard. Making the final script a service is definitively easy, using the ThisService app.

August 20, 2008

Framing protocols

I've been playing with framing protocols.

I need a protocol with:

  • low overhead: it will be used for mobile communications and some of us pay for each byte;
  • multiplex streams: you should be able to open channels inside the same TCP connection;
  • restartable: each session has an ID and I need to restart all of the channels inside it very quickly if my TCP connection dies.

I've looked at AMQP wire protocol, BLIP, BEEP and some others. I plan to look over SSH too.

I'm looking for further suggestions of protocols to reuse. Anyone?

Parallax

I've talked a bit about Freebase over the last few months. I understand that a rich, semantic, structured database of knowledge is not something other people get excited about, but that happens to be my thing.

In the hope to excite others with Freebase, I recommend that you spend 8 minutes watching the Parallax browser in action. That kind of research tool is only possible with something like Freebase on the bottom.

Update: the code for the Parallax browser is open source now.

August 05, 2008

XML::LibXML braindead, or is it just me?

I spent the last 30 minutes chasing down this bug. The following tests should all pass but the last one doesn't:

use strict;
use warnings;
use Test::More 'no_plan';

use XML::LibXML;
use XML::LibXML::XPathContext;

my $parser = XML::LibXML->new;
ok($parser, 'XML::LibXML parser created');

my $xml = '<x:me xmlns:x="some_namespace" />';
my $xdoc = $parser->parse_string($xml);
ok($xdoc, 'Valid XML parsed');

$xdoc = XML::LibXML::XPathContext->new($xdoc);
ok($xdoc, 'Converted to XPathContext');

my ($node) = $xdoc->findnodes('/me');
ok(!$node, 'Not found because no prefix means NULL namespace');

($node) = $xdoc->findnodes('/x:me');
ok($node, 'Found because using document prefix');

$xdoc->registerNs( new => 'some_namespace' );
($node) = $xdoc->findnodes('/new:me');
ok($node, 'Found because using new specific prefix');

# Notice the change of namespace
my $uri = $xdoc->lookupNs('x');
is($uri, 'some_namespace', 'x prefix is some_namespace');
$xdoc->registerNs( x => 'other_namespace' );
$uri = $xdoc->lookupNs('x');
is($uri, 'some_namespace', 'x prefix is still some_namespace');

($node) = $xdoc->findnodes('/x:me');
ok(!$node, 'Not found because using document prefix was changed to diff namespace');

The basic problem is this: you get a XML document in which a namespace N is tied to a prefix P. I was expecting that if I set in my own parser prefix P to something else, that my local prefix P would take precedence over the document P. Apparently it does not.

Right now, what I did was this: before using XPath to match anything, I always used registerNs method of XML::LibXML::XPathContext to make sure I was getting the right elements. Now, I use this new function instead:

sub _safe_ns_register {
  my ($xpc, $prefix, $ns) = @_;

  while ($xpc->lookupNs($prefix)) {
    $prefix++;
  }
  $xpc->registerNs($prefix => $ns);

  return $prefix;
}

This function makes sure that the prefix I choose does not clash with any prefixes on the document. It returns the prefix I must use on my XPath expressions.

It solved my problem, but I still think that my local prefix should take precedence over the document one.

August 01, 2008

STFU

So you have a friend who is going to spend some time traveling and that journey includes the United States. Usually you would buying him a travel guide.

But our times require something different. I recommend that you send your fiend the link to Professor James Duane talk entitled "Don't talk to the Police".

In a country where habeas corpus is no longer controlled by the court system, those 26 minutes could save your friend a lot of trouble (as confirmed in the follow up presentation by a police officer).

(via Schneier)

July 21, 2008

Tweaks to MySQL installation

After you install one of the MySQL packages available for the Mac, there are some steps that you should do.

First, make sure your MySQL installation knows about time zones. This is important if you want to run your MySQL in the UTC time zone.

To update the mysql database time zone tables, do:

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql

Type the password at the prompt (hit enter if you don't have one yet).

And then make sure all your date/datetime fields use the Highlander-timezone. Edit my.cnf and add:

[mysqld]
default-time-zone=utc

Second, make sure your server is using UTF-8 everywhere. Add to my.cnf:

[mysql]
default-character-set=utf8

[mysqld]
character-set-server=utf8

Third, set mysql to strict SQL:

[mysqld]
sql-mode="TRADITIONAL,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY"

Finally, make sure you use InnoDB by default:

[mysqld]
default-storage-engine=InnoDB

There are probably more tweaks to make your MySQL saner. This are the one I feel comfortable recommending.

DBD::mysql and db_imp errors

I'm installing all my MySQL stuff in a Leopard 10.5.4 desktop and I made some mistakes along the way that I though about documenting here for future reference.

First, although the hardware and OS are 64-bit in a lot of places, the standard perl installed is not one of those. So stick with the i386 MySQL package (or try a 64bit server, but use the 32bit client...). I'm using the Proven Scaling MySQL packages mentioned earlier, and I'm happy so far.

Second, make sure your regular user has all privileges to the test databases. I just do:

melo@DogsHouse:lib $ mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 108
Server version: 5.0.62-enterprise-gpl MySQL Enterprise Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> grant all privileges on test.* to 'melo'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> quit
Bye

This will make your DBD::mysql tests much happier.

Third, in case you see failing DBD::mysql tests with:

Can't use dbi_imp_data of wrong size (127 not 124) at ...

Upgrade your DBI. I'm now with 1.605 and no dbi_imp_data errors anymore. Clean DBD::mysql install.

July 04, 2008

No more bazzilion git-* commands

In case you use the script from the last post, be advised that the current master branch of git.git no longer installs all those git-* on your PATH.

The current git/bin/ contents are:

melo@MrTray:melo $ ls -l /usr/local/git/bin
total 14664
-rwxr-xr-x   89 root  wheel  2826820 Jul  4 10:05 git
-rwxr-xr-x    1 root  wheel   573476 Jul  4 10:05 git-receive-pack
-rwxr-xr-x    1 root  wheel  2826820 Jul  4 10:05 git-upload-archive
-rwxr-xr-x    1 root  wheel   994596 Jul  4 10:05 git-upload-pack
-rwxr-xr-x    1 root  wheel   273636 Jul  4 10:05 gitk

I'm using version v1.5.6.1-204-g6991357. This is not the final 1.6 release (the next one), so you might see further commands added (git-shell might join this list).

At least for me, this required some training because I liked to git-TAB to complete...

x-git-update-to-latest-version

This morning in the git mailing list, I wrote a small shell recipe to update your git to the latest version but keeping the previous ones around, in case something goes wrong.

I noticed that what I wrote was a lot better than the hack-and-slash script I was using, so I promoted a cleaned up version of that to my scripts stash. Hence, you can now download x-git-update-to-latest-version and enjoy painless git updates.

You need to tweak two things at the top of the script:

  • the location on your hard drive of the git.git clone (you need to create that with git clone git://git.kernel.org/pub/scm/git/git.git first);
  • the base directory where all the git versions will live.

You'll end up with something like this (I use /usr/local as my base directory):

melo@MrTray:melo $ ls -dla /usr/local/git*
lrwxr-xr-x   1 root  wheel   25 Jul  4 10:05 /usr/local/git -> git-v1.5.6.1-204-g6991357
drwxr-xr-x   7 root  wheel  238 Jul  4 10:05 /usr/local/git-v1.5.6.1-204-g6991357

Each version will be named git-VERSION where VERSION is the output of git-describe. A symbolic link named git points to the latest version.

Just add BASEDIR/git/bin to your PATH and your are done.

Small tweak to my mail setup

I try to keep my inbox empty, but due to a lack of a task manager that I can feel good about, I don't have a place to put pending tasks.

So sometimes I leave them on my inbox. Not good.

Until I find a good system that I like to use, to keep my tasks and projects, I made a small adjustment to my mail habits. I already have an extensive list of rules to filter mailing lists and other regular emails to proper folders.

What I did now was this:

  • created an Interesting Folders smart folder: basically, it joins together the folders (either mailing lists, or my regular inbox) that I want to keep a close eye on;
  • created a Quick Read smart folder: simple condition - show everything from Interesting Folders that is not read.

So each time I open up my email client, I just look at the Quick Read folder. It shows only new mail messages that I haven't read yet. The thing I like the most is that if I need to do something later, I can just leave it there and it will disappear on my next check, but still be safely stored on the original folder.

It's not perfect by any means, but its working very well for me.

July 02, 2008

MySQL advice

When people ask me what MySQL to use, I used to respond "Go to http://dev.mysql.com/ and download the community edition". I recommend it over any version of MySQL that is bundled with your OS.

But I also listen to people who know more than me when it comes to MySQL, and one of those just asked (and presented facts) if the those binaries are in fact dead.

So right now, my new advice on MySQL choices is this: read the latest MySQL Performance blog article and decide what you want.

I'm going to test the Jeremy Cole's releases of MySQL to see if they work correctly with my environment, and I'll keep you posted.

One thing will still send me to dev.mysql.com, though: Mac OS X binaries. Jeremy only has RedHat Enterprise Linux RPMs Update: I was wrong. At least the Enterprise version has Mac OS X binaries. Thanks to Joaquim Carvalho for pointing them out for me.

June 30, 2008

Behind the scenes of Tarpipe XMPP

Alex and Adam asked me to elaborate on the tools I used to implement the Tarpipe XMPP gateway.

I used the Net::XMPP2 Perl module, in particular the Net::XMPP2::Component class. It uses the AnyEvent async framework, which in turn support the EV library , giving you all the love of kernel polling.

Until the Net::XMPP2 author releases a new version, you should use my own copy if you plan on doing external component work (check the component-reply-with-from branch). Net::XMPP2 is widely used for bots, but not that many components, and some methods need some love to work properly in a component environment.

The HTTP part was done using the excellent AnyEvent::HTTP class. I'm using my own version, which includes a bug fix to the http_post function (on its way to release 1.03 of AnyEvent::HTTP) and adds support for HTTP::Request to http_request. I hope to see this included in the main AnyEvent::HTTP distribution but I still need to update the documentation.

The rest is just glue code.

June 28, 2008

*jour

On the topic of Bonjour goodies, take a moment to read about, and install, some of the *jour tools.

Very cool stuff.

Bounjour, CPAN!

For a Perl programmer, a local (on your laptop) CPAN mirror is a worthy investment. The problem is that a full mirror is 5.8Gb of disk space. Fortunately we have CPAN::Mini that creates a mirror of the most important stuff using only 830Mb.

So now you have your local mirror, and after you add the path to your cpan urllist configuration, all your module installations will use this faster mirror.

But you shouldn't stop there. If you are using a Mac with 10.4.x or above, you can share you CPAN mirror with the others on your local lan, and announce it proudly using Bonjour.

To do that, just follow these steps. First create a Apache configuration file at /etc/httpd/users/. I called mine cpan.conf and it looks like this:

#
# My local mini CPAN mirror
#

Alias /cpan/ /Users/melo/Documents/cpan/
<Directory "/Users/melo/Documents/cpan/">
    Options none
    AllowOverride none
    Order allow,deny
    Allow from all
</Directory>

RegisterResource "Local CPAN Mirror" /cpan/ 80

This will share your CPAN mirror (change /Users/melo/Documents/cpan/ to the path of your local mirror) and announce it via Bounjour with the name "Local CPAN Mirror".

Make sure that you start Web Sharing, at System Preferences, Sharing. If yes, stop and then start to load the new configuration file.

To use this CPAN mirror from other computers, you just start cpan, and then type o conf urllist URL where URL is the URL advertised.

Reconnoiter

Something to pay attention to, Reconnoiter.

Its still work in progress but in the last status report, Theo Schlossnagle mentions that he is already using this software to monitor about 2.9k services with less than 0.10 load, peek.

That's impressive.

June 27, 2008

MySQL master/master how-to

New location: the How-to is now live at /how-to/mysql-master-master/.

In the past, I needed to use a MySQL master/master setup for a client. At the time, I got it to work, but forgot to take notes about the process.

In the last days, a friend asked my for help to do a setup like that again, to use with a Tigase XMPP server.

So I wrote this how-to, complete with configuration files, sample data and schemas, that walks you step-by-step through the process of setting up a pair MySQL servers in a master/master configuration.

You can fetch the entire how-to with git:

git clone git://github.com/melo/mysql-master-master-how-to.git

The how-to is written in MultiMarkDown, but a simple Markdown should also format it correctly.

A tarball with everything is also available.

Please let me know if you find any mistakes. Pull requests are the preferred form of collaboration :).

Update: included some tweaks from dbr, and added a HTML version with proper CSS. Download again, and look at how-to.html.

June 25, 2008

Git starters

In the last two or three weeks, I had about five or six people IM'ing me "What's the best way to start with git?". Last time I wrote about this was almost 7 months ago, so I think its time for an update.

First, I personally compile git from source. I do not trust package maintainers yet. Some parts of git use ssh (for example), so they mark ssh as a dependency, and this brings their own version of ssh. Most often that not, I ended up with a broken ssh on my system. So, compile from source is my personal recommendation for now, until package managers understand system-ssh.

Second, git is still being developed at a fast pace. I recompile it once a month at least, and I follow the master branch. That is, after I compiled the latest stable release from a tarball, I clone the git repo and I install the master branch version. Usually a make distclean && autoconf && make && sudo make install does it for me.

You will be happy with the official releases. There was a time that you had to wait a lot between releases, but now, they are coming out at a fairly decent pace. So sticking with the stable tarballs is a good enough option.

Regarding actually using git, I have three recommendations:

If you like dead-tree-stuff like me, watch out for the Pragmatic Version Control Using Git book (now in beta, PDF available), by Travis Swicegood.

For hosting your git repos, you can't go wrong with either Github or Gitorious. I'm using the fist one because I like all the user interface niceties, but the second is also available as GPL project that you can use on your own site.

June 16, 2008

How projects use git: Buildr

The best articles about distributed version control systems for me, are not those who explain the internals and the user interface, but how specific projects use those tools to get things done.

I've come across posts about several projects and how they use git and mercurial, and I'm going to start publish them here.

The first one is a couple of months old but still good: Buildr developers and how they where using a git-svn mirror.

Have fun.

June 04, 2008

Simple Boujour-enabled command line copy/paste service

Sometimes I need to send someone a bit of code in the local LAN, or even to myself on the second Mac.

It would be nice to:

pbpaste | publish_local_lan

And on the other side:

receive_local_lan from_melo | pbcopy

There are a lot of tools like that the run under Mac OS X, but today I found one that can be run from the command-line: pastejour.

Installation is trivial:

sudo gem install dnssd
sudo gem install jbarnette-pastejour --source=http://gems.github.com

And after that, just use it. It will use your short name as the broadcast key. A sample session:

melo@MrTray:melo $ date | pastejour 
Pasting melo...

# in another computer

melo@Screen:melo $ pastejour melo
(melo from MrTray.local.:42424)
Wed Jun  4 10:40:54 WEST 2008

Simple and effective.

June 03, 2008

RailsConf 2008 Git Tech Talk

Scott Chancon did a huge presentation (523 slides...), Getting Git, at RailConf 2008. Don't be scared by the number of slides, the presentation is excellent and you'll end up with a huge knowledge of git.

Extremely recommended.

June 02, 2008

Easy Git collaboration in local LANs

Doing quick hack sessions in a local LAN with friends using git just got a lot easier.

Evan announced gitjour, a Bonjour-enabled Git server. You can start a server for any repository on your laptop/workstation and others can browse the available repositories and easily clone them.

There is already a lot of work going around gitjour by a couple of developers, so it has a nice future ahead.

Nice...

May 30, 2008

Making better use of the XMPP presence in Bots

The current crop of the XMPP-based bots is pretty basic. They provide an online presence in the XMPP network, and you interact with them by a command-line-style interface.

These are some suggestions that bot authors can use to increase your XMPP presence easily. I'm going to stick to easy stuff only, most of it widely deployed.

The first thing that you need to understand is that you can have different per-contact presence information. You can switch the status message and the avatar for each of your contacts. Its called directed presence (basically include the contact JID in the to attribute, but you can always check the spec for directed presence).

You can use this to provide a personalized status message for each person on your roster. If your service has a notion of context, you can use the status bar to show the current context. For example, imagine that you have a ticketing system bot. You type "working on TICKETID", and your bot can change the status message to "Working on: TICKETID - TICKET_TITLE (URL of ticket)".

Optionally you could update the status each five minute with the elapsed time since you started working on it.

You can also switch the avatar. Maybe switch to a red background, meaning that you are busy doing something, or putting it in another way, the bot is telling you that he thinks you are working on something.

Other stuff you should do is to provide an alternative representation of the information you send textually. Twitter (when it used to work...) was a great example of this: every tweet you received by IM, included an Atom entry with the most important information of the tweet. This allows client to improve their interaction with such services, by using a more appealing interface.

But Atom entries are not yet standardized by the XMPP Foundation (there is some effort on Atom over pubsub that could be used as an example though). On the other hand, Data Forms are a standard, and more, they provide interaction possibilities.

You could send a "New ticket" notification, including a data form with the structured information, and a select-box with the possible next steps. The user would see the form, and submit back their decision.

Still in the subject of messages, you should know that there are several types of messages. You have chat, headline and normal (strangely enough the default). There are two more but not important in this context. You should use the correct one depending on the type of message you are sending.

If your system is just sending notification, please use the headline type. It makes it clear to clients that they are not chatting with you, just notifying of something that happened. For example, a smart client can react to those messages differently based on your status: in away, xa (extended away) or dnd, a client can keep a log of new headlines and show it like a email client when you return, without interrupting your workflow, probably using a badge with "You have X pending headlines".

The final recommendation: respect the user status. If the user is dnd, are you sure you should send him anything? Why not change the status message to "You have 5 messages pending. Send 'pending' or go to URL to see them"? Maybe even change the avatar color. You should not disturb someone in dnd. I would also say that away and xa are off limits but that's just me.

On the other hand, you should send "Hey there, welcome back! You have 5 pending messages. Send 'pending' to see them or hop on to 'URL'." when the presence changes to chat or available (not a real <show> value, just lack of <show> value. See the valid values of <show> and their meanings in the spec).

Have fun implementing your bots.

May 28, 2008

Fluid.app + Google Gears = Sweet!

This is great news: Todd Ditchendorf just released a nightly build (link removed) of Fluid.app that includes Google Gears.

I'm a big fan of Google Gears, and Fluid is getting a lot of love regarding integration with the desktop.

BTW, this is a good reason for me to upgrade to Leopard.

update: Todd giveth, Todd taketh away.

TextMate: a new take on Search with Ack

Apparently Trevor Squires found my Search in Project With ack TextMate command at the time it was 404, so he wrote is own.

I haven't tried it yet, but from his description its a lot better than mine.

May 27, 2008

Dan Geer at Source Boston 2008

Watch the video of Dan Geer at Source Boston 2008. Worth every minute of it.

May 26, 2008

Textmate: Search in Project with ack

Just a quick note to point out that my "Search in Project with ack" command for Textmate was updated.

May 14, 2008

My keyboard shortcuts

Inside the "Keyboard & Mouse" preference pane of Mac OS X, you'll find a tab named "Keyboard Shortcuts". This is one of my first stops after any nuke and pave setup of my Macs.

I don't have many shortcuts:

My Keyboard Shortcuts

The Take Rich Note and Append Rich Note integrate all applications with DevonThink. I select what I want to keep, and hit the proper sequence to save it in my default database.

The Quit Safari shortcut prevents me from closing Safari by accident. Its specially useful if you have a lot of tabs open. Recently Safari gained options, like the Reopen All Windows From Last Session, that make this less useful but I still use this.

The Select Next Tab and Select Previous Tab work around the fact that the default shortcuts in Safari for those options just don't work with some international keyboards (like my own, PT-layout).

The final shortcut is something new, that I'm trying out. It gives you a global shortcut to Zoom any window. Not sure if it's a keeper.

April 30, 2008

$science++

Got to love science:

Men who said they had sex twice a week had a risk of dying half that of the less passionate participants who said they had sex once a month, Dr. Davey-Smith’s team said.

via Justin Mason.

So, print a couple of copies of the study before you leave for the pub/bar tonight. You got science on your side now.

April 24, 2008

MySQL optimization quick tips

I'm not a expert on MySQL, but I spent a better part of this past night optimizing a server, and I've collected some notes. This is mostly targeted at InnoDB-based tables.

To change this settings, you should edit your my.cnf and update them in the mysqld section. See the MySQL System Variables manual page for more information. Please remember to keep a good working backup of your previous my.cnf. Better yet, include your my.cnf in your source control tree. I assume that you already have tested backups of your data...

So, do this steps:

These are the basic settings that I pay attention to. Of course there is much much more settings that you can tune (transaction-isolation and innodb_flush_log_at_trx_commit come to mind), but the things above will cover most of what you need.

After this, you should also use the mysqlreport tool. It requires more investment on your part, understanding the data, but its very thorough.

Finally, install Maatkit and get used to the tools it provides. They are essential if you are using replication.

April 17, 2008

Safari 3.1.1

In case you missed it, check your Software Update, there is a important Safari upgrade in there.

I classify this as "important" not because of the security fixes included, but because Mail.app is now working properly again.

With the last Safari 3.1, I stopped being able to paste plain text inside a mail message. I would copy something from TextMate or a Terminal window, and when I pasted it into a Mail.app message window, the line breaks would be lost.

This is most likely related to the same bug that Gruber complained about.

Anyway, Safari 3.1.1 fixes it, so I would strongly recommend that you upgrade to it.

Network throughput "problems"

A friend of mine was complaining that he could only upload a file to my server at 2Mbytes/sec with his FIOS link at home. Some people have interesting "problems".

Anyway, I send him my /etc/sysctl.conf that I have on my Mac for quite some time without any problems, and with significant gains in network performance on my local LAN (specially for other computers with Gigabit ethernet).

net.inet.tcp.mssdflt=1440
kern.ipc.maxsockbuf=800000
net.inet.tcp.sendspace=400000
net.inet.tcp.recvspace=400000

Stick those lines in your /etc/sysctl.conf and they will be active every time your Mac reboots. Make sure you place them on all your Macs. To activate them right now, run this:

sudo sysctl -w net.inet.tcp.mssdflt=1440
sudo sysctl -w kern.ipc.maxsockbuf=800000
sudo sysctl -w net.inet.tcp.sendspace=400000
sudo sysctl -w net.inet.tcp.recvspace=400000

You should see network a better network throughput immediately.

There is a fifth setting:

net.inet.tcp.sockthreshold=0

but I admit not to fully understand the implications of his use. Apparently without it, the sendspace/recvspace settings would not work, but my tests tell me the oposite. I'll update this when I do know if/when to use the one.

For reference, a local FTP transfer to a 100Mbit server was not using all of the bandwidth. With just the first line, I was able to saturate the server link. With all the lines, over a gigabit ethernet link, I was able to reach 35Mbytes/sec between two Macs. Without: a meager 6Mbyte/sec.

All the gory details are available in a study by the Pittsburgh Computing Center.

Update: sorry, miss-typed the values, remove the extra zero.

April 11, 2008

GitHub network view

Speaking of GitHub, they are now open for business.

With the official launch they enabled a couple of features that had been discussed in the past: commit comments (I still don't know if I like them or not), and integration with Campfire and Lighthouse (although you can also roll-your-own integration using their hooks).

One feature though, was not on the roadmap, and its a beauty: an network graph visualizer.

Its a great view across several repos, and it works very well. As far as I know there is no tool on the desktop that gives the same view. Not even gitk.

Most excellent.

Cairo with Quartz support

The 1.6.0 release of Cairo moved the Quartz back-end from experimental to supported.

This is very good news for me because some nice Perl modules for charts use Cairo.

If you are using FF3 betas on a Mac, you're already using Cairo by the way.

Update: an article from one of the authors.

April 01, 2008

Binary XMPP

Last February, Google's Android Team shocked the XMPP world by having the gall of saying that they where moving to a Binary encoding of their GTalk API in the M5 release.

Of course this sent shock waves through the XMPP community, not because they where totally wrong, but because they didn't have the vision to solve it.

We of course understand that they have their hands full (of vapor, some bad mouths would say...) getting the platform in shape for the late 2008 release, and the XMPP.. err, sorry, GTalk protocol is not a top priority.

Well, today Android users around the world can rejoice because once again, the XMPP Standards Foundation has risen to the challenge and unveils to the world the solution to the "verboseness" problem, by introducing the breakthrough Binary XMPP protocol.

This has been in development for quite some time, taking long hours of (sometimes) heated discussion. The goal is to have something that nobody could ever accuse of being verbose (the symbol count is extremely low, and the meaning is clear from the start), but at the same time remain compressible for those extreme cases where bandwidth is at a premium (we tested, and we have code to prove it: a Binary XMPP stream can be compressed to 2% of its original size). We know of clients that live in five-sides-shaped-buildings that, with their hard 9600 baud limits, will be the first to use this.

Its been a great ride, and it feels me with immense joy to be able to put my small signature as a co-author of this spec. Of course, my part is extremely small when you have the documentation genius of Peter Saint-Andre, and Fabio Forno strict guidelines and requirements, as co-authors. Its been a pleasure to work with both on this. And I should also thanks the important Kevin Smith contributions.

And now time to rest. SRV records are up at simplicidade.org domain, and Fabio's CM is running also.

PS: and yes, now you know why the prolonged silence around here.
PPS: for those who have moved on, the code is also available on GitHub.

March 26, 2008

MySQL errno 150 - ERROR 1025

I get the 150 errno a lot when upgrading schemas. Usually this is a indication that the action I was trying to perform is not possible given the current foreign key constraints in place.

Last night I got a variation on the theme:

ERROR 1025 (HY000): Error on rename of './e3/#sql-17f3_f894' to './e3/cls_ev_items' (errno: 150)

The table was this:

CREATE TABLE `cls_ev_items` (
  `id` int(11) NOT NULL auto_increment,
  `evaluation_id` int(11) NOT NULL,
  `criteria_id` int(11) NOT NULL,
  `value` varchar(50) default NULL,
  `modified_at` datetime NOT NULL,
  `rank` int(11) NOT NULL default '0',

  PRIMARY KEY  (`id`),
  UNIQUE KEY `report` (`evaluation_id`,`criteria_id`),
  KEY `criteria_id` (`criteria_id`),
  CONSTRAINT `cls_ev_items_fk_evaluation_id` FOREIGN KEY (`evaluation_id`)
      REFERENCES `cls_evaluations` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
  CONSTRAINT `cls_ev_items_fk_criteria_id` FOREIGN KEY (`criteria_id`)
      REFERENCES `cls_ev_criteria` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8

I was trying to drop the report index to replace it with a new one.

ALTER TABLE cls_ev_items DROP INDEX `report`

The error message is not clear enough, but in this case, the problem is that each CONSTRAINT requires an index on the foreign key field. If MySQL allowed the removal of the report index, he had no way to efficiently check the cls_ev_items_fk_evaluation_id constraint.

The best solution I could come up with is a temporary index on evaluation_id. You are then free to mess with the report index.

After you finish, if the new report index begins with the evaluation_id field, you can drop the temporary index. And everything is back to normal.

It clear the MySQL checks this chaining of indexes and constraints, so I hope to see better error messages in the future.

Update: the error log gives a bit more info, btw:

080326 10:51:32  InnoDB: Error: in ALTER TABLE `e3/cls_ev_items`
InnoDB: has or is referenced in foreign key constraints
InnoDB: which are not compatible with the new table definition.

So in case of a 150 error, check the error log for better pointers.

Update 2: as pointed out by NiN in the comments, run SHOW INNODB STATUS\G (the \G will make the report easier to read) and look for a section labeled LATEST FOREIGN KEY ERROR. The message is pretty good there.

March 20, 2008

Dear lazyweb: CSS question

This is something that I want to do for quite some time, but I haven't figure it out how.

If I have a HTML table styled with width: 80% but the total size of the content on each <td> is smaller than the 80% size, then the browser will try to make all the columns the same width, and add extra spaces at the right side of the cell.

What I wanted is to specify that some columns should shrink to the smallest possible size without causing line breaks in the content.

For example in an invoice with 4 fields, item description, quantity, value and sub-total, the default layout would be something like this:

| item        |         10 |         100 |        1000 |

and I wanted this:

| item                               | 10 | 100 | 1000 |

I wonder if there is some CSS combination that does this, that I can apply to a <colgroup> to say "give me the minimal possible width for these columns.

Any tips?

March 13, 2008

Link files in Catalyst error messages to Textmate

When you develop with Catalyst, if you have an error condition, you get a pretty interface with access to all the major objects in the request.

At the top, Catalyst will place the classical perl error message like "Caught exception in MODULE, at FILE line LINE."

This hack takes that classical format and links the "FILE line LINE" with a txmt: link. If clicked, it will open directly into your project in TextMate.

The code is simple. Stick this into your main application class:

sub finalize_error {
  my $c = shift;

  $c->NEXT::finalize_error(@_);
  return unless $c->debug;

  my $error_msg = $c->response->output;
  return unless $error_msg;

  $error_msg =~ s{(\s+at\s+)([\/]\S+)\s+line\s+(\d+)}
                 {"$1<a href='"._mk_textmate_link($2, $3)."'>$2 line $3</a>"}ge;
  $c->response->body($error_msg);
}

use Cwd qw( abs_path );
sub _mk_textmate_link {
  my ($file, $line) = @_;

  my $abs_file = abs_path($file);
  return "txmt://open/?url=file://$abs_file&line=$line";
}

It works for me so far. If this breaks anything for you, you get to keep both parts.

Here is a sample of the output with this hack applied (click for bigger version):

Safariscreencapture004

Update: a new version. The big change is the use of the abs_path method to make sure you get the absolute path. This solves problems that I was having with symbolic links. TextMate was opening a new window, because the project and the file path in the error message had different prefixes.

March 07, 2008

perl warnings

You have to learn to ignore the forrest.

There are some perl warnings that hide the real problem. My most hated perl warning is this, the first three lines below:

"my" variable @prob masks earlier declaration in same scope at sbin/some_script line 1640.
"my" variable $count masks earlier declaration in same scope at sbin/some_script line 1641.
"my" variable $t masks earlier declaration in same scope at sbin/some_script line 1642.
syntax error at sbin/some_script line 1504, near "next "

Those lines are there because the parser had to bail out after detecting the error on line 4, and failed to notice the end of scopes.

This could be less of a problem if the warnings and errors where ordered by line number, but they are not. So learn to look at the line numbers first to decide which warning to pay attention to.

Ultimate Game

A most excellent Ultimate Game strip at xkcd.

By the way, in case you haven't figure it out yet, half the fun of xkcd is usually buried in a alt/title tag on the image, so always hover over the image to see it.

March 06, 2008

Search in Project with Ack command for Textmate

One of the biggest problems I have with TextMate and large projects is "Find in Project":

  • it is a bit slow;
  • it searches everywhere, even files it shouldn't.

There is a nice alternative that uses grep by Henrik Nyh. But I'm a big ack fan, so I hacked Henrik command to use ack, and the result is Search in Project with ack-command for TextMate.

A future version might move to tm_dialog. Maybe I'll copy the GrepInProject++ as the basis for the next version.

Update: ok, this has been working better than expected. The difference in speed makes me use it without thinking about it. Also, it does not block TextMate. So for really huge projects, you can start it and keep working on your code.

Update 2: oops, a disk problem and the link went 404 on me. I've updated the code to the most recent version I'm using. The uuid of the command is changed so you might need to remove the old version before updating.

March 04, 2008

git hosting sites

A couple of weeks back I started looking for Git hosting sites. My two picks where Gitorious and Github.

At the time, and in the weeks that followed, I was too busy to start comparing the sites. But this week I want to start pushing my code out to these sites to see how they work.

In terms of feature list, Github received a lot of love in the past weeks, as you can read from their new blog. But Gitorious is not standing still either. Both are working to provide an easier collaboration feature around git pull which is very cool.

Right now, based solely on the feature list, I think Github has a slight advantage. If I had to point one Github feature that stands out, I would have to pick a Web Hooks-powered Post-Receive hook.

update: arrgh, I'm a moron. I forgot to add my Github password to my Keychain, so I'm locked out of my account. There is no "recover password" functionality on Github yet (its on the Feature Requests page though) so I sent an email to their support account, and I'm hoping for the best.

update 2: Ok, access to Github restored, big thanks to Tom Werner. My profile page is at http://github.com/melo/. I have 5 invites now, so if you want one, send me an email or ping me via XMPP/Jabber. My address for both is melo @ simplicidade.org.

February 22, 2008

todo.pl

I'm trying Hiveminder again.

My first attempt some months ago was not successful, and I must admit that although the new interface (specially the IM-based one) is very easy and fast to use, I still see the lack of offline-mode the biggest barrier to use this type of services.

Be sure to check pjf presentation Effective Procrastination with HiveMinder, its really good.

One thing that I was having problems just now was the todo.pl command line interface. Apparently, it doesn't like characters with accents. I looked at the code and it seemed to deal with them fine, figuring out the encoding of the terminal and using binmode to set the encoding filter of STDOUT to the proper charset.

But still, I was getting this errors: "\x{00e3}" does not map to ascii at /Users/melo/bin/todo.pl line 276.

Part of the problem is that terminal encoding detection fails on Mac OS X terminal. The problem seems to be inside I18N::Langinfo, because he reports us-ascii, but thats a XS module, so it might take a while for me to figure it out.

In the meantime I added a encoding setting to the todo.pl program, to force a specific encoding. Apply this short patch to todo.pl, edit ~/.hiveminder and add a line encoding: utf-8 and all should be well.

I don't think this patch is worth pushing upstream right now. The correct fix is in I18N::Langinfo, but I needed to get some work done today.

Update: fixed big 404. Sorry bout that.

February 19, 2008

Working with Wikipedia data

The good people at Freebase launched WEX, a Wikipedia Extraction tool, and made their exported data downloadable.

This allows you to use our usual text, XML or even SQL processing tools to analyze Wikipedia entries.

I mention this because the FreeBase project is one of the most interesting thing happening in the online knowledge space. If you haven't noticed, this project is building a huge structure data repository, open and extremely linked, with strong meta-data.

You should check it out.

February 16, 2008

ezmlm survival guide

Every time I have to do something with ezmlm-idx, I end up reading a lot of documentation.

It really tells you something about the quality of ezmlm. The need to tweak it is rare, so I forget how it all works.

Anyway, this time I wrote a couple of notes, a small survival guide if you will.

Continue reading "ezmlm survival guide" »

February 15, 2008

Apple TV as a AirTunes destination

A quick tip that I just read, and I didn't see this mentioned anywhere else: the Apple TV, with the latest software update, is now a AirTunes destination.

This is very cool. I can send the music from my iTunes to the big speakers easily now.

February 14, 2008

High-performance SSH

A very interesting patch to the standard OpenSSH code base, to improve SSH throughput.

Something to try in a week or so.

There is also a TCP tuning article at the same site worth a read.

Operations Mantras

A big Operations Mantras article by Alan Kasindorf.

Loads of interesting stuff, but it does take a bit to digest.

February 13, 2008

git hosting

If you need to host your project somewhere using git, I suggest you try gitorious.org.

Setup was a breeze, painless and just worked. I was up and running in less than 5 or 10 minutes.

I tried repo.or.cz a couple of days ago, but I still can't push to a project I created there...

February 09, 2008

github.com

There is a new service starting up, github.com, that provides Git hosting including a very nice Web interface with some cool twists like LightHouse integration.

Another cool feature is the read-only Subversion repository of the git master branch.

Unfortunately the service is still in beta. I've subscribed for an account, I hope they open up more slots soon.

Sniffing browser history

Came across this article about using JavaScript to sniff your browser history.

Very interesting stuff. I wonder how long it will take to see a jQuery plugin for it?

(via 43 folders)

February 08, 2008

.bash_completion.d

André Cruz asked me why wasn't his ~/.bash_completion.d/ being used by default.

Well, its a feature of the bash_completion system that you must activate by hand. In my ~/.bashrc I have the following code:

bash=${BASH_VERSION%.*}; bmajor=${bash%.*}; bminor=${bash#*.}
if [ "$PS1" ] && [ $bmajor -eq 2 ] && [ $bminor '>' 04 ] ; then
  if [ -f ~/bin/bash_completion   ] ; then
    BASH_COMPLETION=~/bin/bash_completion
    BASH_COMPLETION_DIR=~/.bash_completion.d
    export BASH_COMPLETION BASH_COMPLETION_DIR
    . ~/bin/bash_completion
  fi
fi  
unset bash bmajor bminor

The trick is the BASH_COMPLETION_DIR setting before you source the bash_completion script. The BASH_COMPLETION environment is required because I keep a local copy of script.

I'm using the latest version of bash_completion, 20060301. If you download the 20060301 tarball, check the README file. There is a FAQ at the end that mentions this and other cool tricks.

February 07, 2008

git bash completion

If you download the git tarball, you'll find a git-completion.bash script that adds shell completion to your git day-to-day usage. Look for it in the contrib/completion/ directory. (update: if you can't find it there, look for it in the $sharedir/git-completion/ directory. A recent patch by Johannes Schindelin promoted git-completion.bash to the big leagues.)

When I started using git, I just sticked the script into my ~/.bash_completion.d/ and bash picked it up after a exec bash -login.

One thing I didn't do was read the file, and I missed some cool stuff. One of them is a PS1 hack to show the branch you are on:

4) Consider changing your PS1 to also show the current branch: PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

The argument to __git_ps1 will be displayed only if you are currently in a git repository. The %s token will be the name of the current branch.

Be careful to use '' and not "" in the PS1= statement.

If you have several branches active at the same time, this really helps.

February 05, 2008

git gc --auto

For normal usage, git is written to be as fast as possible. That means that certain house-cleaning tasks are not done on every command and from time to time, you have to let the git gremlins cleanup the place.

For the last couple of git releases, the most common command to do that is to use git gc --auto. This will check the repository, see if it needs cleaning, and perform the necessary operations. Its a safe command, you can even start it, and keep on working on the same repository.

Personally, I run it when I remember or when git gui reminds me that I have to many loose objects.

In case you might be wondering the sort of impact this can have on your repo, I'll quote an extreme case. Notice that the user didn't use git gc --auto but choose a different, more aggressive, set of options. The tradeoff is execution time and you cannot use the repo while git gc runs.

I used git-svn to import a repository with 33000 revisions and about 7500 files. It took about 18 hours to import. When it was done, my .git folder had 242001 files that comprised 2.0GB. I ran git gc --agressive --prune and let that sit overnight (I wish it was more verbose, it went for over an hour without printing anything), and that managed to compress the repo down to 334 files and 64MB.

So from time to time, don't forget to git gc --auto.

January 11, 2008

PHP bashing and PHP prase

For my PHP friends out there:

Even if you are not a PHP user (left that wagon around PHP4), the setup wizard bundled with the Simplicity framework is very cool, an extjs-based application to create the application, including designing your database schema.

January 10, 2008

Network Mafia

In a stunning move, Network Solutions renamed it self as Network Mafia.

I wonder if they also do this if I query the whois database from another registrar website.

January 05, 2008

Regarding reverse proxys

Right now I use both Perlbal and Lighttpd as reverse proxy.

Lighttpd does a lot more than reverse proxy, and I also use it as a web-server for static content and mod_secdownload for certain content. But the current stable 1.4.x version is not that good reverse proxy. For example, it does not keep persistent connections to back-end servers, and sometimes he thinks all my application servers are dead when they are in perfect health.

Perlbal is a much much better reverse proxy. It keeps persistent connections up to back-end servers, and before using them for client requests, it makes sure the server is really ready to answer requests. It also caches file uploads to disk before allocating a back-end server to process the request. All this features keep the back-ends pretty busy and without stalls. But as a web server for static content, its just not as good as Lighttpd.

I've been working with three upgrade paths.

The first is use Perlbal for reverse proxy for the application servers and remap all static content to a different site. This also makes sense because if you move your static content to a CDN, your site will load a lot faster.

The second is using Lighttpd 1.5.x. The last time I tested it, it was still a bit unstable with my setup, so its in the back-burner for now.

The third is to use Varnish. Its wicked fast, extremely configurable, and it does caching. But it takes a bit to get into, so its a bigger time investment than the other two.

Right now, I think that Varnish would be the best bet of them all, but I don't have the time right now to get up to speed with it. So I'm going to use the first option, using Lighttpd 1.4.x to power my CDN server, just because I need modsecdownload.

Easy staging server setup

Before a new release of a web site, I like to give it a whirl using the production environment, including servers and database. To make the experience of the staging server as close as possible to the production server, I want to use the same hardware and server name for both.

At first I used different IPs, aliased to the same server, one for production and another for staging, and I used my local /etc/hosts file to switch between the two environments. This works but its a pain, because I have to restart the browser to pick up the new address, and also, I have to do this on every computer that wants to check the new version.

The new setup is much better.

It uses two application servers, production and staging, running on the same server, on different ports, and then use the front-end Lighttpd reverse proxy to select between the two instances based on a cookie.

The relevant part of the Lighttpd configuration is this:

# Staging server
$HTTP["cookie"] =~ "(; )?app_version=staging:" {
  proxy.server  = (
    "" => (
      ( "host" => "127.0.0.1", "port" => 7005 )
    )
  )
}

# Production server
$HTTP["cookie"] !~ "(; )?app_version=staging:" {
  proxy.server  = (
    "" => (
      ( "host" => "127.0.0.1", "port" => 7000 )
    )
  )
}

This setup uses a cookie named app_version. If app_version is set to staging, Lighttpd will use the application server at port 7005. If not, it will fallback to port 7000. This works even if we don't have any cookie.

Currently, only internal users use the staging server, so I have an option in the back-office site to select the version they wants to use.

You should notice that I don't do any security checks on the cookie, so anybody could set the cookie and use the staging version. To prevent this, my app_version cookie includes a SHA1 checksum of the content using a secret string (so its really app_version=staging:40-char-hex-sha1-checksum). Yet, I don't check the SHA1 in the front-end proxy, that would be wasteful of CPU for the vast majority of users using the production server. Instead, I have optional code to validate the checksum in the main application server. That code is only enabled on the staging version and displays an error message.

The final problem I needed to solve is identification of the current server. People got confused, they didn't had a clear indication if they where using the staging or production server. To solve that, I just enable a small HTML '<div />' and appropriate CSS in the staging server, and this will float a nice "You are using the staging server. Switch to Production"-message. The "Switch to Production" part is a link to the back-office tool that removes the cookie.

With these things in place, I can push release candidates to staging without worrying to much about it, have the other people at the office try it out, and then if all is ok, run a script on the main server and deploy the staging version to production.

Update: this also helps for rolling updates. For example, you could use this setup to keep two production versions available, and use the login process to move users from the old version to the new one.

December 21, 2007

Compiling git on Mac OS X

I had to install git on my temporary G4, and I didn't kept proper notes before, so for future reference, here is my way of compiling git.

This was tested on 10.4.11. I haven't upgraded to 10.5.x yet, maybe when 10.5.2 comes out.

I don't use MacPorts or fink, I don't like them. Its a personal thing, and if you do like them, you might as well use them instead.

First, let me tell you that I keep several git versions around. I install them in /usr/local/git-VERSION and switch a symbolic link at /usr/local/git to point to the version I want. Also, when I compile directly from a git checkout, I use the SHA1 of the HEAD as my VERSION. My PATH includes /usr/local/git/bin to find the latest version.

Second, I haven't bothered yet to compile asciidoc, so I don't compile/install the documentation.

Compiling the released version

Download the latest version of git from the git home-page.

mkdir ~/src
cd ~/src
GIT_VERSION=1.5.3.7
curl -O http://kernel.org/pub/software/scm/git/git-${GIT_VERSION}.tar.gz
tar zxf git-${GIT_VERSION}.tar.gz
cd git-${GIT_VERSION}
./configure --prefix=/usr/local/git-${GIT_VERSION}
make all

If you get a compile error, something like:

    SUBDIR git-gui
    MSGFMT    po/de.msg make[1]: *** [po/de.msg] Error 127
make: *** [all] Error 2

It means that you would need to compile GNU gettext to get msgfmt. Just do:

export NO_MSGFMT=1
make all

and it will use a alternative TCL script instead. According to pfig, 10.5.x does not need this export at all.

To install:

sudo make install

Now, create a symlink:

sudo ln -s /usr/local/git-${GIT_VERSION} /usr/local/git

And add /usr/local/git/bin to your PATH:

echo 'export PATH=$PATH:/usr/local/git/bin' >> ~/.bashrc

Make sure you have the new PATH:

exec bash --login
git --version

Now go play with it.

Living on the edge

But I don't stop here. The stock version of git is fine, but I usually use the master branch. So after you gone through all that, you do it again.

I keep a clone of the git master branch and update it from time to time. To clone the first time, do:

mkdir ~/projects
cd ~/projects
git clone git://git.kernel.org/pub/scm/git/git.git

If your firewall doesn't allow outgoing TCP on port 9418, try:

git clone http://www.kernel.org/pub/scm/git/git.git

You only need to do this steps once. From then on, whenever you want to update to the latest git, do:

cd ~/projects/git
git-pull
make distclean
GIT_VERSION=`git-show | head -1 | cut -c8-`
# I should be able to do GIT_VERSION=`git-show --pretty=format:"%H"`
# but I could not get it to work...
autoconf
./configure --prefix=/usr/local/git-${GIT_VERSION}
export NO_MSGFMT=1
# See above why export NO_MSGFMT=1
# You can try without it and use it only if it fails...
make all
sudo make install
sudo rm -f /usr/local/git
sudo ln -s /usr/local/git-${GIT_VERSION} /usr/local/git
git --version

That's it. You should be using the latest git available now.

December 20, 2007

cpan tricks

Our beloved cpan command line has some tricks up his sleave. In case you haven't read the fine CPAN manual in a while, let me point out some features I'm using right now to install all the needed modules for my day-to-day operation.

cpan .

You have a local directory with a module already unpacked, or your own personal module. The usual way to install them is doing the dance:

perl Makefile.PL
(manually deal with missing dependencies here)
make
make test
make install

The second step, the missing dependencies part, is the not so good part of the whole experience. Module::Build authors and users would suggest that the first step is the really not so good part, but I digress.

A better way is to do:

cpan .

This will start a CPAN shell, and run the install process on the local directory, including fetching dependencies from you preferred CPAN site.

failed command

Inside the shell, after you installed a long list of modules, the failed command will list all the modules that failed the tests and did not install.

o conf init /REGEXP/

The command o conf init will go through the entire configuration process. In recent versions it has become a long process and one of the things I tweak from time to time, the URL list of CPAN mirrors, is the last item asked.

To speed up the process you can o conf init /urllist/ and only configuration options matching urllist will be asked.

Don't forget to o conf commit at the end.

The smart CPAN urllist

The urllist parameter lists the CPAN mirror sites that the shell will try to use to fetch the packages.

My favorite site is a local CPAN::Mini mirror, the best 750Mb used space I have on my systems. It allows me to install any module from CPAN even when I'm offline.

But this requires that I keep updating the mirror, and sometimes, I just forget. And I don't want to cron it because I don't want to run it while I'm connected via UMTS.

A trick here is to set urllist to several CPAN mirrors in your country and include your local CPAN::Mini mirror at the end with a file: URL.

CPAN is smart and will fetch the CPAN indexes from one of the sites (see o conf init /random/ for some fun) but will always try to download the actual package from the file: url first.

So you get up-to-date indexes for free.

SSHKeychain `Buggy password in keycahin workaround` error

If you upgrade SSHKeychain to 0.8.2, and start seeing

Buggy password in keycahin workaround

in Console.app (btw, it really is spelled keycahin in the source :) ), then go into Keychain Access and remove all SSHKeychain entries.

Problem solved.

December 08, 2007

Learning Git

I'll be talking a lot about Git in the coming weeks. I'll update this post with the best references I can find about understanding, using, and maintaining Git and Git repos.

Understanding Git

Using Git

SVN deserters

Example project workflow's

I collect mail messages or URIs where project leaders explain their usage of Git, branch organization and merge policy. Some will be overkill for most of the projects I work on, but you can always tweak and simplify one of them to suite your needs.

General Reference

Git ramblings

December 07, 2007

Google Charts

The new Google Charts API is very very nice.

I still prefer Open Flash Chart for back-office use, but for front-end sites, this might be much better, given that it works everywhere.

One thing to take notice: there is a daily limit of charts that you can ask directly from Google, so if you think you'll use more than those, use a reverse-proxy with cache (like Varnish or Squid) or even have your code call Google API and cache the image locally.

Now, let's see how much longer we'll have to wait until we get a Perl module to generate those URLs. :)

Update: Google Charts API launched the 6th of December, Perl module Google::Chart the same day. Yeah, Perl is dead.

December 04, 2007

Git vs Mercurial

I've been talking about Git in the last posts, and Bär asked if I tried Mercurial.

I did try Mercurial, but very superficially. I used it with a project shared with Rui, who uses Mercurial and he seems to be very happy with it.

Whatever you choose, I would stick with either Mercurial or Git. CVS is dead, and Subversion is not good enough. And right now, all non-personal projects I work on use Subversion, so I have some experience with it. Of the others distributed systems, only Darcs, still the best out there in terms of day-to-day usage, deserves a honorable mention.

My take on Mercurial vs Git is this:

  1. when I started with Git, Mercurial was better if you need to run on Windows. Right now, Git has binaries available on the home-page but I don't do Windows, so I don't know if it is good enough, or stable enough. On Linux, Git just cannot be beaten in terms of performance. Update: Git seems to be doing fine on Windows now.

  2. I'm not comfortable with the code quality of Mercurial. I followed the Mercurial commit list for a month or so at the time, and some of the bugs where pretty nasty. It seems that Mercurial is still fixing core bugs. On the other hand, Git fixes in the past weeks are mostly cosmetic, and reorganization of code into a libgit. Also, the organization of Git release management is very conservative, and that feels good.

  3. I liked the GUI tools that come with Git better than those of Mercurial. I'm usually not a big fan of GUI tools to interact with SCMs but I admit that I've been using git-gui for all my commits. The history browser (hg view or gitk) are virtually identical (probably one of the projects "stole" the code from the other :) ).

  4. git-svn rocks my world. It makes working with Subversion less painful. Mercurial doesn't seem to have nothing as good.

In the end, there are big projects using either of them, so its no big deal. There are also tools to migrate repositories from one to the other, so I don't think there is much risk on any of those two.

I freely admit that I could choose any of the two, but the above 2), the need to keep on working with Subversion projects, and the fact that Git is used to manage the linux kernel, X.org, Samba and OLPC, made me trust Git more than Mercurial.

December 03, 2007

hunk-based commit with git-gui

I've being using git as a front-end to Subversion using git-svn (more on this later).

A cool feature that you just get for free is hunk-based commit. For those who haven't using Darcs, hunk-based commit allows you to select which parts of the diff you want to include. It allows for a much cleaner commit history.

With git, I've started using git-gui to create my commits. It's really useful. Inside, you can select a modified file, and then right-click the hunk you want to stage. Then, when you are happy with the staging area, type in the message, and commit.

I usually leave git-gui open, in the background, and have a Quicksilver shortcut to bring it to the front.

This setup works very well for me, and it almost makes me forget how much I miss Darcs.

(more info about git and hunk-based commit in this thread)

November 12, 2007

git-darcs-record

A cool script, git-darcs-record, that adds darcs-style interactive record to git.

Recommended.

October 10, 2007

MySQL Community server with CentOS 5

In case I need this again. After installing the official RPMs from the MySQL site for Redhat Enterprise Linux 5 on a fully patched CentOS 5, the startup script does not work properly, failing to start the server:

[root@centos5 log]# /sbin/service mysql start
Starting MySQL Couldn't find MySQL manager or server        [FAILED]

One possible solution is this patch. To apply do:

cd /etc/rc.d/init.d
patch -p0 < PATH_TO/mysql.rc.patch

And be done with it.

Also useful for DBI-related work is this. Log in to MySQL server as root and do:

grant all privileges on test.*  to 'melo'@'localhost';

Adjust melo for your local username.

This will make the installation of DBI and DBD::mysql work out of the box with cpan, testing everything in the process.

August 18, 2007

Startup performance of DBIx::Class

In a project I was working on, I had some performance problems to startup a DBIx::Class schema with about 75 sources. It took about 19 seconds to startup.

After a quick thread in the mailing list, the startup time is now 2 seconds.

The two-part solution is this:

  1. move all your load_components() into a common class and use that class as the base for your sources;
  2. use the schema provided load_classes(), its very fast. If you need per-source tweaking, do it afterwards looping over Schema->sources().

Many thanks to the dbix-class mailing list, in particular mst and Hartmaier Alexander for the tips in the right direction.

August 17, 2007

Tip: use growlnotify for long running Terminal tasks

If you have Growl installed, and you use the Terminal.app a lot, then make sure you also install the growlnotify command line tool.

This will allow you to send Growl notifications from the command line. The most useful script I have, that I use constantly is this:

#!/bin/sh
#
# Runs script, and prints a notification with growl when it finishes
#

$*
growlnotify -m "Script '$*' completed" -s "Background script notification" &

My version is called n, just the single letter n.

This allows me to do:

n scp server:some_big_file .

and a Grown notification will appear when the process terminates.

Update: a couple of updates with great suggestions from Ranger Rick, Tim Bunce and Ruben Fonseca. The new version below should also work on Linux systems, using libnotify. It was updated to deal with arguments containing spaces and we keep the exit status of the command intact. Also we assume exit code 0 is success, all others mean some failure has occurred. Thanks to all.

The Linux version needs checking, I didn't have a Linux server with libnotify, and used Ruben suggestions blindly. Please leave a comment if it doesn't work, specially the detection code. You might need a -h in there.

The full script (download):

#!/bin/sh
#
#  Runs script, and prints a notification with growl when it finishes
#
# Written sometime in 2006, posted 2007/08
#
# With Tips from Ranger Rick, Tim Bunce and Ruben Fonseca
#

# Run the command, including arguments with spaces
"$@"
status=$?

# decide which status to use
if [ "$status" == "0" ] ; then
    result="completed"
else
    result="FAILED ($status)"
fi

# decide which notifier we have
env growlnotify -h > /dev/null 2> /dev/null
has_growl=$?
env notify-send -? > /dev/null 2> /dev/null
has_libnotify=$?

# notify the user, growl or libnotify
if [ "$has_growl" == "0" ] ; then
    growlnotify -m "Script '$@' $result" -s "Background script notification" &
elif [ "$has_libnotify" == "0" ] ; then
    notify-send "Script '$@' $result" "Background script notification" &
fi

# exit with the original status
exit $status

Update 2: typo in "has_libnotify" corrected. BTW, I'm using env to check availability of a command because which (my first choice) returns the same exit code if the program exists or not, at least in Mac OS X.

Update 3: Thanks to Ruben Fonseca, the above script (and the download version) now work correctly on Linux with libnotify.

July 09, 2007

Tip: Comment selection in Texmate and Skitch

If you use TextMate and Skitch, you might have found that the excelent ''Comment selection'' command from the Source bundle will no longer work with the ⌘/ shortcut.

The problem is that one of the hot-keys of Skitch is exactly that key combination and for something that I don't use that much (Capture Frame).

So go into Skitch Preferences and change it.

iPod Reset Utility

Between me and my brother, we had three iPod Shuffle that iTunes wouldn't see, nor the previous iPod reset utility. They where pretty much dead.

Apple released a new version of the iPod Reset Utility, and this latest version (1.0.2) was able to recover all of our iPod's.

So if your Shuffle is not recognized by your iTunes, check out this latest version. It worked for us.

July 04, 2007

iPhone

There. I did a iPhone post. I'm cool.

Not.

July 03, 2007

You’re a liability to civilization

Great read to start the day.

Don't you just love hyped-up releases?

April 27, 2007

Google power, duh!

This is cool and helpful: placing your pet transponder ID in a web page for Google to index it.

It is also somewhat obvious, so, duh...

Now we just need a micro-format and profit!

April 24, 2007

Raising your head, once a year

Once a year, I look around and re-evaluate my SCM decisions.

In the last year, I've been using darcs as my main SCM and I love it: simple and powerful. And given that I don't branch and merge a lot, I don't run into his issues.

Yet, for fairly big projects with lots of files, it starts getting a bit on the slow side. Also, I haven't find any graphical change-set browser yet. I can have one for SVN and git but not for darcs (not a web frontend, a graphical desktop application).

Also, the projects I work on are all using SVN, and I don't like to move from one to the other.

So my requirements now include the following:

  • offline operation;
  • real branching and merge (basically this requires change-set history);
  • darcs-style interactive commit;
  • supported on Mac OX X and Linux;
  • SVN gateway.

The last bullet point is new this year.

So I went looking for something that matches all of this and I found the set of git, cogito and git-svn. They seem to fit the bill perfectly.

Right now, I've compiled git from source. Its as simple as:

curl -O http://kernel.org/pub/software/scm/git/git-1.5.1.tar.gz
tar zxf git-1.5.1.tar.gz
cd git-1.5.1/
make configure && configure --prefix=/usr/local
make all
sudo make install

You should now have all the git goodness installed in /usr/local/bin. If you type git at a Terminal and you get an error like -bash: git: command not found, it's probably not in your path, so:

echo 'export PATH=$PATH:/usr/local/bin' >> ~/.bashrc

A freshly open Terminal should find it now.

After git, you should install cogito. The first is the core SCM, a meta-SCM, a set of tools to create a SCM. The second is a porcelain layer, a more user friendly user interface to git. You can download and install it with:

curl -O http://kernel.org/pub/software/scm/cogito/cogito-0.18.1.tar.gz
tar zxf cogito-0.18.1.tar.gz
cd cogito-0.18.1/
make
sudo make prefix=/usr/local install

That's it. The README of cogito is actually quite good to get you up to speed, and I would recommend reading it as the next step.

With this environment you already have everything you need to start working. To test it out, you can download the cogito git repository with:

cg-clone http://www.kernel.org/pub/scm/cogito/cogito.git ~/cogito

To see the history of change-sets graphically, you can:

cd ~/cogito
gitk --all

If the fonts and layout seems butt-ugly and unreadable, check out the alternative configuration file for gitk.

For now, I'm just testing this setup. I'm pretty happy with it. I will have to try git-svn and the cool-looking git-hunk-commit script (to have something like darcs interactive record) before I decide to switch, but it is looking very likely.

A good thing about modern SCM systems: you can switch from one to the other in a relatively simple way...

April 16, 2007

Tip: XML::LibXML and Debian

If even after you apt-get install libxml2 libxml2-dev you still can't install XML::LibXML, look for this error message:

CPAN.pm: Going to build P/PH/PHISH/XML-LibXML-Common-0.13.tar.gz

enable native perl UTF8
running xml2-config... ok
looking for -lxml2... no
looking for -llibxml2... no
libxml2 not found
Try setting LIBS and INC values on the command line
Or get libxml2 from 
  http://www.libxml.org/
If you install via RPMs, make sure you also install the -devel
RPMs, as this is where the headers (.h files) are.

Basically, the installation of XML::LibXML::Common failed.

To solve, try this:

apt-get install zlib1g zlib1g-dev

Aparently libxml2 depends on -lz but the .deb doesn't notice or something.

March 20, 2007

Macbook Pro shuts down

Houston, we have a problem. This is happening to me to. After the 10.4.9 upgrade, when I'm using battery, the Macbook just shuts down without any warning whatsoever.

Crap.

Update: Today I got this behavior with a TiBook G4 at work, very weird. This could point to a problem with the 10.4.9 upgrade.

Articles I'm following about this:

I think I have a swollen battery. I'll drive by the Apple tech support tomorrow.

Quicksilver Proxy objects

Recently, Merlin Mann did a show about Quicksilver Proxy Objects.

If you have problems getting that to work, follow the excellent mini-tutorial about enabling Quicksilver proxies and application menus by Robert Daeley.

In fact, you should read the short tutorial before watching the show.

Quicksilver Proxy objects

Recently, Merlin Mann did a show about Quicksilver Proxy Objects.

If you have problems getting that to work, follow the excellent mini-tutorial about enabling Quicksilver proxies and application menus by Robert Daeley.

In fact, you should read the short tutorial before watching the show.

March 16, 2007

Local networking

In my new work setup, one of the things I did was to upgrade the switch between the two laptops to a D-Link Gigabit switch (less than €30 at a local discount shop if I remember correctly).

I did some benchmarks to compare the speed of copying a 2.4 Gbyte Parallels XP image in several scenarios:

On the old TiBook, local disk to local disk:

$ time cp winxp.hdd winxp.hdd2
real    3m24.703s
user    0m0.032s
sys     0m29.427s

On the Macbook (IM going on, iTunes, a bit of Safari), local disk to local disk:

$ time cp winxp.hdd winxp.hdd2
real    4m15.725s
user    0m0.019s
sys     0m9.391s

From the Macbook to the TiBook over Gigabit Ethernet:

$ time cp winxp.hdd /Volumes/melo/Drop\ Box/
real    5m29.288s
user    0m0.018s
sys     0m23.898s

Nice.

So I moved a lot of FireWire external disks and DVD burners to the TiBook, my "server" and when I want something I just copy it over the network. Less cables to plug in when I get to the office.

Probably the best €30 I spent in the last weeks.

March 09, 2007

Installing Ruby XMPP Simple

I'm no Ruby expert but a friend of mine was asking me how to write a simple XMPP bot using Ruby, and I had seen a couple of recent posts announcing a simple ruby library.

So after installing RubyGems (I had a total virgin Mac OS X stock install of Ruby), I did the usual:

sudo gem install xmpp4r-simple

That did not work well. One of the dependencies, rcov, does not install in my system. The error message is something like this:

gcc -fno-common -g -Os -pipe -fno-common -pipe  -fno-common -pipe -fno-common  -I. -I/usr/lib/ruby/1.8/universal-darwin8.0 -I/usr/lib/ruby/1.8/universal-darwin8.0 -I.   -c callsite.c
callsite.c:121: error: parse error before 'event'
callsite.c: In function 'coverage_event_callsite_hook':
callsite.c:131: error: 'klass' undeclared (first use in this function)
callsite.c:131: error: (Each undeclared identifier is reported only once
callsite.c:131: error: for each function it appears in.)
callsite.c:136: error: 'mid' undeclared (first use in this function)
callsite.c:141: error: 'node' undeclared (first use in this function)
callsite.c: In function 'cov_install_callsite_hook':
callsite.c:162: error: 'RUBY_EVENT_CALL' undeclared (first use in this function)
make: *** [callsite.o] Error 1

To solve this, I did the following: first I manually downloaded the latest version of rcov from the rcov site. Doing the classic ruby setup.rb dies with the same error, but there is an alternative:

sudo ruby setup.rb all --without-ext

This works. Basically it skips installation of a C-based extension to the rcov package that makes the package a lot faster. Given that I don't intend to run coverage tests in ruby, I don't mind skipping this.

At this time, gem install xmpp4r-simple still tries to install the dependency of rcov although we already have rcov installed. So we skip the dependencies:

sudo gem install --ignore-dependencies xmpp4r-simple

This works, and now we can play.

February 22, 2007

Lighttpd, file uploads and Safari

On one of the web applications I have, the setup is more or less like this:

In this case, lighttpd is used as a reverse HTTP proxy to Apache, using mod_proxy.

I was trying to get file uploads to work on this setup. Uploading directly to Apache works flawlessly, but via mod_proxy, Safari would hang at the end, never terminating the request.

The solution is simple: for now, just disable keep-alives. Add this to your Lighttpd virtual host configuration:

$HTTP["useragent"] =~ "^(.*MSIE.*)|(.*AppleWebKit.*)$" {
    server.max-keep-alive-requests = 0
}

Given that this is a back-office, this is acceptable, but for other situations it might not be. I must try to target this just to file uploads, even if that means moving them to a special virtual host.

Module of the day: File::Inplace

The task was simple enough: edit a already existing file, to change something, keeping the name intact, and if possible doing a backup of the original version.

We could start dealing with all sorts of errors in open, rename and friends, dealing with temporary files and all that stuff.

Or we could just jump to CPAN:

cpan File::Inplace
perldoc File::Inplace

In my case, I wanted to remove a line from a file. A regexp matching the line in question was something like this:

qr/src="trivantis-titlemgr.js"/

So the code becomes:

  my $editor = File::Inplace->new(file => catfile($path, 'titlemgr.html'), suffix => '-'.time().'.bak' );
  while (my ($line) = $editor->next_line) {
    $editor->replace_line(undef) if $line =~ qr/src="trivantis-titlemgr.js"/;
  }
  $editor->commit;

Nice and sweet.

'update:' to clarify some point raised in the comments, yes I know about perl -i.bak, but I needed this inside a Catalyst web application, and thus this module.

February 05, 2007

Tip: given a coderef, show me where the code is

In Perl, sometimes you have a coderef, normally a callback, and you need to know where is that code located in your source.

This will do the trick:

sub _dude_wheres_my_coderef {
    use Devel::Peek;
    $code = \&$code; # guarantee a hard reference
    my $gv = Devel::Peek::CvGV ($code) or return;
    return *$gv{PACKAGE} . '::' . *$gv{NAME};
}

Taken from dump_vars.pl. BTW, $name also includes filename and line number.

January 31, 2007

Good stuff coming to a Kernel near you.

Yesterday, Zack Brown (of kernel-traffic fame) posted a patch to run any Linux syscall asynchronously. One of the problems with people doing high-performance event-driven network programming like Lighttpd, Perlbal, or DJabberd was that some syscalls block, and there wasn't a good clean way to do it. One of them mentioned regularly is stat. For kicks, Lighttpd has a patch to move stat to a fcgi app...

Good news is that Linus liked it.

Happy, happy, joy, joy!

(via Brad)

January 25, 2007

Top 3 tools for MySQL

At the moment, my top 3 tools to tune MySQL are:

  • mysqlreport: a friendly report about the status of your MySQL server (or SHOW STATUS for real persons);
  • innotop: a top-like program to see what's going on with your MySQL server (works with multiple servers);
  • mysqlsla: analyses MySQL statement logs (including slow queries).

Other tools are also useful sometimes:

  • Duplicate Index Checker: checks your indexes to see if you have multiple indexes covering the same information, in the same order;
  • MySQL Query Profiler: reports on what work MySQL will have to do for a given batch of queries.

I discover most of this tools reading the Xaprb blog, recommended.

January 22, 2007

JQuery Love

I'm not a Javascript guru, I just dabble a bit, and most of the time I would only reuse stuff from other persons. Until I spent a weekend with JQuery.

I needed a small JS lib to do some stuff in a application I'm working on. Basic stuff, like edit in place, building HTML live and adding more options to forms.

After looking at the tutorials and "how it works"-kind of pages, I started to try it out.

One of the things I wanted was a dismiss specific paragraphs that we use for success or error messages.

I found an example on the JQuery blog that looked promissing. It would hide the message after 5 seconds. The code is simple:

$(document).ready(function(){
    /* add class timed_fadeout to fadeOut after 5 seconds */
    $('p.timed_fade_out').animate({ opacity: 1.0}, 5000, 'linear', function () {
      $(this).fadeOut("slow");
    });
});

You just need to add a class named timed_fade_out to any <p>.

But this was perfect. I didn't want it to just fade way automagically. I wanted the user to click on a dismiss link. So I wrote this:

$(document).ready(function(){
    /* Add dismiss class to append link that closes box */
    $(' <a class="dismiss" href="#">(dismiss)</a>').appendTo('p.dismiss');
    $('a.dismiss').click( function () {
      $(this).parent('.dismiss').fadeOut('slow');
      return false;
    });
});
/* CSS: a.dismiss { color:#999; font-size: 80%; } */

And it worked! First try!

The thing about JQuery, is that it fits my brain, and I don't have to think the way the library works. Very cool.

January 10, 2007

Tip: MySQL is not that helpful sometimes

Stupid problem today.

In a listing I had, I had a percent column. The problem was that the database definition for that column was wrong, it used DECIMAL(2,4). Yes, 100% will not fit in there. Also: this syntax was valid in 3.x but with 5.x you need DECIMAL(6,4). I'm using now DECIMAL(7,4) to fit the edge case.

My peeve with MySQL is that if I give him 100 to put in that column he will not complain nor store NULL. He will help you and store 99.9999.

But the problem was the lack of a test for the edge case. Added, committed, happy happy joy joy.

January 02, 2007

Tip: deleting all objects in a many-to-many relation with DBIx::Class

This might be obvious to many of you, but it wasn't for me, so here it is.

If you have a many-to-many relation in DBIx::Class, you can remove all the relations with:

$self->set_RELATION_NAME([]);

The set_RELATION_NAME method is created automatically for each many-to-many relation you setup. It can receive a list of objects that will become related to $self.

The obvious

$self->set_RELATION_NAME();

will croak on you.

GraphViz under Mac OS X

In case you are trying to install the Perl module GraphViz under Mac OS X 10.4.x, you might need this script.

First, install a decent version of GraphViz. I use this snapshots of Graphviz by Ryan Schmidt. Be careful not to use this version on MacIntel's, the package is PPC-only.

After this is done, add /usr/local/graphviz-2.12/bin to your PATH.

Second, while installing the GraphViz module via CPAN, one of the dependencies is IPC::Run, which might fail on your system. It's a know issue with FreeBSD that seems to also be present on Mac OS X. It hangs while testing t/pty.t.

To skip those tests, do this:

  1. inside cpan shell, type look IPC::Run;
  2. edit t/pty.t, go to line 97 (looks something like this: my $platform_skip = $^O =~ /(?:aix|freebsd|openbsd)/? ...;
  3. add |darwin after opendbsd, save and exit;
  4. type the usual perl Makefile.PL && make && make test;
  5. if all goes well, type sudo make install or the version that works for you.

After that, you should be able to install GraphViz without any problems.

December 31, 2006

How to convert a UNIX timestamp to a date in Oracle

Might come handy - How to convert a UNIX timestamp to a date in Oracle:

TO_DATE('19700101000000','YYYYMMDDHH24MISS')
+ NUMTODSINTERVAL(unix_epoch, 'SECOND')

Coolness.

MySQL and UTF8 support

I've talked about this before. Basically, if you use UTF8 in your database, the scalars returned by DBD::mysql wouldn't have the utf8 flag turned on for text fields.

Until now.

With the latest release (DBD-mysql-4.0000), things have changed. A patch from Dominic Mitchell was applied (in DBD-mysql-3.0008_1 according to the changelog) that does the right thing regarding UTF8 text fields.

Check the documentation (search for mysql_enable_utf8) to see how it works. You can also look at the utf8.t test script for examples of usage.

I'll post back my findings after I play with it.

November 10, 2006

Tip: Subversion and accentuated characters in commit messages

One of the good things about Subversion is that all his internals (paths, commit messages, etc) are always UTF-8 encoded. I think that was one of the brightest decisions they made.

Yet, for quite some time I was getting errors when I tried to use commit messages with accentuated characters:

subversion/libsvnsubr/utf.c:466: (aprerr=22)

In my case, the problem is that I didn't have a proper LANG environment variable. If run svn ci like this:

LANG=en_US.UTF-8 svn ci

the problem goes away.

Now, having a LANG environment is not something I want just now. I still use software that does not work well with LANG. So my solution was to create a small shell script in my utilities directory like this:

#!/bin/sh
#
# Make SVN commits work with accentuated characters
#

LANG=en_US.UTF-8
export LANG

exec /usr/local/bin/svn $@

Nirvana!

May 22, 2006

Buying season I

In the following weeks me and my wife will be traveling a bit and mostly outside our usual places of work.

That means no net.

Or at least it usually did mean that. I bought her a Vodafone Mobile Connect Card, the 3G, not the 3.5G "broad band". I'm a TMN subscriber and she is a Optimus subscriber, but I went to Vodafone because they have better coverage.

The card was pretty much plug & pray in Windows, and we where online fetching mail in no time. Very nice.

They I turned to my Mac. The card brand is Option, and they have Mac OS X drivers, so I downloaded them, installed them and bingo, I was online also with my Mac.

I then tried to log-on to the VPN at work, but it seems to have vanished. I realized that the installation program for the card completely obliterates the VPN setup you had previously. Not good.

All in all, it was a good experience and we now have a way to fetch our mail in the next few weeks. Even with the VPN-fiasco, I would recommend this card to anyone in Portugal in search of a 3G connection. Personally I would prefer a phone-based approach but I didn't want to wast money on a 3G phone that I don't need.

The problem will arrive later, when the Macbook Pro arrives and the PCMCIA slot disappears...

Update: yuck, yuck, yuck... The Vodafone "Dashboard" erases my entire VPN setup every time I activate the card. And that is a required step in the process of connecting. Sucks big time, Vodafone.

May 06, 2006

Tip: reset the TOP format in Perl

Here is something that took me a while to get.

If you are writing scripts in Perl to do some reports and statistic analysis, and you are not using format, stop right now, do a perldoc perlform.

Then, in case you need to switch reports in the middle of the script, for example if your script dumps the data collected in three different layouts, use this code between each report:

$~ = 'REPORT_NAME';
$^= 'REPORT_NAME_TOP';
$= = 100;  # Number of lines on each page
$- = 0;    # Force the print of the new header

You can use English and use the long versions of these variables. See perldoc perlvar for those.

The $- is the one that had me pulling my hair. If you switch to a new layout and start writing data, the system doesn't know that this is a new layout, and will not print a header until he needs to switch the page. Setting this variable to 0 forces a page feed, and forces the new header to be printed.

Technorati Tags: ,

April 28, 2006

Textmate goodness

I'm still impressed with TextMate and although I still run into Vim from time to time, most of my work is now done inside the new sweetheart.

Of course, we are still in early stages of our relationship, and the old and trusted Vim can still regain is rightful place, specially with the newest version 7 (in late beta now).

Yet sometimes I see another set of possibilities with Textmate that blew me away. The latest one is this new concept, by Duane Johnson, of multiple arbitrary simultaneous carets (MASC). Go and see the screen-cast.

Great stuff.

Technorati Tags:

April 27, 2006

MySQL updates

The MySQL Users Conference is almost over in sunny California and the output for those of us we didn't make the trip has been overwhelming with over 100 blog posts in a few days.

MySQL is booming. There are three new engines: Solid, an old friend that I used for testing on my Telenet days and had to drop due to management distrust (lessons we learn when we are young: don't listen to management when choosing technical solutions); PBXT, a brand new engine, fully ACID-compliant with some novel ideas about database layout on disks (white paper worth a read); and Falcon, an MySQL.com engine, developed in house by Jim Starkey, a ACID/MVCC guru (two articles about his presentation). And as a final item in the storage engine department, Oracle renewed the InnoDB aggrement with MySQL. All in all great news.

I've also been collecting all the talks that are online, and reading all that I can find. So far the one I liked the most is from Ask Bjørn Hansen (online at his site). But there are others:

Some Flickr information, partly MySQL-related, was also covered in a recent O'Reilly Radar by Tim. This last one is specially useful for people dealing with tags (Hint, Hint, you know who you are :) ).

Unrelated to MySQL UC, J. Scott Johnson also made a presentation on php|tek about MySQL at Ookles.com.

There are common themes:

  • data partitioning;
  • small (mostly two servers) master-master clusters, many of them;
  • InnoDB, some MyISAM;
  • memcached, perlbal (Brad should be proud);

Other presentations I would love to see online, or some notes:

Great times for MySQL. Great times indeed.

Update: got a couple more presentations related to MySQL. I'm just linking them here for future reference:

April 22, 2006

MySQL replication

An excellent article about MySQL replication is online at OnLamp.com.

Definitively check it out.

Technorati Tags:

March 17, 2006

Lighttpd + MT 3.2 + PHP + Fast CGI

After battling with lighttpd and friends for a couple of hours, I finally have a very cool setup for this site.

I use the Moveable Type software to publish this articles. I modified the templates to publish each zone of the site in a separate HTML file. Then I use PHP to include them all in the final result. This allows me to generate the right hand-side column just once, as a index template, and include it on all the other articles.

So even if you go an see an old article, you'll still see my current right hand-side column.

In the past few weeks, though, Apache has been burning a lot of memory with all the traffic we've been getting on the several sites that are housed here. So I wanted to test another server, single threaded, and try to use FastCGI to reduce memory usage.

So after a PHP recompilation (use this page as an how-to) and some tweaking, I have a setup that works. I need to tune the other sites before moving to lighttpd but for now it looks and feels great.

My final configuration file looks something like this:

$HTTP["host"] == "www.simplicidade.org" {

  server.indexfiles    = ( "index.html" )

  $HTTP["url"] =~ "^/notes/?(.+\.html)?$" {
  fastcgi.debug        = 0
  fastcgi.server       = (
      "/" => ( (
          "socket" => "/path/to/site/simplicidade.org/fcgi.socket",
          "bin-path" => "/path/to/bin/php",
          "min-procs" => 1,
          "max-procs" => 1,
          "bin-environment" => ( 
              "PHP_FCGI_CHILDREN" => "4",
              "PHP_FCGI_MAX_REQUESTS" => "1000"
          ),
          "bin-copy-environment" => (
              "PATH", "SHELL", "USER"
          ),
          "broken-scriptfilename" => "enable"
      ) )
    )
  }
}

The $HTTP["url"] match makes sure that only files with the HTML extension will be processed by FastCGI, leaving all the images and CSS to lighttpd static file engine, which is very fast.

I also don't use the min-procs/max-procs settings for now. They seem broken to me because lighttd will always start max-procs no matter what I do. I think I'm missing something from the documentation.

Anyway, I'll move the other sites to similar setups and then I'll switch DNS to the new IP address with all this sites on lighttpd.

Technorati Tags: , , , ,

February 13, 2006

Tip: show process sorted by RSS

Several times I needed a way to quickly see which process are using the most memory on a Linux system. After reading a bit of the man page and with a tip from the Apache performance article I was reading, I settled for this:

ps ax -yl --sort:rss  (BSD syntax)
ps -eyl --sort:rss     (standard syntax)

There you go.

Update: corrected the command, thanks to Daniel Fonseca. I messed up BSD syntax (which I prefer) with standard syntax. I included both versions.

Thanks Daniel.

Update: To clarify, this is in a Linux system. On my Mac, the better I was able to do with standard ps command was:

ps alwxm

Not as good, specially because the results are sorted in descending order.

Technorati Tags: , ,

February 08, 2006

Tip: generating a good password with Mac OS X

Today I needed to generate a good password for a special site. I remember I saw a tool to do that and I could not remember where.

After some searching, here is the deal:

Password Assistant
  • open System Preferences and select the Accounts preference pane;
  • select your account from the list, and click Change Password button. Don't worry, you don't need to really change your password;
  • Click the small key icon next to the New Password text field. A new panel, Password Assistant, will open. You can now close the change password window by clicking Cancel;
  • In the Password Assistant, chose the preferred type of password you want, and check the sugestions. You can also type your own desired password and see how strong it is by looking at the colored bar. In this case, bigger and green is better.

That's it.

Tip: does my CPU support EM64T extensions?

This was a question I was wondering for quite some time. All the things I read about this pointed to a positive answer: I have a Xeon CPU and all Xeon CPUs since second quarter 2004 support EM64T.

But I wanted something more reliable, like a cat /proc/cpuinfo flag. And I finally found a post that gave my some hints about detecting EMT64 from the cpuinfo flags. It describes the lm flag as "long mode", having the 64bit extensions, but it talks of those in AMD context.

Further googling turned up this Debian mailing list post and a MySQL commit message, and they seem to confirm this: even on Intel CPUs, lm CPU flag means 64bit extensions.

Now the question for all: if I have a dual Xeon processor, is it worth it running CentOS with EM64T extensions or should I just run normal i386 version?

February 03, 2006

Removing all mail from your gmail account

When gmail first appeared I subscribed as soon as I could get my hands on an invitation. I forward a lot of mail to try it out.

Some months later, I saw that I could use gmail as an archiver for certain things, those mails you don't know if you really want to keep, but the problem was that now my gmail account was full of garbage.

I wanted to purge all mail from my account to start from scratch but there seems to be no way to do it easily. Clicking on 80 pages (8k message, 100 messages per page), selecting all and delete, was not an option.

So I wrote a script to login to gmail POP3 service, delete all messages, exit, and try again, until I got 0 messages on my inbox.

I now have about 10k messages in the trash, and those will be removed in 30 days.

Done!

In case you need to do the same, here is my x-gmail-expunge script. It requires Perl and the Mail::POP3Client from CPAN. Run it without parameters for usage, but it's really simple: x-gmail-expunge.pl LOGIN PASSWORD.

Technorati Tags: ,

January 14, 2006

Old habits die hard

After using CVS for the last 10 years, the cvs sequence is hardwired between my brain and my hands when I think about source control.

So for the time being, I'll use this:

melo@roadrunner(~)$ cat ~/bin/cvs
#!/bin/sh
#
# cvs wrapper: old habits die hard, and don't remember which VCS I'm using most of the time
#

if [ -d CVS ] ; then
  /usr/bin/cvs $@
elif [ -d .svn ] ; then
  svn $@
else
  darcs $@
fi

That covers the three source control systems I use on a regular basis.

Technorati Tags: , ,

WMV for Quicktime

If you need to see WMV files on Mac OS X, until recently you had to use the Windows Media for Mac.

If you knew that and used it, you know that it wasn't that good.

Recently, there where some mentions about Flip4Mac on Mac OS X Hints, and this week, Microsoft said that it would stop developing the Media Player, and made and agreement with Flip4Mac to distribute their WMV Quicktime plugin and standalone player software for free.

I've downloaded a version last week, and installed. In the next few days, I experience crashes in both Mail.app and Safari, something that I hadn't before. I decided to reinstall and yesterday I downloaded again. I don't remember which version I downloaded the first time, but yesterday I picked up 2.0.1 and since then the crashes have stopped.

I don't know if they released a minor upgrade due to some problems, but if you are having random crashes since installing Flip4mac, try uninstalling (there is an uninstaller in the /Applications/Flip4Mac folder), download the 2.0.1 release and install again.

Technorati Tags: , , ,

January 13, 2006

Saft

Saft is a plugin for Safari. Some days it seems the other way around. Safari is an Saft extension because Saft adds more useful features to Safari than the other way around.

I was wondering what feature I find the most useful, the most "I can't live without this". Its not the way it allows you to reorder tabs, and not even the type-ahed search (similar to the Firefox incremental search). It could be the session restoring features, that kick in to restore all your open tabs in case of a crash (giving you an opportunity to pick and choose which tabs to restore).

For me, the $12 feature (yes, it just costs $12...) its Cmd-Z. In those situations where you get Cmd-W-happy, and close that tab holding some important page you had visited, Cmd-Z will restore the tab just closed. And more: keep hitting Cmd-Z and all your closed tabs will come back to life.

That alone, makes it worth the registration fee.

Technorati Tags: , ,

January 07, 2006

Textmate goodness

Well, after three weeks inside TextMate, I must say I'm hooked.

I knew it existed and it was cool and stuff, but until you read the TextMate manual, you don't know half of the power it packs. If you want to see if it is for you, I would recommend that you try and read through the fine manual.

Around our corner of the world, I count five converts already, and given that we had abused the local mac users group mailing list with TextMate stuff in the last couple of days, I went ahead and created a TextMate Portuguese User mailing list.

To subscribe to the mailing list, send an empty message to tm-subscribe@lists.simplicidade.org. You'll get a confirmation back that you must reply to.

Version 1.5 was announced yesterday, and although not mentioned anywhere I could find, it comes with an Input Manager that allows you to edit any text field (therefore including Safari text areas) from any Cocoa application in TextMate. I've been using this to edit my posts from inside ecto.

Some applications, even ecto, already had a feature like this, but this strategy makes it available on any application. Also, the update is instantaneous on each Save, and that gives me a sensation that my work is always safe in two different apps (Tip: set TextMate to Save your work when it looses focus, and it becomes even more magical).

To install it, use the command Install "Edit in TextMate..." available at Automation > Run Command > TextMate, and follow the instructions.

Technorati Tags: , , ,

December 31, 2005

Windows zero day exploit

The vast majority of software has bugs. The ones that haven't are either two small to do something useful, or written with rigid rules, during long years. We can argue the fine points of this, but it's rare the software that doesn't have them.

The issue with bugs is not that they exist, it's how long you have to wait until they get fixed.

It seems that we now have active exploits roaming around for the latest Windows bug. Even XP2 is not safe, and basically you just have to browse to a site containing a special image (not Britney, a special crafted one).

This worries me because a lot of friends and family use Windows. My wife has a small business and uses Windows. So we are in for a ride in the next couple... what? Days? Weeks? Months? Microsoft hasn't acknowledge the bug yet...

Let's hope someone is able to write a Firefox extension to block these pesky images, and make sure your anti-virus are updated, people.

I wont rant about how Mac's or UNIX in general are more safe from this kind of thing. It's wasted time, really. Security has become some sort of battlefield between OSs (my is bigger than yours kind-of-thing), and most OSs can be made safe, if you are willing to loose flexibility (a anti-windows friend of mine would say that if you are willing to loose network connectivity, Windows is pretty safe...). I myself know that I set up my Mac in certain ways because it's more flexible that way, but I loose security. It's a trade off.

Update: there is fix for the WMF exploit. I find it amusing that it doesn't come from Microsoft, and predates the (so far missing) acknowledgment from them. There isn't a single mention of this in the Microsoft Security page.

Update 2: Ahh, found the Microsoft Security Advisory 912840. It's not in the Security home page, so this is probably me that don't understand where we should look for these things. The wording of the title is amusing: Vulnerability in Graphics Rendering Engine Could Allow Remote Code Execution. Emphasis mine.

December 29, 2005

The quest for fastcgi MT setup

After successfully placing Trac under fastcgi with Lighty, I'm now targeting Moveable Type.

It's not as important as Trac was, I'm pretty happy doing it via CGI, I'm very low traffic. One of the caveats with this approach is that some plugins might not work correctly, due to the persistent nature of fastcgi processes. But, hey, this is the way Yahoo! is doing it, so if that is really a problem, I think they will be corrected shortly.

Anyway, I've collected some links to how-to's about this:

Technorati Tags: , ,

Tip: use TextMate and Ecto

Quick tip for those who are using ecto and Textmate: you can edit your drafts using Textmate directly from ecto. In case you use Markdown to write your posts, this is great. The Markdown bundle in Textmate is very good.

Just choose Edit > Edit with > TextMate and you're done. The text updates in ecto after you save in Textmate.

For more information, see this article on the ecto blog.

Technorati Tags: , ,

Trac and Lighty

I'm moving this and other simplicidade.org sites to Lighty.

My main motivation was memory usage and the fact that I'm starting to use Trac a lot, and using that as a CGI is painful. I wanted to use fastcgi and Lighty is high-rated in that area.

After reading some articles about Trac and fastcgi, I settled on this config:

$HTTP["host"] == "projects.simplicidade.org" {
  server.document-root = "/servers/sites/projects.simplicidade.org/docs"
  accesslog.filename   = "/servers/logs/lighty/sites/projects.simplicidade.org_access_log"

  server.indexfiles    = ( "index.html" )

#  fastcgi.debug        = 1
  fastcgi.server       = ( "/project" =>
                           ( "trac" =>
                             ( "socket" => "/servers/workspace/lighty/sites/projects_simplicidade_org-trac-fcgi.sock",
                               "bin-path" => "/usr/share/trac/cgi-bin/trac.fcgi",
                               "check-local" => "disable",
                               "min-procs" => 1,
                               "max-procs" => 3,
                               "max-load-per-proc" => 1,
                               "idle-timeout" => 30,
                               "bin-environment" => (
                                   "TRAC_ENV_PARENT_DIR" => "/servers/workspace/trac/instances/simplicidade"
                               )
                             )
                           )
                         )
  alias.url            = ( "/trac/" => "/usr/share/trac/htdocs/" )

  auth.backend         = "htpasswd"
  auth.backend.htpasswd.userfile = "/servers/workspace/trac/passwd/simplicidade/trac.htpasswd"

  $HTTP["url"] =~ "^/project/[^/]+/login" {
    auth.require       = ( "/" =>
                           ( "method" => "basic",
                             "realm" => "projects at simplicidade.org",
                             "require" => "valid-user"
                           )
                         )
  }
}

Works very well. The trick to have several Trac instances with a single fastcgi server is the TRAC_ENV_PARENT_DIR environment. You should point it to the parent directory of all your Trac instances.

Oh, and the site projects.simplicidade.org is not up and running yet. I'll announce it soon though.

Technorati Tags: , ,

TextMate: cool stuff so far

In my quest to implement my 2006 Resolutions, I decided to start early and I've switched to TextMate already.

I will not dump all the links to the documentation on you. I will only point to some "Wow, cool"-type of things I found in the last couple hours browsing stuff.

Some of the links are from the Macromates blog, a resource I would recommend you to, if you are also switching to Textmate. The author posts some lengthy articles, and I was able to pick up general Mac OS X tidbits I didn't knew about.

So far, so good. Snippets and Macros are my new best friends.

Technorati Tags:

utf-8 and DBD::mysql

After an afternoon trying to understand why some of my output from a utf8 table in MySQL was coming out garbled, I finally realize that:

  • even if your tables and database are all created with utf8 charset;
  • even if you set your connection charset to utf8 with SET NAMES 'utf8';

your scalar results in perl will not have the utf8 flag set, so any print, concatenation or XML generation further on will result in a mess, when finally printed out to a XMPP stream, for example.

So, on all your code, after you retrieve data from MySQL, you must set the utf8 flag on that scalar.

For now I'm using this code. Probably not the best one, but it suffices for now.

if (! utf8::is_utf8($message)) {
  utf8::decode($message);
}

There is some discussion about this online. It seems that the DBD::mysql people are waiting for a general solution for the problem to appear in a future version of DBI. There is also a patch floating around that sets the flag on utf8 content.

If you use Class::DBI, you can also look at Class::DBI::utf8 that does the right thing.

Regarding support for this in DBI itself, there is a thread by Tim Bunce that talks about utf8 support in DBI in a future version, in particular bullet 4 of the initial post. But the next bullet points the responsibility of the utf8 flag to the drivers.

This quote in particular should self explanatory about Tim's reasoning:

Some features, like charsets, vary greatly in how they're handled by database APIs. For these kind of features the DBI usually lags the drivers. Once a few drivers have implemented their own driver-specific interfaces, and had them proven as practical by users, then I can work with driver authors to see how best to extend the DBI API in a way that'll work well for those drivers and others.

And a more specific one regarding DBD::mysql:

Basically it should be the job of the drivers to set the uft8 flag on data being retrieved if it is utf8. I believe that the new mysql v4.1 protocol does provide information about the characterset of each colum. DBD::mysql can use that.

I would like to see that patch into the DBD::mysql mainline. It seems that Tim Bunce is passing on the responsibility of the utf8 flag to the driver author. It makes some sense. If the DBI layer was responsible to set the flag, it would need to obtain charset information from the DBD driver anyway. In that case, if the driver already knows which charset it is using, why not just set the flag? This would make it easier to work with utf8 in the meantime...

Stay tuned for the next chapters in the utf8+DBD::mysql saga...

Update: another interesting link about MySQL, utf8, and Moveable Type.

Technorati Tags: , , ,

December 15, 2005

Blog recommendation

One blog that I follow with interest is Bruce Schneier Schneier on Security. I like his practical approach to security, but most of all I like to read some of the clippings he collects from several sources and from anonymous emails he receives.

Just today he posted two articles, all good:

Recommended.

Technorati Tags: ,

December 14, 2005

Using Launchd in Mac OS X

In case you where wondering where all your crontabs went after you upgraded to Tiger, check out Mac DevCenter article about launchd.

Good stuff.

Technorati Tags: , ,

Tunning your broadband connection

Apple recently released a Broadband Tuner application for Mac OS X. Basically it sets some kernel parameters in /etc/sysctl.conf, regarding network buffers.

Well, Aaron Adams (one of the switchers in the Apple campaign, in case you forgotten), has some alternative configurations that you might want to try.

I'm using them with no problems.

Technorati Tags: , , , ,

Tip: change some key combinations in Mac OS X

There are some key combinations that drive me nuts in Mac OS X.

For example, Cmd-Q and Cmd-W. The first quits the application and the second one closes the window. They are too freaking close of each other, and its very easy to trigger the wrong one. After quitting Safari with N tabs open, you either buy Saft and activate session history, or read below. Or both as I did.

To solve this, just open System Preferences, and select Keyboard & Mouse preference pane. Then select the Keyboard shortcuts tab.

To remap the Quit Safari command, click the + button, choose Safari from the Application drop-down, type Quit Safari as the Menu Title, and then choose the new keyboard shortcut (I choose Cmd-Opt-Ctrl-Q, all three modifiers and Q). Click Add to finish.

After this, relaunch Safari. You should now have your new shortcut in place. No more premature quittings.

By the way, the other two shortcuts I always change is the Reply and Reply All in Mail.app. I switch those two, giving Cmd-R to Reply All and Shift-Cmd-R to Reply. I usually use Reply All much more than Reply.

Technorati Tags: , , , , , ,

October 27, 2005

More about Trac and Darcs

Previously, I was using a setup with Tailor to see my darcs repositories with Trac.

It works ok, but there are some issues, the most annoying one is that the commits don't preserve the original author in the meta-data of the change-set, only in the comment. So, all the SVN change-sets you see in trac are made by the user under which you run tailor, and in the comment with each changeset you have something like:

Tailorized "bugfix: make sure curso_id is in the stash" 

Original author: Pedro Melo <melo@simplicidade.org> 
Date: 2005-10-06 08:00:11

Not that nice.

In the process, I discovered the trac+darcs project (also mentioned by Shae Erisson in the comments of the original story) and started peeking from time to time, to see how it was. And tonight I had some hours to kill, so I upgraded my 0.8.4 Trac install to the latest trac+darcs version.

The upgrade consists mainly about following these two pages: the Trac Guide about upgrade, and the Trac+darcs install page.

In my case, this commands took care of things:

# remove old trac
rm -rf /usr/lib/python2.3/site-packages/trac /usr/share/trac

# upgrade to new version
# note: me thinks darcs get --partial is faster in the next command...
cd
darcs get http://artiemestieri.tn.it/~lele/projects/trac+darcs
cd trac+darcs
sudo python ./setup.py install

# and now convert your trac environments
# for each one, do
trac-admin . upgrade
trac-admin . resync
trac-admin . wiki upgrade

I was very well impressed: it went beautifully well, I could still see my SVN repos of current projects, and after changing the configuration of a new project from SVN to darcs, and using trac-admin . resync in the trac environment directory, I could see all my darcs change-sets in glorious detail.

With this upgrade, the Trac version is now 0.9pre2 with all the goodies, and I also installed the WebAdmin plugin (nice), the AutoNav macro (simple and effective; must also try the TracNav macro) and the LegendBox processor (the version online did not work with 0.9pre2, but I fixed and uploaded a new version that does work).

Please note that the trac+darcs project also supports Bazaar-NG repos, but I don't use them, so I don't know how well it works.

All in all, a smooth upgrade, recommended to all the darcs users everywhere.

Kudos to Lele and Zooko for such a nice system.

October 20, 2005

Some rich text editors

In a project that I'm working on, there is a requirement to use some sort of rich-text editor for textareas in a web-based forum system.

I did a quick survey and found a lot of systems out there. I collected some links and I'm dumping them here, for future reference.

Right now, my prefered solution is TinyMCE.

I also found a long list of editors with useful comparisons between them.

October 12, 2005

Upgraded individual article archive templates

I like to have the same sidebar on the right of my pages, both in the main page and in the individual article pages.

With MT, I always edited the templates to add the same sidebar on both the main index template and in the individual archive template.

Today, I tried something new. I added a new index template, sidebar.html, and copied the sidebar from the main index template. Then I removed that content from the main index template and replaced it with a include of the new template. Also I edited the individual article template in the archive section and added the same include there. You also have to change the class of the element to ''layout-two-column-right''.

Now, my sidebar code is in a single template, for easy updating, and I can update a single template to change the sidebar on all the pages.

The include part is done in realtime using PHP. You can, of course, use whatever you want, but for simple include stuff, PHP is the best solution.

So, recap: to make certain elements of your blog layout easy to update (as in, having them in a single place), do this:

  • create a new index template with the common elements;
  • replace the places where they where being used with includes;
  • adjust css classes if needed.

For example, to make the main page sidebar appear on all the templates, do this:

  • from the main index template, take out the <div> with the ''beta'' class;
  • replace it with a <?php include "sidebar.html" ?>;
  • create a new sidebar template, and paste the original code there (make sure you check the ''Rebuild this template automatically when rebuilding index templates'');
  • alter the individual article archive template, change the <body> element to the class ''layout-two-column-right'' and after the ''alfa'' <div, add something like <? php include("../../../sidebar.html") ?>. You have to adjust your path to the correct number of directories you crawl back, it depends on your archive URL layout.

Then make sure you setup Apache to filter all the HTML files via PHP. And then rebuild your entire site.

October 01, 2005

Tip: notify long running terminal jobs via Growl

Sometimes you need to run some script or program that will take a long time.

What I usually do is opening a new window, start the script, and then minimize or send to the back that terminal, leaving a small corner visible to serve as a status. I could glance to the visible corner and look for the prompt to see if had ended.

Now, I use this little super simple script (could also be a bash alias, probably):

 #!/bin/sh
 #
 # Runs script, and prints a notification with growl when it finishes
 #

 $*
 growlnotify -m "Script '$*' completed" -s "Background script notification" &
 # --script ends here--

My version is called n, just n. So I can type n update_mini_cpan and have a nice notification when it finishes via Growl.

Test with n ls -la if you want.

August 20, 2005

Trac + darcs = I'm in heaven

In the last couple of months, I've stumbled across Trac for several times. It seemed very nice, extremely well integrated, but it would only support Subversion as his SCM system, and as you may have read, I'm more a darcs kind-of-a-guy.

So I never payed much attention to it.

I did jump when Justus Pendleton announced a patch to add support for darcs some months back, but I've read somewhere (can't seem to find the post now) that this patch only supported a older version of Trac. There is some talk about support darcs and other SCMs natively sometime in the 2.0 milestone, but that seems way of. The ideas regarding expanding Trac to other SCMs are also mentioned in this thread.

Last thursday though, I had lunch with João Gomes (who seems to have misplaced his blog/homepage), who is working in his course final project, and he is using Trac to manage his work between him and his partner. He gave me the tour, and frankly, I was totally sold. I had to find a way to make it work with darcs, and also, try to make it work at work, where we use CVS.

After some hours thinking, searching and browsing, I had a plan: use darcs to manage your code, just export your changes to a SVN repository on a regular basis to make Trac work. This idea should also be applicable to other SCMs like cvs, monotone, git and the arch family.

So now I had this tasks:

  • install Trac and all it's dependencies;
  • install and learn how to use svn;
  • learn how to configure it and make all the features work;
  • export a darcs repository to a svn repository, with regular updates.

Installing Trac

The server had a Fedora Core 3 setup, so I figure it should be easy. Rui spend most of the day telling me that it would be a nightmare, because the last time he tried to install this, it was a mess with dependencies. I have to admit, I was bit scared.

Yet, it was pretty painless. I followed the instruction on how-to setup a Trac in a Fedora Core 3 environment at the Trac project website, and I was happy.

I add to manually install the SilverCity Python module to have syntax coloring in Trac. It was painless with the usual python setup.py install.

Also, I had to download a Trac post-commit hook for Subversion from the Trac project site. Use the original raw format download link at the bottom, but be careful: in my case, it added some HTML to the end of the script. You should only copy-and-paste the script up to the HTML Doctype declaration. Instruction on how to use this post-commit are at the top of the script.

Installing svn and learning how-to use it

The Trac how-to on Fedora Core3 included the Subversion installation instructions. They worked without any problems.

As I don't need or want a Subversion server, there is no setup of those components, just the creation of Subversion repositories with the svnadmin tool.

Learning to use it was also easy. The Subversion book is excellent. The command line interface of svn is essentially the same as CVS, which I use for quite some time. No problems there. The administrative side of Subversion, creating repositories, choosing data stores, and configuring post-commit hooks was also simple.

All in all, I browsed the first 4 chapter of the SVN book, and read chapter 5 with some care. It took me less than an hour.

Configuring Trac

My advise: start simple. You can read the Trac Guide in less than an hour, but don't change everything at once.

The thinks I changed where:

  • Set up the post-commit hook: allows you to comment and close tickets directly from commit messages;
  • Setup permissions with the trac-admin command to your self: I gave me TRAC_ADMIN privileges for now.

The Apache configuration was simply copy and pasting from other places. Eventually I arrived at this one

<VirtualHost *:80>
  ServerName mytrac
  DocumentRoot "/servers/sites/mytrac/docs"
  ErrorLog "/servers/logs/httpd/sites/mytrac_error_log"

  RewriteEngine On
  RewriteRule ^/$ /cgi-bin/trac.cgi [R]

  ScriptAlias "/cgi-bin/" "/servers/sites/mytrac/cgi/"
  <Directory "/servers/sites/mytrac/cgi-bin">
      AllowOverride None
      Options None
      Order allow,deny
      Allow from all
  </Directory>

  <Location "/cgi-bin/trac.cgi/login">
      AuthType Basic
      AuthName "trac"
      AuthUserFile /servers/sites/mytrac/etc/trac.htpasswd
      Require valid-user
  </Location>

  <Location "/cgi-bin/trac.cgi">
      SetEnv TRAC_ENV "/home/melo/sites/trac_instances/test2"
  </Location>

  Alias /trac /usr/share/trac/htdocs/
  <Directory "/usr/share/trac/htdocs">
      Options Indexes MultiViews
      AllowOverride None 
      Order allow,deny
      Allow from all
   </Directory>
</VirtualHost>

I had to copy trac.cgi to my local cgi directory making sure it was executable, and also I changed the ownership on the Trac environment to the user that Apache runs on.

Although the instructions say that you should also change the ownership of your subversion repository, I think that this is only required if you use the Berkley DB data store, which requires a writable repository even for read-only operations. I'm using the FSFS data store that does not have this restriction. I didn't change the ownership of my repository, and so far it seems to work ok.

After getting this to run, I also make the setup of the tracd standalone Trac server. You just need to create a htdigest password file with the htdigest command of Apache, making sure that you use the trac realm.

I think I'm going to use the tracd live instead of Apache, at least until I try the FastCGI stuff. I like the support of multiple projects that tracd has, it seems better than the Apache setup.

You can also mess around with the config.ini file of your Trac environment. I did it just to change the default_charset to UTF-8, and enable logging.

After this, I was up and running and I had a working Trac installation.

Exporting a darcs repo to a svn repo

Now the fun part: how to keep using your favorite SCM and use all the Trac goodness.

Enter the Tailor script: a almost universal bi-directional SCM converter.

I'm using the old version for now, only because I found some other people using the same setup, but my final setup is going to use the currently in beta tailor, that gives a lot more flexibility (my favorite feature of the new version is the ability to preserve the author and the date of the change-set when you sync from darcs to svn).

After a quick download with darcs get --partial http://nautilus.homeip.net/~lele/projects/cvsync/, and a quick read over the README, you let it run:

  1. check out your target svn repository;
  2. bootstrap your svn repo with the contents of the darcs repo;
  3. update the sync after each change-set.

I created a directory on my home directory, darcs2svn, to keep all the stuff to do this migrations.

mkdir darcs2svn
cd darcs2svn
svn co file:///home/melo/sites/svn_repos/test2/trunk test2
Checked out revision 4.
~/src/cvsync/tailor.py \
    --bootstrap \
    --source-kind=darcs \
    --repository=/home/melo/sites/darcs.simplicidade.org/docs/repos/darcs2rss \
    --target-kind=svn \
    ~/darcs2svn/test2

After this we have a single change-set in the svn repository and I'm able to see it from Trac.

This is not that good. The history is lost. I found two workarounds to this.

The first one is to use the --revision parameter of tailor.py. It allows you to specify a tag on the darcs repository. It will then import all the change-sets up to that tag as a single one into svn. Then, further updates will be imported one by one. It's not perfect, because if you don't have a tag early on in your darcs repository, you might lose a lot of change-sets.

A better way is to use a temporary darcs repo and then hack a bit the tailor status file. First you create a new empty darcs repo. mkdir x; cd x; darcs init. Then to prevent tailor from failing, you create an empty change-set. The best way to do it is to tag your empty repository with the darcs tag command.

Now you can run the tailor bootstrap command changing the --repository to point to the temporary darcs repository. It will create the initial bootstrap and state information.

Now you need to change the tailor.info inside your svn repository. You'll see that the 4th line is your source darcs repository (notice that I'm not using the latest betas of tailor, the format of the tailor.info might have changed in later versions). Just change it to your real darcs repository.

Whenever I do a new change-set in darcs, you just need to run ~/src/cvsync/tailor.py ~/darcs2svn/test2. All the change-sets that are missing from the svn repository will be imported, one by one.

To make sure that Trac is in sync with all this changes, run the following command: trac-admin environment_path resync.

That's all there is to it.

One small warning: after the import, you might notice that the number of change-sets in darcs is bigger than the revisions in svn. The tailor script tries to import darcs change-sets that the darcs tag command creates, but those change-sets don't change any files, so no revision is created for them. This means that the subversion repository will not have your darcs tags, but it explains the difference between the number of change-sets.

Things to improve

This setup works but it's still not perfect. I haven't figure it out how-to bootstrap the svn repository with all the change-sets from the darcs repository. Right now, he bundles all the darcs change-sets in a single one. I assuming that this is a limitation on my part. We'll see.

Update: You can use an empty temporary repository and some small hack to work around this and import all the change-sets. See above for detailed information.

Also, the author and date of the change-set is not kept between repositories. This limitation seems to be solved in the new version of tailor. I'll have to try it

Final Notes

Until Trac gets darcs support, this setup works very well for me. I'll be moving my projects to Trac in the next weeks. I'll probably wait a bit to use the new tailor script.

By the way, it took me more time to write this how-to than to setup up the entire system...

After I wrote all of this, I found this message in the Darcs users mailing list. Well, I never said I was the first one to think about this stuff, but I hope this helps others to use Trac with darcs or other SCM supported by the tailor script.

I'll keep this post updated as I refine my solution.

  1. 20050819: initial version
  2. 20050820: workaround to import all the darcs change-sets
  3. 20050820: spelling, some clarifications about number of change-sets

Technorati Tags: , , ,

July 29, 2005

The best WTF so far, IMHO

Pure fun. For me, the best WTF bar none..

June 04, 2005

Quick startup to Apache bliss

So you need a Apache setup with PHP or mod_perl, and you need it fast.

Head on over to ApacheFriends website, and download XAMPP. It's a pre-compiled version of Apache with all the best and most useful modules already compiled and configured.

They have four distributions: Linux, Mac OS X, Windows and Solaris. All the versions are stable except the Mac OS X. They are still ironing out the last details on that one.

The Mac OS X version includes:

The distribution for Mac OS X contains: Apache, MySQL, PHP & PEAR, SQLite, Perl, ProFTPD, phpMyAdmin, OpenSSL, GD, Freetype2, libjpeg, libpng, zlib, Ming, Webalizer, mod_perl, phpSQLiteAdmin.

Very cool stuff.

May 28, 2005

CPU usage of Skype

My Skype is idle right now, sitting there on the second monitor.

But top tells me that Skype is using 10% of my CPU. 10%!?!?!? WTF? It's just sitting there!

Doing a ktrace on the process, gives:

   484 Skype    CALL  getrusage(0,0xf03b6c20)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0xffffffff,0xf03b6c20)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0,0xf03b6c00)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0xffffffff,0xf03b6c00)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0,0xf03b6ca0)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0xffffffff,0xf03b6ca0)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0,0xf03b6bb0)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0xffffffff,0xf03b6bb0)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  select(0x1b,0xf03b6ad0,0xf03b6b50,0xf03b6bd0,0xf03b6c50)
   484 Skype    CALL  getrusage(0,0xf01b2c00)
   484 Skype    RET   getrusage 0
   484 Skype    CALL  getrusage(0xffffffff,0xf01b2c00)
   484 Skype    RET   getrusage 0

Pages and pages of this...

And looking at a lsof of the process, there is a single connection to a specific IP address, but running a tcpdump to catch all the traffic going to the address, shows that there is very little traffic.

So what is Skype doing with the idle time?

May 26, 2005

Password corrupted with Skype upgrade: Fixed

I was using version 1.0.0.18 and decided to upgrade to the latest 1.0.0.24 (and skip the beta .30). After restarting Skype, I got a famous "password corrupted, inform skype" message.

Anyway, my fix was going into Keychain Access, and deleting both Skype and Skype.credentials entries.

Just for experimental value

So my new cable modem backup service is a per-usage plan. When I connect my computer, I have to go to a specific web site and authenticate. Then I'm able to connect to everywhere.

The thing is, before authentication, I can ping anywhere on the internet.

hmms...

Ping Tunnel: sending TCP traffic over ICMP

I think I'm going to set this up, for experimental purposes of course.

Update: OK, I couldn't wait to get home. Download the software, run make on the client and the server, start the ptunnel on the server. On my laptop, run it and redirect port 22 of the remote server to 8765 of my localhost. Try a ssh session on top. Worked first time. It took me less than 3 minutes to do it all.

Update 2: Yes, it works with my cable provider and it's metered service. Large transfers are a bit flaky but occasional ssh sessions work very well.

April 30, 2005

Tiger notes

As I said before, I'm not upgrading to Tiger until sometime next month. Yet, I need to keep some links to things that I need to test or fix in my Tiger setup.

The good news in Tiger for scripting fans like me, is that Perl and Ruby have decent and recent versions. The bad news is that at least Ruby has a bad configuration file. From what I could understand, it seems that they left a CFLAGS=-arch i386 in a config file... I can't wait the rumor sites to get this one... A fix is available.

One thing I want to test is the new CoreData framework. Cocoa Dev Central has two articles entitled "Core Data Class Overview" and "Build a Core Data App" that seem very nice.

I'll keep this post updated with all my things-to-see in Tiger-land.

April 24, 2005

Clarification about last post

The fact that I had to write so little code is a tribute to two things: the design of Apple APIs is very clean; and Ruby integration with ObjectiveC runtime is almost perfect.

Doing a web browser with XCode is equivalent to the early 90's 'Hello world'.

One interesting thing. I'm almost sure that I can reuse the same NIB file with a CamelBones project. I'll try it next week and write the same app with it.

April 23, 2005

Burn and distribute

First, grab a couple of blank CD-R discs, one for each manager of a tech company you know. Put an extra for you. Make a list.

Then, go to http://www.itconversations.com/shows/detail188.html and download the best quality audio file of the "Great Hackers" presentation by Paul Graham, OSCON 2004.

After you have burned all the CDs send them to each manager on the list.

The best presentation I ever listen to from IT Conversations.

April 19, 2005

Learning Ruby

I needed something new to do, to learn, and I decided to learn a new language. That was 2 months ago more or less.

I though about learning OCaml or Haskell but I don't have the correct mental model to learn something like that just now. So I went with something easy, given my background.

I picked up Programming Ruby a month or so ago, and I just finished my first reading of it. I must say that I was very impressed. The language is very clean, very simple and seems extremely powerful.

I was browsing some Ruby sites and I went to Ruby on Rails website. I had heard about it from several places, the latest on was Tada lists.

Let me put it this way: if you want to get hooked, download this 47Mb video and watch the full 30 minutes of it. It's something short of a revelation.

I've been a perl programmer the last 13 or 14 years, but this small video made a small place for Ruby in my toolbox. will see how this small place grows in the next months. Now, I must find a small project to try Ruby on Rails...

April 15, 2005

Psi nightly builds

So I'm a lazy bastard, so far nothing new. But I also want to look at the latest Psi, ze mozt powerfull Jabber client out there, releases for Mac, and lo and behold they have nightly builds...

This is the script I ended up with to update my Psi every day:

melo@RoadRunner(~/Desktop/psi-night-build)$ cat update_psi.sh
#!/bin/sh
#
# Updates Psi with latest nightly build
#

today=`date +%Y%m%d`

if [ -d "psi-mac-$today" ] ; then
  echo "You already have build $today, restart your Psi to use it, maybe?"
  exit 1
fi

curl -O http://spike.ulyssis.org/psi/mac/psi-mac-$today.zip

unzip psi-mac-$today.zip
rm psi-mac-$today.zip
rsync -a into_frameworks/ psi-mac-$today/Psi.app/Contents/Frameworks
rm -f psi-latest
ln -s psi-mac-$today psi-latest
growlnotify -n "Psi Nightly build updater" -s -a Psi 'Psi is updated!' <<EOF
You Psi was updated with the latest nightly build, $today. Please Quit and restart you Psi.
EOF

Make sure you chmod 755 update_psi.sh.

Now you just need to restart you Psi every morning after the nice growl notification appears. I keep ~Desktop/psi-night-build/psi-latest/Psi.app on my Dock.

Couple of notes: on the directory where you run update_psi.sh, you must have a directory into_frameworks. Inside you need to put all the Frameworks required to run Psi. Right now, you need to have the libqt-mt.3.dylib (found on the same directory where you can find Psi nightly builds), and the Growl framework.

I used the copy of the Growl framework that comes with Ecto. Use locate Growl.framework to find a copy on your system, and copy it into into_frameworks.

My into_frameworks looks like this:

melo@RoadRunner(~/Desktop/psi-night-build)$ ls -la into_frameworks/
total 18376
drwxrwxr-x  4 melo  melo      136 15 Apr 11:01 .
drwxrwxr-x  9 melo  melo      306 15 Apr 11:29 ..
drwxr-xr-x  7 melo  melo      238  4 Apr 04:17 Growl.framework
-rw-r--r--  1 melo  melo  9404804 16 Aug  2004 libqt-mt.3.dylib

The main directory where I keep update_psi.sh looks like this:

melo@RoadRunner(~/Desktop/psi-night-build)$ ls -la
total 128
drwxrwxr-x   9 melo  melo    306 15 Apr 11:29 .
drwx------  23 melo  melo    782 15 Apr 11:21 ..
drwxrwxr-x   4 melo  melo    136 15 Apr 11:01 into_frameworks
lrwxrwxr-x   1 melo  melo     16 15 Apr 11:12 psi-latest -> psi-mac-20050415
drwxr-xr-x   5 melo  melo    170 15 Apr 02:25 psi-mac-20050415
-rwxr-xr-x   1 melo  melo    598 15 Apr 11:27 update_psi.sh

Although the Growl framework is required in latest builds, I cannot seem to get it working. If you know how, xmpp me at melo funnychar simplicidade.org.

Update: Growl works very well if you use the Remko versions. See his Psi page.

Update 2: small bug in the if [ -d ... in the script, change '' to "". Thanks go to rebarbado.

March 08, 2005

It depends

When writing Perl, should you use this:

  if (!unlink($file)) {
   $log_main->warn("Could not delete temp file '$file'");
 }

or this?

$log_main->warn("Could not delete temp file '$file'") unless unlink($file);

The second one is shorter, and has extra-geek-points, but I say that "it depends".

In this case I settled on the first form because the important part, the important action, is the unlink() call. You should always put the important action first.

February 19, 2005

Some things you learn or discover

During the last 48 hours a lot of things have happened and I dealt with a lot of hardware. I took some notes for future reference.

First, always take a camera with you. Take pictures of everything, and then catalog them. You can always go back and see how things where inside your servers or around them. Don't remember if a ethernet cable is connected to the onboard ethernet or the external one? Just go and look.

Second, some vendors have some strange notions about accessibility, so be prepared.

Third, forget about boot floppies and CDs, just install a recent linux distribution in a brand new IDE disk, and use that as a rescue system.

Fourth, make sure your backups work, and make sure they are compreensive.

Fifth, if your monitoring system is hosed, don't wait a month or two to put the new one in place.

That's all folks,

Found it!

In case you're wondering why this server was off the network from time to time, well, the poor thing was crashing. On a regular basis. Like a religion.

I made all the upgrades, I tried some tips from friends, but it kept crashing.

But now I know. Bad memory.

Well, I've removed the bad memory (at least I hope I did remove the right one...) so it should be more stable now.

BTW, memtest86 is available directly from any Fedora boot CD. Just type memtest86 at the prompt.

February 05, 2005

Using mairix with Mail.app

I was reading Sam Tregar post in use.perl regarding mairix and I though: "that would be cool with Mail.app and my bazillion folders...". At least until Spotlight gets here...

So after 15 minutes trying to understand how Mail.app stores the IMAP messages locally (it stores them in a mh folder, at least similar enough for mairix), I did a small .mairixc, set the output as mbox into a newly created folder, and pointed mairix to a incoming IMAP folder.

Run it once without parameters to index all the stuff, and run it again with a string to query the database.

Matched 82 messages

Whoa! Already? :) Jump back into Mail.app, look at the folder created and lo and behold, they where there.

Very cool!

So, the steps you need to take (this is a beta how-to, ok? the steps will get better):

  1. Download and install mairix: it compiles cleanly in my Mac, but the install failed for some reason, just copy the binary to someplace in your $PATH;
  2. Create a Results folder on you Local Mac. It must be created in the Local Mac.
  3. Use my modified .mairixrc and adjust the following variables:
    1. base: the base where all your Mail.app info is stored, usually /Users/**your_short_name**/Library/Mail;
    2. Comment all the lines starting with maildir and mbox;
    3. mh should be the path of one of your folders in a IMAP account, something like IMAP-**name_of_account**/INBOX/Sent.imapmbox/CachedMessages. One way to find this out is to run this command (find ~/Library/Mail -type d -name CachedMessages) from the terminal. You can put as many lines as you want;
    4. mfolder should be changed to point to your results folder that you created above. If the name is Results, mfolder should point to Mailboxes/Results.mbox/mbox;
    5. Remove the comment in the mformat=mbox line: this specifies the result folder as a mbox-style folder;
    6. Point database to a place in your home directory: mine is /Users/melo/Library/Mail/mairix_database;
  4. run mairix once. It indexes all the folders you specified in step 3;
  5. now query the created database: mairix some_string - it should display the number of messages found. Jump back into Mail.app, check the Results folder and they should be in there.

Don't forget to run mairix from time to time to update the database. Use Cronix or edit a crontab maybe.

Next steps: I only indexed one folder as you can see from the example .mairixrc. I need to write a small perl script to generate my .mairixrc file with all my mailboxes, and do a AppleScript to call the mairix command with the query.

January 22, 2005

RSS your yum updates

If you like RSS and use Fedora Core 3, then you can have the nightly yum update reports delivered to you via RSS.

The following setup requires: * using fedora core 3, or having the command rss-generate in yum (check man yum); * your mail is delivered via qmail; * you can install .qmail files.

You probably could workaround the last two. I might do that someday. For now, consider this as a quick (took me less than 20 minutes after I noticed rss-generate in yum manpage :) ) hack.

The setup is simple: each night, a cron runs on each server, and generates the RSS file. The file is then sent to a special mailbox, and there we filter the message and place it in a public HTTP site. You then configure your favorite RSS reader with that URL.

First I created a email address with .qmail file in my homedir:

cat > ~/.qmail-yum <<EOF'
|/home/melo/bin/convert_yum_rss_mails_to_http.pl
& youraddress@yourdomain.top
EOF

The second line should be a good email address. If something goes wrong with the script, it will fallback to the email address. At least you'll know that something has gone bad.

Then you place the perl script in the correct place. You can download the convert\yum\rss\mails\to\_http.pl script, or paste it:

#!/usr/bin/perl -w

# Where to place RSS files? Should be available via Web
my $dir_for_rss_files = "/var/httpd/htdocs/yum_rss";
my $server;

while (<>) {
  $server = $1 if /^Subject: Yum changes for (.+)/;
  last if /^$/;
}

fallback('no_server') unless $server;

my $filename = "$dir_for_rss_files/$server.xml";

umask(022);
fallback("could not create file '$filename.new'") unless open(RSS, ">$filename.new");
print RSS <>;
close(RSS); 

sucess() if rename("$filename.new", $filename);

unlink("$filename.new");
fallback("could not rename file '$filename.new'");

# qmail exit codes, see qmail-command man page
sub sucess   { exit(99) }
sub fallback { my $m = shift; print STDERR "Err: $m\\n"; exit(0)  }

That's it. If something goes wrong, check your qmail-send logs. A message is written there.

Now you only need to generate the mails on each server. To do that download the script yum-rss-report.cron. It looks like this:

#!/bin/sh

/usr/bin/yum --rss-filename=/dev/fd/1 -R 120 -d 0 -e 0 generate-rss |  \\
/bin/mail -s "Yum changes for `hostname`" youraddress-yum@yourdomain.top

This command will generate the RSS (the -R 120 will make sure that each server will wait a random number of seconds), and mail it to your address that you set up before. Make sure you change the address in this script, and make sure you can receive mail at that address.

Then, copy the edited script to all the servers that you want to report to you. On each one, copy the script to /etc/cron.daily and make sure you make it executable:

cp yum-rss-report.cron /etc/cron.daily
chmod 555 /etc/cron.daily/yum-rss-report.cron

That's it! Just subscribe each server in you favorite RSS reader and each morning you'll know which servers got upgraded and to which versions.

I have some improvements to make to these scripts, but they are good enough for now.

Enjoy!

January 21, 2005

Line noise

I was editing some Perl code and I needed to comment the function I was working on, from the cursor till the end.

My editor of choice is vim for quite some years now, so I typed this:

:.,/^}/-1s/^/#/

It never ceases to amaze me as line noise can be so productive and intuitive if you know your editor and your regexps.

December 08, 2004

darcs2rss

I've been using darcs for some weeks now, and I like it a lot. I'll write more about darcs and why I like it in a future post, but I'm switching from CVS to darcs for my current and future pet projects.

Last friday, Bennett Todd was asking in the darcs-users mailing list for a script to generate a RSS feed of darcs changelog. I send him a quick perl script to do just that. I also got some sugestions and improvements.

So I created a darcs repo for this project. You can find it at http://darcs.simplicidade.org/repos/darcs2rss/. You can browse all my repos at http://darcs.simplicidade.org/.

Next, I have to make a small page about it. Not today, though. Real life is waiting. Meanwhile, you can use the readme to know more, specially the roadmap.

November 19, 2004

darcs on Mac OS X

In case you want to try darcs with Mac OS X, these links are usefull:

Have the appropriate amount of fun.

Update: fixed the URL's, Markdown was messing them up.

October 26, 2004

How-to: controlling your iTMS bill at the end of the month

Ok, so now you have the iTMS available, and you see that it's the first day and you bought around 100 songs. And you think: I need a raise.

Have no phear, I is here, and with a solution. Give your self an allowance. From the iTMS home page, select Monthly Gift, and select the amount you want to spend each month. You'll need to create a second .Mac account for this controlled shopper, but it works.

September 24, 2004

Quick del.icio.us & QuickSilver tip

Before you say anything, yes I know that Quicksilver has a plugin for del.icio.us, but this is different.

Quicksilver can be configured to do web searches. You can use that feature to jump directly to a tag-filtered URL like this.

To do that, follow this steps:

  1. bookmark in Safari the following URL: http://del.icio.us/melo/***. Change melo to your username or to tag if you want to search globally. Give a short name to the bookmark (I use del). You'll use this name in Quicksilver;
  2. activate Quicksilver (CMD-Space in my setup);
  3. rescan the Catalog (CMD-R) so that he finds your new bookmark;
  4. type the bookmark name, press Enter and type your tag query: css or css+ie_bugs or whatever. Type Enter to execute.

From now on, you can use this by just doing step 4.

Simple and efficient. I love Quicksilver.

Making Markdown the default formatting in ecto

It took me some time to figure this out, so to those of you who are as dumb as me, here is how you do it:

  1. First, go to your blog setup, and make sure Markdown is the default formatting option. In Moveable Type, you need to select Edit Configuration > Preferences > Default Text Formatting for New Entries;
  2. go back to ecto, and choose Window > Accounts, select the blog you want, and click Update cache in the toolbar.
  3. In my case, I also changed the ecto preferences, in Editing, the default mode to HTML. If you are going to use Markdown, you might as well change all the HTML Text colors in that pane to black.

That's it.

Ecto2 is comming

There is a new version of ecto ready to be released. I've been using the betas for some weeks now, and I'm very happy with it.

Check out the latest features. The main one seems to be WYSIWYSG editing, but I started using Markdown a couple of days ago so I cannot say if it works or not. For me, having multiple posts at the same time, and a improved entries & drafts window are welcome additions.

My remaining problem is the trackback interface: it seems that I cannot add trackbacks with the keyboard only.

Anyway, it's a very good release, and if you are looking for a Mac OS X app to blog (Windoze version also available), highly recommended.

September 18, 2004

Flickr tips

Flickr is really great. I joined a couple of weeks ago, but I only got a chance of doing some serious setup with it.

I'm using the following tools:

  • iPhoto plugin for flicker: really really simple way of uploading you photos directly from iPhoto. It lets you choose the title, tags and even resizes the photo before uploading. The current version (0.7) has a small bug: it strips the EXIF information, so it's not really ready yet, but the author is looking into it;
  • Flickr Screensaver: really nice, but could be better: I would like to be able to choose to see all my contacts photos instead of my own only. You can choose to see favourites, which is nice;
  • Keyword Assistant: not really Flickr related, but if you use iPhoto, this will help you tag all your photos and that makes it easy to upload them to Flickr.
  • Don't forget that you can upload photos via email, and even better, you can also post a photo to your weblog via flickr by email. It's twisted but it works: just setup you blog in Flickr, and add a email address to it.

And now the wishlist: open the Pro Accounts soon, 10Mb upload per month is not enough :)

September 08, 2004

This site best viewed with IE7

Amazing what you can do with Javascript. Checkout this Javascript library by Dean Edward that you can include in your pages that makes IE actually support standart web stuff like CSS2.

Very cool. Via The Lunatic Fringe

August 29, 2004

Mirroring CPAN in your laptop

For some years now, I always have a full CPAN mirror on my laptop. I work a lot offline and I like to have it around.

Until today I did a full rsync from a public mirror from my ISP, but Nuno sent me a script to mirror only the latest versions of each module but I never got around to use it.

But I noticed that the script he sent me is now (well, I noticed it today...) on CPAN itself, so I just did:

$ sudo bash
# cpan CPAN::Mini
# cd /sw
# mv cpan cpan.full
# mkdir cpan
# minicpan -l /sw/cpan/ -r file://sw/cpan.full/ 
# du -hs cpan cpan.full
409M    cpan
3.2G    cpan.full

hmms... 1.8Gb free space... nice... I keep it up-to-date with minicpan -l /sw/cpan/ -r ftp://ftp.ip.pt/pub/perl/.

Is your Linux server too slow?

Have no phear, hdparm is here...

Check out the article of USA Lug about hdparm, it's really cool and it helped me to tune my Linux server.

I knew hdparm for some time now, but its one of those things that I didn't feel quite comfortable to do until I read about it some place else.

Coral your RSS

Now that Coral is in public beta, what about telling your RSS subscribers to use it? It will reduce your bandwidth bill.

If you subscribe to this blog, change your subscription to http://www.simplicidade.org.nyud.net:8090/notes/42.xml (notice the nyud.net:8090). You should start to get this feed from your nearest cache of Coral.

Seems to work.

If Coral is here to stay, you can even add to your config:

RedirectPermanent /notes/42.xml http://www.simplicidade.org.nyud.net:8090/notes/42.xml

Update: Rui popped up on IM and told me that this does not work, and of course he's right. It will cause a loop. I'll have to check if Coral sends a specific user agent that I can filter. Rui tells me that it sends a HTTP_VIA and he has the configuration to solve this. Now I'll wait for him to post it and copy it :).

I wonder if the choice of port (8090) was a good one. Some corporate firewalls don't allow nothing unless is on port 80.

By the way, Microsoft DNS servers have a problem with Coral. The developers are thinking about a workaround for the fact that MS DNS chokes on DNAME records (I confess I didn't know about them either, but they look pretty cool).

August 20, 2004

Virtual hosts via Apple Rendezvous: solution

Just finish writing a hint to MacOS X Hints, about publishing virtual hosts via rendezvous.

Here is a copy, in case it's not accepted. I'm using this as we speak.

In a previous hint it was described how to use the RegisterResource of mod_apple_rendezvous to publish additional paths of your local server via Rendezvous.

Virtual hosts are more complicated because you cannot send the hostname via rendezvous, apparently.

But that's easy to solve with mod_rewrite. To do it, create a file (I named mine virtualhosts.conf) and place it in /etc/httpd/users. Place the following inside:

# you can put anything after /.vh/ as longs as
# it starts with the site name followed by an 
# optional path
RegisterResource "MacOS X Hints" /.vh/www.macosxhints.com/
RegisterResource "Send me one, please!" /.vh/www.apple.com/displays/

# Add other RegisterResource as above to add more sites

# Magic.....
RewriteEngine On
RewriteRule /\.vh/(.+) http://$1 [R]

and then restart your Apache webserver:

$ sudo apachectl stop
$ sudo apachectl start

You can also restart with sudo apachectl restart or even sudo apachectl graceful. I prefer stop/start.

That's it. Check you Rendezvous menu in the Bookmarks Bar and you'll find "MacOS X Hints" that will redirect to this site.

We can also use this method to support HTTPS. Just add this line to the file:
RewriteRule /\.vhs/(.+) https://$1 [R]

and register resources as:

RegisterResource "Amazon" /.vhs/www.amazon.com/

We use a prefix, /.vh/ (and /.vhs/), that should be unused in most configurations. If you happen to use it in your own configuration, change all the occurrences to something else, like /.idontusethis/.

July 31, 2004

Having fun

I have to move one mailbox from one office to another. The first is in Porto, and the second in Lisbon.

The connection is ADSL, but the wrong direction: I have to push the entire mailbox via the ADSL uplink, 128kbits.

Well, it's a 6.5Gigabytes mailbox (actually a set of maildirs), so I'll just check it monday :)

July 17, 2004

Abuse of CSS

This is the latest abuse of CSS I found.

This reminds me that I have to sort the 100+ links of CSS stuff I have in Safari.

June 24, 2004

Mail.app enhancements

Nuno was complaining about the lousy support of multiple profiles in Mail.app. I agree with him, but today I noticed that I already was using some obscure features that make it all easy :).

First, in the preferences account panel, you can put several email addresses, comma separated. They will all show up in the drop-down in the Compose window, so you can change the From address easily. With MailEnhancer, you can even change the signatures auto-magically.

For this and more plugins, try this MacZealots article, or this list.

June 05, 2004

iTunes usage

I was discussing with Nuno how he uses iTunes, and I was amazed that he likes party shuffle...

I don't use party shuffle, too lazy to pick songs and stuff. So I have a special playlist, a smart one, that I use. This playlist assures me a continuous stream of music with 3.5 days turn around time (assumes you are listening to itunes 24 hours a day, of course, so it's more like 10 days).

Here is the playlist I use. It assumes you like to rate songs. It will work out-of-the-box because iTunes gives 3 stars to new songs and I only filter those below that:

  1. Create a smart playlist
  2. Add 4 rules:
    1. Genre is not Spoken Word: remove almost all the audiobooks
    2. Genre is not Audiobook: remove the rest of the audiobooks
    3. Genre is not Speech: IT conversations
    4. My Rating is greater than XX (two stars)
  3. Make sure you have Match all the following conditions
  4. Don't limit the number of songs. If you do, chose least recently played as selection style
  5. if you disable songs with the checkbox, select Match only checked songs. Otherwise clear it.
  6. Live Updating on

I also have the prefpane Rating Bar.prefPane installed. There are other options, but I like this one.

So, this playlist makes sure I listen to different stuff all the time, and if I'm listening to a track I don't like, I just change it's rating to 1 or 2 stars and it stop's playing (you must press play after, unfortunately).

One thing I noticed, is that I don't have enough music: 10 days to rotate my music collection is small. I'll have to start ripping more CD's...

May 21, 2004

Favorite bash aliases

alias bi=vim
alias bim=vim

cfengine articles

cfengine is a great tool to manage servers. I'm starting to use it with my personal servers and at work.

O'Reilly ONLamp.com site has two articles on it. The last one shows a very nice setup for starting your own config:

Distributed Cfengine by Luke A. Kanies -- Automation is the most important skill an administrator can develop. Cfengine is great at automation and even supports distributed automation. Luke A. Kanies demonstrates how to distribute Cfengine rules to multiple machines.

It has a couple of errors though. The files update.conf (take 2) and cfagent.conf shown don't work for me. They assume a class cfengine_server, but that class is not defined anywhere. You see it defined in the cfservd.conf file, but that file is not relevant.

So to make the examples work, add the following lines to the top (before the control section) of both files. You need it in both files because cfengine discards classes defined in update.conf before starting the cfagent.conf.

groups:
  cfengine_server = ( my_cfengine_server_here )

I'll post more cfengine tips as I get to know it. So far, I'm sold.

Contacts

melo@simplicidade.org (XMPP/email)
+351 302 029 050 (voice)
melopt (Skype)

IronMan challenge

Iron Man badge Are you ready to be an Iron Man? Join the challenge and find out! (what is the meaning of this little man?)

Moosaico

Junta-te!

Recent Comments

Powered by Disqus
Creative Commons License
This weblog is licensed under a Creative Commons License.
Powered by
Movable Type 3.2