Maybe boys will be boys and we just enjoy lewdness, or maybe there is a grain of truth in it. But 6 years ago when I started watching the show, I never though the roller coast of emotions it would trigger.

Based on a set of never ending love stories, some peaceful, other just normal and a big sad one, it keeps rocking my socks off with the impossible real situations, and the fulsome dialogs.

You get to live an alternative universe in a even dozen instalments every year, and the only thing left after you are done, is a longing for the next fix.

Thank you, Hank Moody, you degenerate impossible lucky bastard. See you next year.

You'd be right. My favourite topic in college was distributed operating systems. Today, when I design a system, I always envision it as a set of cooperating processes, working together and in parallel for a common goal.

Over the past 15 years, I've kept refining a set of rules of what I think is the ideal features any application that does sync should have.

I've narrowed down to three.

1. Close your laptop and go

If you see a "sync now" button, they blew it.

To paraphrase Vince Lombardi:

Sync is not a sometime thing; it's an all time thing. You don't sync once in a while, you don't do things right once in a while, you do them right all the time. Sync is habit.

If you share application state with your co-workers (say for example a project management app, with issues, goals, notes, design documents, whatever), when you are sitting in your office, your own version of the application is running on your laptop. It has a copy of the state. Maybe not all of the state, maybe just the part that relates to you, but for now let's assume that it has all of the state.

When you close the laptop lid and move to another place, the information on your local copy should be the update version of the shared state up to the moment you lost network connectivity.

There should be any "oh I'll just wait a bit and sync before I disconnect" though crossing your mind. Sync should not be part of your thinking process. It should just be the normal world view, it should be subconscious.

One of the best outcomes of this always-in-sync subliminal state? Given that you'll be always seeing, manipulating, the latest version of something, the opportunity for conflicts based on mis-information is rare.

2. Cherish thy conflicts

Which brings us to the big bad wolf of synchronisation: conflicts. This is the primary fear most people who think about implementing a sync solution share.

What I'm here to tell you, is that conflicts are you friends. They are the tether that binds you to humanity, because they only come up when someone has a different world view from your own.

Having someone who disagrees with you is glorious! It gives you the opportunity to get out of your bubble and interact. And you get to choose how to do so. Maybe you just IM or mail him, or maybe you call him, or even share a few minutes of his time in a hall near a whiteboard.

Conflicts are nature way of telling you that you need to get out a bit and talk to someone.

3. Be a packrat, collect it all

The most important lesson I got from git was not the graph of objects on which it was built, but the fact that git never stores diffs. It will always store a complete version of the new state.

The main advantage of having the entire state blob for each version is simple: you can always improve your conflict resolution, or your diff algorithm, or any part of the UX of both, because you have the raw data available. If you store diff's, you'll be using the state of the art diff algorithm at that moment, and going back to any version means replaying all the diffs since the last full version.

You might think that this is a huge waste of space, and it could be if you have big state blobs on which only a small percentage change between versions, but I posit that this is not the most common case. In fact, big state blobs should sound warning bells in your head, and start to break them up into smaller concepts.

No, you want small state blobs, connected together. They are faster to sync individually, and they provide a smaller surface for conflict resolution when those blissful events are casted upon you.

And with small state blobs, when state changes, you should always store

Let's be about it

So you are hyped now, you want your next application to have sync. What is your next step?

The Dropbox generation

The sync experience of Dropbox is comparable to the automobile experience of a Model T. Or the the sexual pleasure of masturbation. It gets the job done, barely, but you end up thinking that should be more to life than this.

It should, and there is.

Dropbox is a fine product, I depend on it daily for a lot of things, but most of them are files.

To sync files between devices and across several operating systems, there aren't many other solutions out there with the same proven track record. So in recent time, a lot of applications that deal with files have added builtin Dropbox support as their solution for the sync problem. And it works fine, for files, and single user scenarios.

But when you start talking about application state, with multiple users, if you plan on using Dropbox, you better start modelling your data store as a series of independent files. And forget about data protection of any kind.

But it can be done. For the single user case, it is more than enough.

The best example I know (and use, and recommend by the way) is 1Password. A couple of versions back, they switched from a single file to a file bundle as their storage system. You can read all about the Agile keychain design, good stuff in there.

What they did was break a large state blob into several smaller ones.

But the Dropbox API lacks any way to be notified in real-time of changes others may have made to your files. Sure, the desktop client uses a private Dropbox API to be notified of new stuff, but that is not available to you.

So, if you plan on using Dropbox as the sync service for your app, remember that always-in-sync scenario requires the Dropbox desktop client, and you have to monitor (fsevents on Mac OS X, or inotify on Linux, kqueue on FreeBSD...) the filesystem yourself to detect changes.

But, but, but git is magical!

Yes it is. But only for some things. Files, source code, text articles.

If you plan on basing your sync solution on git, thats fine. It is a viable option now that libgit2 is stable.

Just remember that source code conflicts are easier to solve because they happen on something that has structure, something that can be validated,at syntax level by your language compiler or interpreter, and at semantic level by your test suite. You do have one of those, right?

There is no possible test suite that covers your application data semantics. You can have some business rules that must be checked before accepting changes to application state, but they will never be comprehensive.

Also, git lacks any way to notify you in real-time of new commits on remote branches, so you have to roll your own too.

Natives need not apply

You might think that all of this is my subtle way of pushing you from your friendly web to the evil seductive native apps embrace, but no.

Yes, I do believe that the vast majority of your app logic should be executed on the client side. It's not only my distributed systems engineer wet dream, but also your ecological duty not to wast all those processor cycles sitting on your lap or nicely wrapped in your hand.

I also believe that no matter how much bandwidth you have, how blazing fast your servers are, you can't beat the latency of a local app working on local data.

And besides, even if you have always-hopefully-on internet connectivity, isn't it better to hide all those wishful thoughts about "when the user presses this button I will have network access, my server will be up, and everything is peachy" in the smallest part of your application as you can manage it? If your entire app uses local data, and then, in the background, those changes are sent to the remote peers, the user experience is much faster.

So keep you mad Web skills, they are your best bet it this the brave new continent you're about to explore. Just think about bringing along a simple HTTPS server, sitting right there on your local device, holding a copy of your app, and your state.

Interactive sync

There is a crop of products that focus on real-time collaboration. Software like SubEthaEdit and recently Ex-Google Wave.

They are very specific niches of the whole synchronisation topic, and I don't think they are a good solution for most apps. A bit of overkill.

Action!

Finally, my brain dump from having done some synchronisation work.

Divide and conquer. Start with a simple problem. Don't make sync a feature in a future version, plan it from the start. It is very hard to bold sync later on, believe me.

Cherish immutable content, or single-owner content, like comments. No conflicts there. A post has a set of comments, this set is conflict free.

Embrace that conflicts are a human problem, not a technical one, and instead of wasting your time dreading them, focus on the best UX you can provide to make the work of two or more people involved easier to do: how to pick the new world view?

Start with a central meeting place, where all changes flow through. It is easier to do at first, just make sure this central place never accepts conflicted information. All conflicts must be solved on the client side befor any changes are sent.

Start with "rolling sync", the simplest approach to merges: if you have been offline for a bit of, first undo all your offline changes, apply all remote changes done since your last sync, and then roll your changes over that to catch any conflicts. If you know git, think rebase, not merge.

Use UUID's or SHA1 of content as identifiers: there is no clear cut rule on which one to pick, it really depends on the situation.

Remember the truth about clocks: everybody has one, and each one keeps his own perfect time. If only they could agree on what that perfect time is...

Don't ask permission to collect all of what you can from your peers, just do it. Later, your person can always check why was something changed, and who did it.

Its a brave new world

Sync is not just a feature anymore, it must be something that its just there, unquestioned, subliminal.

Do your part, and start designing your apps with "Sync first!" mentality.

require_cmd tidyall 'Requires tidyall, install from CPAN'

### Have a default tidyall.ini on your home, override per project
config="$HOME/.tidyall.ini"
for ini in tidyall.ini .tidyall.ini ; do
  if [ -e "${TM_PROJECT_DIRECTORY}/${ini}" ] ; then
    config="${TM_PROJECT_DIRECTORY}/${ini}"
  fi
done

## Override target file if the current scope is perl: useful when
## tidying a selection of perl code inside a Mason document
target="$TM_FILEPATH"
if [ "${TM_SCOPE:0:11}" == "source.perl" ] ; then
  target="${TM_PROJECT_DIRECTORY}/perl.pl"
fi

tidyall --root "$TM_PROJECT_DIRECTORY" \
        --conf-file "${config}"        \
        --pipe "$target"

To use it, create a new command in your own personal bundle, and set:

• Save: Nothing;
• Command(s): the above code;
• Input: Selected Text or Document
• Output: Replace Selected Text
• Activation: Key Equivalent Ctrl+Shift+H (the same as Tidy);
• Source: source.perl, comment.documentation.perl

The last one, Source, you should also add other scopes that your tidyall config covers.

My current tidyall.ini is this:

[PerlTidy]
select = **/*.{pl,pm,t}

[PodTidy]
argv = --column=100
select = **/*.{pm,pod}

[Perl::AlignMooseAttributes]
select = **/*.pm

[Perl::IgnoreMethodSignaturesSimple]
select = **/*.{pl,pm,t}

[MasonTidy]
select = **/*.{mc,mi,html}
select = **/{autohandler,dhandler}

The MasonTidy section is still new to me, I'm not sure if I like it or not. The rest is very good, specially the clean integration to tidy Method::Signatures declarations provided by the Perl::IgnoreMethodSignaturesSimple plugin.

Although this plugin targets the Method::Signatures::Simple, I'm using it with Method::Signatures, my preferred version of the concept, without problems

I've also used the recent git pre-commit hooks code for a bit, but I'm having problems with it and haven't had the time to really look into it. The code stashes stuff that is not in the index before running, to make sure it is only tidying the parts you are going to commit, and then pops the rest from the stash. This last part causes me a lot of merge failures.

If you don't use hunk or line-based commit building, you should be fine though.

One interesting side-effect of this behavior is that I haven't had a full night sleep since the day she was born.

I usually can get right back into dream land after she finishes her milk, but lately I find myself sleepless in bed after each meal. I've now taken extreme measures and doubled the number of paperback books on my bed stand. We'll see how it goes.

Usually I'm more of a audiobook person, but I find that carbon-based versions are more sleep inducing than audio. And no, I don't have a e-reader of any kind yet.

I was thinking about this just now, and how I wished she was older and over this night-time adventures into milk land. I even said out loud to my wife "I wish she was 18 and I could sleep all night"...

... Yeah, I really don't know what is about to hit me.

I guess I'll enjoy this next few years, knowing that she is safely a sleep in the bed just a couple of meters from me. I'll keep enjoying the tiresome moments together during the night. They will go away soon, and that will be a sad day.

I've used GitFlow for a couple of weeks when I found it for the first time, but dropped it eventually because it didn't fit my needs. I'm not dissing it, it is very well though out and it provides a nice set of tools to make it all work, but it also adds a complexity layer that may not match your environment.

There are two situations in particular where I would recommend not using these *Flow setups: single developer, and if you use a faster release train.

The first should be obvious: all those shenanigans moving code from topic branches to develop to release branches make sense if you have a team of people with defined roles, like module owners who must approve changes to their own turf, or release managers who take a set of features accepted into the develop branch and make a release out of it.

The second is more of a personal preference. I hate the release manager notion. I'm sure that it has value in a lot of projects, but I can say that I'm fortunate enough not to have worked on one of those. My preference came from a previous job were I was the de facto release manager, not because the position was official, but because I was more adept (or masochist, depends on the point of view) to merging CVS branches... Oh yes... CVS... Branches... Hell.

So I moved to keeping the trunk/master branch of my projects always deploy-able, to automate the release process, and to make sure releasing is a one command away, available to anyone on the team.

*Flow setups get in the way of those setups a bit. The develop branch sits between topic/feature branches and master, and delays the move from feature to deployment status of the code. I don't like that.

Having said that, if you have a team of developers, if you use git, and your rough setup is getting in the way, I would suggest you to try one of these *Flow setups. At least read them thoroughly and take the ideas that fit your local conditions.

For future reference, this is the setup I use for single-developer situations. It is based on git, but most of this could be applied to any SCM that has decent branch/merging semantics.

The master branch is the production version: it's always deploy-able, and it's also regularly smoke tested with two configurations:

• one that mimics production environment as much as possible. If your DB is small enough to have a copy available for smoking, do so. Also use the same versions of the dependencies you have in production. If you are a Perl user, cartoon is very helpful here.
• the second with the latest version of your language, plus the latest version of your dependencies: I'm using perl 5.14.2 but I smoke it with 5.16.1 and 5.17.latest, with the latest versions of my dependencies. This will catch future problems early.

All development is done on feature branches, one branch per feature. Feature branches can/should be pushed to the central repository to be picked up by the CI to be tested with a more production-like setup. When fully ready, or ready enough to be hidden from end-users behind a feature flag, merge into master and release it. I always use --no-ff when I merge, I like to know certain commits were made outside master.

If a feature is being developed for a long time (more than a month), then either rebase it regularly (my preference) or at least create a integration branch from master and merge the feature branch into it to test. It is essential to test long term branches with the latest production code.

This works for me quite well. Simple, low complexity, I can explain it in less than 3 minutes.

You have an old groff. Update to version 1.20.1 for good Unicode support. If you don't upgrade, wide characters may come out oddly.

Given that I already have homebrew to fix all my UNIX desires, I promptly executed brew install groff to fix this. You might need to brew tap homebrew/dupes before, given that groff is already included on the base system.

After a couple of minutes (and, on Mountain Lion, an extra brew install --default-names gnu-sed because system sed complains about sed: RE error: illegal byte sequence; you can brew unlink gnu-sed afterwards to revert to system sed), I had my new groff.

But now a new error message awaited me:

Error while formatting with Pod::Perldoc::ToMan: open3: exec of /Users/melo -man -Kutf8 -Tutf8 /.homebrew/bin/groff failed at /Users/melo/perl5/perlbrew/perls/perl-5.14.2/lib/5.14.2/Pod/Perldoc/Toman.pm line 327.

Notice the command Pod::Perldoc::ToMan is trying to execute mixes parameters with the command path.

The problem lies with Pod::Perldoc::ToMan. At some point it decides that it should use groff -man -Kutf8 -Tutf8 as my renderer, and it figure out that my groff is inside my local homebrew install (under /Users/melo/.homebrew, notice the .homebrew). Eventually this command is splitted into command and parameters (at line 301 of ToMan.pm, version 3.17 to be exact) and that is where the problems lies: the regexp used doesn't support dots (.) in the pathname, like my .homebrew and splits at the wrong place.

I've already sent a pull request to fix this, and it was accepted and merged into the distribution, so the next version will work fine. In the meantime, if you come across this problem, you can just hand patch your Pod::Perldoc::ToMan file like I did.

... will give you a behind-the-scenes look at Google Maps and share our vision. We’ll also demo some of the newest technology and provide a sneak peek at upcoming features...

I translate this to: "we don't really have nothing ready to ship, but we are scared shitless about what Apple is going to announce at WWDC, because it might be really really cool, and it might show just how much we dropped the ball with Maps. We got cozy with all cool stuff we did way back then, and sat on our asses, just toying with new stuff (like 3D, even), but without the focus required to really have something to launch. Yeah, we fucked up, but maybe we can trick you into looking this way for a couple of days, you know, until the WWDC buzz is over."

I really hope I'm wrong. But if I'm not, this is just sad.

Para contexto do que vos quero perguntar, sugiro a leitura deste artigo: Matem o monstro.

São dois projectos de lei que andam na alma de todos os que dependem da tecnologia para viver cá em Portugal.

Se souberem de uma maneira eficaz pela qual eu possa lutar contra este absurdo, digam-me. Já estou com um desafio novo este ano, em que umas entidades públicas mudaram as regras de certificação à qual estou sujeito (para melhor acredito sinceramente, assim que os sistemas estiverem estáveis) mas a que já estamos obrigados sem que o regulamento oficial tenha sido distribuído publicamente. Dizem-nos no final do mês... Se tudo correr bem... Uma parte interessante desse regulamento novo (porque sim, o preçário já está on-line no site deles) é que podem fazer quantas fiscalizações quiserem por ano mas eu tenho de pagar algumas centenas de Euros por cada uma delas, independentemente de ser detectada qualquer infracção.

Eu queria focar-me nisso, afinal é o meu negócio, mas parece que tenho de perder tempo com esta pérola que é a PL118 (ainda não li o PL119 mas dizem-me que é do mesmo calibre), uma vez que discos é algo que preciso regularmente para manter o meu serviço no ar, serviço esse que usa apenas conteúdos produzidos pelos meus clientes, mas pronto, pagam todos o dizimo à santa SPA.

Sinceramente, digam-me o que fazer para lutar contra isto. Este ano parece que também podemos vir a ser responsáveis pela segurança social dos nossos colaboradores (são colaborações esporádicas, pontuais) se 80% do rendimento deles for feito via a minha empresa. Eu compreendo a necessidade de apanhar fugas a esses pagamentos, mas imaginem que uma das suas empresas se preocupa em obrigar todos os seus colaboradores e parceiros a passaram facturas e recibos, e se preocupa em declarar tudo o que ganha e paga. Entre essa empresa e outra que não o faz, quem é que vai apanhar com a factura da segurança social?

2012 vai ser um ano giro para nós, os nossos clientes vão ter ainda menos dinheiro para gastar, novos regulamentos oficiais mas não públicos aos quais temos de obedecer, a potencial continha da segurança social, e outros desafios que nós próprios nos atribuímos para crescer o nosso negócio. Sem dúvida muitos desafios. Mas acreditamos que ainda vai ser possível, e até temos um terceiro filho como prova de que acreditamos que pode melhorar.

O que eu não preciso é de uma lei, criada à medida da SPA, cujo modelo de negócio está a desaparecer porque as pessoas compreenderam finalmente que é idiota comprar pequenos discos brilhantes quando é mais prático obter legalmente as coisas on-line, e que decide que todos devem pagar a uma entidade privada só para o caso de alguns andarem a piratear conteúdos. Não preciso, nem tenho tempo ou dinheiro para o que querem implementar cá, mais um regime de IVA.

Por isso, digam-me: como é que se luta contra uma injustiça destas? Ou devo desistir já de investir em Portugal, e ir procurar trabalho lá fora, onde apesar de já ter 40 anos, consigo obter emprego com bom salário amanhã, com base no investimento que Portugal já fez em mim?

tl.dr; The migration went smoothly and I'm very happy with the service. I still have two things I need to do (SSL certificate and migrate my XEP-0114 external components), but for personal reasons I won't have the time to finish the migration until next month. So expect a small update when I finish those two.

For now, I'll just describe the service and my migration process, the good parts, the could-be-improved parts and other tidbits.

The Hosted.IM service provides a hosted XMPP service powered by a carrier-grade ejabberd XMPP server. The service was created and is maintained by Process One, the main driving force behind ejabberd development over the past years. I had the pleasure of working with Process One and Mickael in particular when we migrated the SAPO XMPP server.

The basic service is free (5 user accounts, a single domain).

The main roadblock preventing my migration was the couple of custom-made external components I wrote to interact with some web and TCP-based services that I currently host, so I needed proper XEP-0114 support. I asked Mickael for it and less than two weeks later, they delivered. The XEP-0114 support was officially announced yesterday on all non-free plans.

With this final roadblock lifted, I was ready to migrate my domain.

I've registered a new account, and added my domain, simplicidade.org. The first thing they ask you to do is to add a DNS TXT record to validate that you own the domain. I don't understand why they need this TXT records. The XMPP service requires you to add or update a couple of SRV records to point to the Hosted.IM XMPP servers. If you aren't the owner of the domain, you won't be able to update those records, so why ask for an extra DNS record? I hope they clear this process and remove this particular requirement. Or, if this is really something that they really need, add a FAQ explaining why its needed.

I also immediately updated my SRV records to point to them, using the provided examples. This turned out to be a mistake that you should avoid.

If you are migrating an existing domain, I strongly recommend that you don't update the DNS SRV records at first. You should first create the accounts on the new service, migrate the current users rosters and vcards, and only then switch the DNS records. This should be pretty obvious stuff, but I was eager to move and failed here.

The Process One support personal will accept a SQL dump of your rosters and vcards and load them up on the service at Hosted.IM. I was lucky because I was already using ejabberd with the SQL backend, so I only had to cleanup old accounts, dump the SQL database and send them the file.

This data migration process is unfortunately not documented yet. New users don't even know the possibility exists. I had to ask for it on Twitter to discover the possibility. I also don't know what other formats or other servers export files they support. So check with support before you decide to switch to figure out how you'll migrate the roster information.

As I said, I didn't prepare that part, so I had to scramble to dump the rosters and send them the SQL, so that my users didn't end up with a empty roster. Fortunately the support staff was awesome and I quickly had my SQL dump loaded onto the service.

After this was done, I closed the firewall for my old server C2S ports, and started up my XMPP client. I connected without any problems to the new service.

From start to finish, and even with all the discovery and learning the layout of the administration interface, it took me less than 2 hours to have my service migrated. Pretty good.

I then selected the cheapest plan, at €8/month. The payment system is not clearly explained at the site. It works as a prepaid system: you load your account with €nnn and they remove the amount you owe every month. You also get some bonuses if you load your account with large amounts. After spending 10 minutes wondering where the payment interface was (after you change plans, the interface appears in the account section; while on the free plan, it's not visible - this is suboptimal, a link in the Plans & Pricing tab, or near the costs values in the domain administration tab would be more helpful), I loaded my account with €100 and I got a €8 bonus.

Some services cost extra. For example, using your own SSL certificate is a €2/month, and connectivity to other IM networks costs €4/month. Unfortunately, these extra costs are not described in the service homepage. You have to register for the trial account, and then check the Plans and pricing tab inside your domain management admin page.

Aside: on the service homepage, if you select Plans & Pricing in the navigation toolbar at the top, the javascript scrolls down to the #pricing section but fails to update the page location, which makes sharing the direct link harder.

And this is were we stand right now: accounts, rosters and vcards were migrated successfully, and I was able to load my account with enough money to last me for a bit less than a year (I was not counting on the extra €2/month for the certificate).

The next step is creating my own certificate. This part of the process could be improved a lot. For technical reasons, you have to upload the certificate private key. But if that is a requirement (and it is, I understand that part pretty well), then they could save a lot of work to their clients if they just took care of all that: add the option to generate the key on their servers, and send me the Certificate Request file so that I can request a certificate from a CA that supports XMPP certificates (which are slightly different from HTTPS certificates, they require an extra extension). It would be helpful if they recommended a couple of CAs providing the service, but they do not.

In the past, the XMPP Software Foundation provided a free service of XMPP Certificates, but it was shutdown sometime ago. According to their page above, you can buy a XMPP certificate from StartSSL CA, but I'm still figuring out how to do this. It should be straightforward, the same process as a HTTPS server, and I'll update this article after I've successfully done it, but the StartSSL site lacks XMPP-specific information.

After I have that part done, I'll move my external components. Some of them are sub-domains of my main simplicidade.org domain and those should be straightforward.

Others use a completely different domain name. This is an unusual setup. I basically used my own ejabberd server as a XMPP router for some domains. I connected those domains as external components, and pointed the S2S DNS records to the ejabberd server.

The Hosted.IM does not yet support this mode of operation, but I again asked Mickael about it, and this unorthodox configuration should be supported very soon. Awesome.

All in all, a pretty smooth ride.

One particular problem I face from time to time is related to helpful APIs. Those tend to help the programmer complete its task by adding common fallback processing paths. It usually goes something like this: the API designer convinces himself that X this is the most common operation, so when user doesn't do it, we do it for them. The programmers have less to write because the API will do the right thing™.

My problem with those APIs is that they tend to fail silently. The program keeps on going, using the default processing path, and eventually the programmer lack of decision will bite him with an exception or a core dump (pick your poison).

I tend to prefer non-helpful APIs: if the programmer should make a decision, force him to do so and die as soon as possible if he doesn't, compile time if you have that concept and if you can make it happen then.

The decision the programmer needs to make could even be to do the default processing path, but he must make one.

It is a little bit more code to write, but it pays of enormously in long-term readability by eliminating the hidden default behavior. Every action is explicit, so a new programmer looking at the code (eg. you, the same one who wrote it the first time, only 6 months later) will need less background information to understand it.

So help you API users. Die often, die early, be explicit and stop helping lazy programmers.

From that, an explosive number of articles sprang up about Apple doing a integrated TV set, a smart(er) TV. The term disruption has been in almost all of those articles. I'm sure everybody would love to see what Steve Jobs and his team would come up on that space. I have to wonder though, if this was not the last grand-master chess player move against his competitors.

Wouldn't it be very Jobsian to send all of Apple competitors in one last great wild goose chase?

To explain my rationale (and tackle that overused "disruption" term), let me start with a question: please point me to a recent (lets say since 2000) disruption that took hold of a market (and not just a figure of imagination of pageview-whoring tech pundit), that was not accompanied by, or based on, an explosive growth of that market? I'll wait.

You see, I could not remember of any. I'm sure there are some, and I'm hoping that someone will point them out to me, but I though about this for a day and I could not remember a single one.

The last two great disruptions, smart phones and tablets, are still seeing close to 100% growth rates. Maybe Square is sitting on one, with their gorgeous simple payment interface, but they are US-only for now, and they still use VISA and MasterCard, so I don't think they even qualify as disruptive, given that they are not defying the incumbent.

But back to the disruption thing. I don't think it is possible to be disruptive on the TV space.

First, you don't have explosive growth on TV set sales. And without explosive growth, the incumbent has time to adjust to the changes introduced by the contender.

Second, the TV space is filled with big players, with huge financial interests at stake. They form a interlinked web, that is much stronger than any of them standing alone. Apple may have (had?) an influence on Disney, but those are just one single player.

Another example: look back to this 18-week periond in the spring in the USA, with the NFL labour dispute. The entire sport is supported with over one billion dollars TV revenues. I don't know about you but I would be weary of telling 32 times 50 players, each one showing about 2 square meters of hardened muscle, that their business got disrupted (apologies to all NFL players that might be reading this and take it the wrong way, I'm a big fan, and watch your games regularly; I'm sure most of you are cuddly, and friendly to kids and animals).

Third, the fact that the current TV sets are passive. It allows TV programmers to play with the ratings and follow a hit TV show with a less popular one in the hope that the audience is too lazy to change channels (Nielsen tells us, yes, they are...). And this helps them push more ads. The entire business and operation is hoping that the user turns the box on and leaves the remote alone, dozing over average (at best) quality content while you wait for your hit show.

So no, I think disruptive is asking too much. I believe subversive is the best the internet technocracy can expect in the near future.

They will slowly take over functions that current contenders in the same space are doing. For example, the AppleTV with AirPlay, could take over casual gaming from the Wii. Both Google's and Apple's offers, specially teaming up with Netflix, will slowly kill the DVD/BluRay player and the cable operator pay-per-view services, not because they are better at providing movies and TV shows on the TV set (at best they slightly better, and only because the cable operators have been slow to or bad at reacting to the threat; technically the operators are better placed to provide that service), but because they can integrate better and faster with other methods of consumption, from tablets to smart phones to the Web.

And that is what the current TV add-ons boxes (like Google's and Apple's current offerings) are already doing, starting with the basics, moving into games, and soon small apps controlled via smart phones or tablets (and solving the user interface problem in the process).

Given all this, we can allow a different interpretation on Steve Jobs quote: maybe he hasn't cracked the problem, maybe he just finally understood the size of it, and that it would take a long time and effort to slowly chip away at the wall.

And maybe, just maybe, with that final hurrah, he could just send his competitors into a all out race against that same big strong wall.

(skip the rambling, take me to the summary: tl;dr)

I think nobody can dispute that this is a good thing. HTTPS will provide an almost (if only Certification Authorities (CA) weren't so prone to hacks...) perfect assurance that you were indeed talking to the correct server. HTTPS with CAs is a solution to man-in-the-middle attacks.

We can only hope that this redirect will eventually become the default behavior, because regular and non-authenticated users will still be using the plain HTTP search, and need to specifically ask for the https://www.google.com/ secure site to be safer. But given Google's goal of securing personalized search results I think is acceptable to limit this to logged in users only for now, given that they are the ones with access to the personalized search results. Mind you that this might also be a way for Google to load test their HTTPS setup.

But starting on the third paragraph things take a turn into a new reality: Google will no longer provide the query information to the site you click on the organic search results page. There is no direct explanation on that article why they will start (or stop) doing this.

Before going on further, lets make one thing clear: there is no technical limitation that prevents Google to forward the individual query. In fact this is working right now. When you use the HTTPS version of Google Search, the URLs are rewritten in a way that they go through a Google jump, and from that they are redirected to the final page. But this Google jump page is hosted on a plain HTTP site so the final redirect to your page includes this as the Referrer, with the full query information. Google could just keep using this scheme to provide sites with the information they want.

What they do tell you is that if you click on a ad from the results page, the query information does get sent to the destination site.

This might seem as a double standard of behavior, but sites that appear on organic search results and sites that appear because they payed Google are clearly two different populations and so they can have different treatments from Google if Google chooses to do so. After all, when I pay to get my site on the search result page, I have a right to get all the information about why my ad was shown there. I payed for that right.

It seems we sometimes forget that Google is a company, and as all companies, the goal is to make a profit selling goods. The goods Google sells are your searches. The fact that Google has become so large and useful as to be considered indispensable, and that some small changes in its behavior can make or destroy entire business models, is something that we should be aware, and if possible try to fight against. Its never good to have so much power in the hands of a private company, be it Google, Apple, Microsoft or Facebook. If Microsoft was investigated in the 90's, I will not be surprised to see the same happen to Google in the next decade.

The point is: Google is deliberately choosing, as is their right to do so, to stop sending valuable information to outsiders for free. And I think this will move them closer to a investigation by some governments.

One of the business models that is threatened by this change is the land of search engine optimization. As we can expect, they are livid, and react accordingly. There is this particular article that caught my attention: Google Puts A Price On Privacy.

The premise of the article is that with this change, Google will only share is search data if you pay them.

Well, duh...

Google is a company, a for-profit company, of course it wants to get payed.

The second paragraph is even better:

Google’s a big company that goes after revenue in a variety of ways some critics feel put users second.

(emphasis mine). Now this is laughable. There are two different types of people that classify as users:

• users of the search engine that use it to find stuff: we get a free (and great) service, and in exchange we get to see some ads - I think these are the real users of the service;
• users of the ad system: they pay to show ads of their products on the search result pages - I call these customers of the service.

But no matter which group of users you pick, the change Google announced is good for them.

As users of the service, we get a little bit of extra privacy from third parties: strangers on WiFi networks, governments with intent to control their citizens civil rights, you name it, it gets better.

For customers of the service, they get more value from their payments because now they will have exclusive access to valuable information.

So who are those users who loose? I think they are two other groups, and only one of them really looses a lot. The first is all of us who have a site and were used to receive the search query information. Some of us used it for actual useful features on our site, others would only see them via the analytics system they were using. So we loose a little. We can still get to part of the analytics information via Google Webmaster Tool if we want for some reason optimize our search ranking, but not adjust in real-time to our users queries.

People who make money re-selling the search queries that leaked from Google Search are the really big losers, though. They would use the search queries to target ads on their own sites. And it was a good business.

And I believe the article was written by someone in the second group, or someone who writes for the second group.

The claims are all interesting:

• "[Google is ] perfectly happy to sell out privacy": not news, the moment ads started showing up on Google Search results, we knew they were selling our privacy - we are the product. Besides, explain why someone who makes a living on the search keywords that leak via the referrer is not violating our privacy also?
• "...blocking is a pesky side effect to a real privacy enhancement Google made": no, blocking is not a side effect. Today you can search on HTTPS Google search site and still get the information on your non-HTTPS site. Blocking is a deliberate decision by Google, don't blame technology about this one;
• "Google could have pushed many sites across the web to become more secure themselves": this is based on the logic that if all sites were HTTPS then the referrer information would still flow to the sites. Google is actively trying to change people over to SSL-based communications, to the point that they designed a protocol that requires it (SPDY) and bundled that protocol into their own browser. I don't see how you can accuse Google of not doing plenty for the improvement of the security on the web. Even this change is a clear prod in that direction - as the author notes, if you want to keep receiving the data, switch to HTTPS;
• "Google could have [...] its default search [redirected] to be secure. [...] using Google’s own figures, [logged in users filter will ] protect less than 10% of Google.com searchers": true, but I believe that this is an engineering decision - lets try with 10%, and if all goes well, switch everybody over.

I guess that if my business was based on referrer information I would be pissed today, and even I don't depend on them, I really do hope that Google keeps sending those lovely q's query parameters in the referrers.

But most of this article facts and complaints are at best self-serving, if not just plain wrong.

My biggest doubt is the SPDY angle. If you use a recent version of Chrome, your access to Google is done using SPDY, which includes TLS security by default. So even if you just type www.google.com you are using a secured version of the search engine, but still the following click will be plain old HTTP with the full referrer information. Will they change this too? What are the rules for a SPDY to HTTP transition? Should they be the same as a HTTPS to HTTP transition?

Bottom line: I appreciate the default redirect to HTTPS, and I agree that Google has a right to provide a better service to paying customers. But I don't believe they will be able to sustain this policy of not forwarding search keywords. Not only its petty, it might also trigger a government investigation on the subject, something that I think they would not want.

(image from Ars Technica article about the life of Steve Jobs)

I leave you with my all-time favorite ad. When I learned of this and whenever I think of his impact on our lives, I think of this ad.

I've read about games on the iPad/iPhone using AirPlay to use the HDTV as a display device, and it is a wonderful idea: you sit down with one of your favorite games in front of the big screen, flip a switch on your phone and now you can use the entire screen as a controller. Kind of like a Wii U, I guess.

But there is another scenario I like even better. Let me tell you the story of the "iPad for the kids" here at home.

A couple of months back, when the iPad2 was first announced, we decided not to buy any portable game consoles anymore. The games on those might be better, but the economics are wrong: you pay less for the console, but after the 4th or 5th game, you are already into iPad2 price points.

On the other hand, if you get an iPad2, you can have games between €0.79 and €5 a pop, even without counting the enormous amount of free games. So we bought an iPad2 for the kids and created an account with an allowance on the App Store. Entertainment problem solved. And as a bonus feature, we can load great education apps like Mathboard or Algebra Touch too.

So we have the iPad, mainly for games, and AirPlay (as soon as the Apple TV arrives) to make use of the HDTV. But I'm skeptical about real-time games via AirPlay. I very much doubt that the delay will go unnoticed.

Fortunately there is a better option, one that also might promote the Apple TV from a hobby to another billion dollar business for Apple.

You see, the new Apple TV is powered by an ARM processor and iOS, exactly the same combo the other iOS devices use.

If you could download the games over the air from your iPad/iPhone, they would run on the Apple TV, using the HDTV as display, and use your iOS portable device as a remote control, then your Apple TV becomes a games console too, with no display delay at all.

It would not be powerful as a XBox, PS3, or a Wii, but it would not matter: it would have a killer price point, and you could start a game on the iPad, arrive at home, keep playing on the HDTV, and switch back to the iPad later.

Now that would be awesome.

But today, on a Mac OS X 10.5.8 system, I noticed something similar:

$ curl -LO http://xrl.us/cpanm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   185  100   185    0     0    301      0 --:--:-- --:--:-- --:--:--   301
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Now, you can work around it quickly if you add --insecure to that command line, but that feels dirty.

I checked on my other Mac, running 10.6.6, and I had no problems. The curl version in Leopard is just too old, and lacks some of the new certification authorities:

### 10.5.8
$ curl --version 
curl 7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3

### 10.6.6
$ curl --version 
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3

If you check curl SSL certs documentation you'll see that, yes 7.16 is very old and until 7.18.0, the bundled CA file is "severely outdated".

The solution is to update the bundled CA file. First we need to find it and curl-config --ca is your friend:

$ curl-config --ca
/usr/share/curl/curl-ca-bundle.crt

I though "I'll just copy the file from 10.6.6..." and be done with it, but no such file is present on my Snow Leopard. I assume that curl uses the system keychain in 10.6, but I don't know for sure.

So we do it the hard way. I'm just interested on accessing GitHub without problems so I checked the CA GitHub uses and downloaded the CA chain from them: you'll need both the "DigiCert High Assurance EV Root CA" and the "DigiCert High Assurance EV CA-1".

Put those file in a directory, open a terminal to it and type:

cat /usr/share/curl/curl-ca-bundle.crt \
    DigiCertHighAssuranceEVRootCA.crt \
    DigiCertHighAssunceEVCA-1.crt \
    >> curl-ca-bundle-new.crt

To test this new CA bundle you can use:

curl --cacert curl-ca-bundle-new.crt -LO http://xrl.us/cpanm

and the download should work perfectly.

To make this change more permanent you can replace the original curl-ca-bundle-new.crt with this commands:

sudo cp /usr/share/curl/curl-ca-bundle.crt /usr/share/curl/curl-ca-bundle.crt-backup
sudo cp curl-ca-bundle-new.crt /usr/share/curl/curl-ca-bundle.crt
sudo chmod 644 /usr/share/curl/curl-ca-bundle.crt
sudo chown root:wheel /usr/share/curl/curl-ca-bundle.crt

And that's it! All your HTTPS downloads from GitHub should now be CA errors free, including clones using https:// URLs.

Although I had this problem on a Mac, the solution should work as well with other operating systems, as soon as you find the location of the curl-ca-bundle.crt file.

My answer was always "I'll do that when I turn 40".

Well, I turned 40 today.

I suppose now I should do all that, but last December I noticed that I had put a lot of extra weight. I was always a 72/75kg person, but since my surgery I had gained an extra 10kg, and was dancing around the 82kg mark.

Lets just say I was not happy... So, given that my deadline was coming up, I decided to start early. January 1st I had 82 and I wanted to reach 40 at 70kg.

I started eating better (no fast food, no soft drinks, less bread, more soup and fruit) and more often (4 snacks, two in the morning, two in the afternoon). I made my membership at the gym something more than a cost, and started going 3 times a week for a mix of cardio and weight training. And, and this is probably the best thing I ever did, I started running.

I used one of those 5K programs, that promise you to be running 5km in 9 weeks. At first I did that at the gym. If I had problems at least I was not alone in the street. Then on March 7, I did my first 5km run outside and became addicted to RunKeeper. You can follow my improvements on my profile page, but right now I'm running 10k in 1:10:00/1:15:00 3 to 4 times a week.

And I feel great.

The result of all this can be seen here, courtesy of my Withings body scale:

I didn't hit 70kg, I'm exactly at 71kg today. I blame a couple of weeks in late April where I was stuck in the 72kg mark because I tripped and hurt a knee, and couldn't run, but I'm on the right path and I'm sure before the month is over I'll be clearly in the 70's.

So, after loosing 11kg (going on 12) how do I feel? Great. I have more energy and stamina and that shows where its most important: with my two kids (third on the way by the way, expect delivery late December).

Hitting 40 is sort of a mark on everybody's life. I can honestly say that I don't feel different than yesterday, but I'm also certain that I'm much better physically than I've been for a long long time.

So, have a great day. I know I will.

Twitter will live on as a mainstream platform, used by countless millions of regular every-day non-technical users for which these new rules are meaningless. They will suffer and question why their previous favorite client no longer works, while downloaded the latest official Twitter abortion of an interface. They will complain about it (their $Deity given Internet right), but will move on, and keep on using it.

But those of us more technically-able will probably look for a new partner, while we lick the stab wounds inflicted but our former mistress.

As always, when something wrong happens to a centralized system, the geek-gene arouses and looks for the greener pastures of decentralization. "Would it be good to have a Twitter without the central service?"

Apart of the love/hate relationship with centralized systems, of which the current cloud-fad is one mighty engine, I have to say that the only word who comes to mind about all this is dejavú

Almost-optimal solutions for this problem exist for decades. The first one I remember is good old multicast. Assuming unlimited address space, assign each person his own multicast group and let the routers take care of it. Its unpractical of course, but the architecture of any solution that emerges will have a lot of common points with this old codger.

But more recently some systems tried to tackle the problem. My pet favorite, XMPP, is one of them. We have per-user publish/subscribe that can be used to implement such a system. And we have a large network of XMPP servers already.

But XMPP has two drawbacks. The first is that the explosion of a message to all his subscribers is done at the source server. This means that when the Bieber opens his mouth and bleats, his own server would have to send his pantomime individually to all members of his flock. So even if a thousand of them were on the Google Talk network, instead of a single message, the Google servers would receive a thousand messages. I would be remiss if I didn't mention that there is a deferred extension to tackle this problem, but few in the community showed interest on it.

The second drawback is more perception than technical: over the years I felt that people don't see XMPP as Webby enough. Its a strange, dark technology, that few understand, and the passion for it by those who do, makes those who don't stagger.

But the pretty solution of a multicast network to deliver information, even in the XMPP approximation of a network of centralized servers, where each one acts as a aggregator for a part of the community, providing distinct local services for local people, has one glaring drawback: there is no privacy.

If I delegate one instance of my message to a Google aggregator that serves their clients, I can only hope that they will do the right thing and deliver said message to the right people.

If this requirement is soft for public messages, it becomes very strong for protected streams. If you want some assurances that your protected tweets don't end up in the wrong place (and I set aside the discussion about what kind of guarantees we have about that from Twitter itself in its quest to monetize your content) then you can't use this remote-server-side explosion but have to go back to exploding your messages individually to each of your approved subscribers.

There is an extra cost on resources for this kind of privacy.

So if you want to try and tackle the distributed Twitter problem, please remember that it is one of the most discussed architectures in our Internet history. It has been done before (or at least tried) several times, some with success, others less so. It doesn't matter if the next real-time micro-blogging tool will be based on Usenet, SMTP, RSS, XMPP, or the latest fad-du-jour. The basic problems are the same ones our fore- fathers tried to solve with multi-cast.

As I said, dejavú.

post scriptum rant, or why I need to support <aside> on my CSS: But is a decentralized system with some central aggregation nodes based on trusts workable? History has something to reminds us again: the usenet was the elite messaging tool amongst geeks in the early 90's (oh how I miss thee, trn). Then the AOL horde invaded and it became mainstream. The geek clan left, and later the usenet collapsed with spam, and now is a strange BitTorrent look-a-like.

The clan moved over to blogs and RSS. And those too were colonized, although not as thoroughly because that RSS thing is still a bit on the complicated side.

So the social networks were born, to hide all those pesky RSS feeds, add a dash of real-time, and place a pretty face on top of all of it.

So: do all decentralized systems evolve into spammer havens or centralized silos by sheer force of capitalism? A topic for another day.

The source of this convention started with Bill Joy, the creator of vi. The reason for this particular set of keys is because Bill Joy used a ADM-3A terminal, and the keyboard has cursor arrows printed on those keys.

You can see a big picture of the ADM-3A on Wikipedia. Zoom in on the keyboard to see the arrows.

Nothing like a bit of geek archeology...

This topic of Android fragmentation always triggers a StarWars-based scene: Ogly Wan Genoobly sitting calmly inside his speeder, waves a hand at an iTropper: "These are not the the fragmented droids you are looking for".

One of the roles nginx is performing is to act as a frontend server for Perl-based Web apps. This apps run under Starman or one of the other PSGI-compatible servers.

The simplest configuration block you need to use is this:

server {
    server_name my.site.com;

    location / {
        proxy_pass http://127.0.0.1:9999/;
    }
}

This will proxy the request to your Web app running at 127.0.0.1, port 9999.

The problem is that the Host HTTP header is rewritten to 127.0.0.1. In fact, the header is rewritten to whatever value you use as the host in the URL given to the proxy_pass directive.

With a bit of /etc/hosts manipulation you can use stuff like:

proxy_pass http://my_web_app.production:9999/;

But this makes the deployment process more complex because you need to keep /etc/hosts in sync with all your apps.

A cleaner alternative is to force the original Host header to be sent to the backend server, like this:

server {
    server_name my.site.com;

    location / {
        proxy_pass http://127.0.0.1:9999/;
        proxy_set_header Host $http_host;
    }
}

With this configuration, the backend server will receive whatever Host HTTP header your browser sent that matched this server block, which is particularly useful if you use a regular expression to match host names.

Or if you have multiple server_name's but want your backend to receive a fixed host name, use:

proxy_set_header Host "my.forced.hostname";

Either way, its much simpler to keep frontend and backend servers in sync, with regard to virtual host names.

They are the ones I always get back too, and their place on the iPhone is clearly assured.

Here are some tips to help you play them:

• use the speed multiplier from the start. Always play at the top speed: you can grow your score quickly, and when you eventually get into trouble, you can slow down the pace a bit to solve the problem;
• try to use the same path for ships of the same speed, like a train;
• Harbor 3D lets you can anchor a ship by tapping it. Harbor Master doesn't. In the latter you can create a circle in front of the terminal and keep a bunch of ships there, doing laps. This is the same as the planes do while they wait to land on busy airports, they keep descending in circles.
• Harbor 3D ports are very wide, so aim each ship to a different part of the port: you can get 2 or even 3 ships to dock at the same time;
• unless a new ship is very near another when he shows up, wait until you see the indication of the next ship before moving it. When you direct a ship to his destination, your hand covers most of the screen, and you'll miss the notification. You would then need to quickly determine where he is. If you wait to see that before directing the new ship, you know were you must move too immediately after. You might even decide on a different port based on the side of the screen the next ship will arrive.

Any tips for me?

On one side we had Google explaining that the word "fragmentation" is meaningless, and on the other we had some app developers (and some Apple fanboys, but I tend to ignore them) explaining what the problem is for them, specially developers porting apps from iOS.

The good

The fact that you have so much devices, with so much choice, is a very good thing for the end-user, because he can pick up the one with the best set of features to match his requirements. Also, choice and competition leads to better prices.

Compare that to the version 4 of the iPhone: you get four combinations between two sizes of internal memory, 16 and 32Gb, and two mobile technologies, GSM/UMTS and CDMA. Everything else is exactly the same. Even compared to previous versions of the iPhone, the screen keeps the same aspect ratio, with a very simple multiplier between resolutions, and new sensors.

I believe that this fragmentation is a good thing. It allows end-users to test and choose the best format, and eventually certain categories will naturally survive the selection process, and dominate all the others. It might take some time, but eventually it will happen.

The bad

The bad part of all this is that application developers have much more work developing in a fragmented environment. With time developers will get better at dealing with this, and better tools will be created to help them out.

Even in the latest Android 3.0, we can see an acknowledgement from Google about this situation, with the improvements to the APIs, and in particular with the new Fragments API.

But its not all doom and gloom. The experience Android developers will gain dealing with this, is something that iOS developers don't have, but might need eventually.

Its a very complex proposition to assume that the iOS-powered devices will keep evolving with easy changes to adapt to. If or when a new iPad resolution is released, will it really be twice the current 1024x768 resolution? It would be awesome for sure, but its a huge number of pixels to get right in the LCD production.

So its very likely to see the same screen resolution fragmentation that we now have on Android appear on future iOS devices.

The ugly

What I don't really understand is Google's attempts at denying the situation. I assume that they feel the mention of the "fragmentation" word is seen as something terrible and they must fight it any way they can.

But this places them in the untenable position of convincing us that there is no fragmentation, that black is white. Its a lost argument.

It would be much better to embrace it, and work really hard to make sure the persons who have to deal with them, the developers, have the best tools to do it. Maybe this is more APIs, or maybe just better testing resources, specially a hardware lab with lots of Android devices, and a new process to define test suites, sending them to the lab, and have your app tested on them.

Whatever it is, Google would be better off stopping denying the Android fragmentation, and openly embrace it as something good.

Why do browsers have to distribute codecs at all? Most operating systems already have a video API: Windows has it, Mac OS X has it, and Linux has several although GStreamer seems to be (from a Linux Desktop outsider point of view) pretty standard.

So why not use them? After all, I can buy the codecs I want if I wanted it for them, and have all my apps gain support for it.

But browsers don't seem to take that route, they don't seem to use the operating system at all for a lot of things, specially codecs. They bundle their own. This even applies to internet plugins now, with Chrome bundling Flash Player.

These moves are being sold as either "support the open X movement" (replace X with a proprietary technology), or "guaranteeing updates in the name of security".

Now, nobody in their right mind can argue against this, right? I mean having open alternatives to proprietary/closed versions of software parts is a good thing, it is a required thing, and we should thank Google for investing on some of them like WebM. Competition is required for the advancement of technology in general.

And we also know that most users don't update their systems, but we all get to pay the price of increasing levels of spam and virus.

But doing all of this inside the browser also goes against personal choice and it is a egoistic self-centered position. With the classic operating system middleware, I can install components at that level that all apps benefit. I can update my Flash Player and all apps, either browsers or RSS readers with Web views built-in, will benefit. Or if I install a WebM codec for my operating system, all the apps that use the OS video layer immediately get support for it. Not so with the current browser approach. We read stuff like "we update your Flash plugin so that you are always safe" and then I use another app, running a old version of the plugin and bam, there goes my "security".

I partially understand the browser vendors position: they are trying to solve their own support problem. Is much easier to support a application (specially a browser that is installed millions of time each day) if they release their entire world in a single package. No external dependency hell. I do understand this.

But with this decision from browser vendors to lower their support costs, we users loose our freedom of choice. Browser vendors (all of them) end up imposing their views (specially political views in terms of "open" vs "closed" technologies) on all of us. The tool that would enable us to exit the local App chains and level the playing field for all participants on the Open Web is now a political carrot.

I want more open systems, I really do. But I'm not willing to sacrifice my freedom of choice to get them. Freedom of choice means that I can choose proprietary, closed technologies if I want to, either because they are a better fit for my use case, or because I'm used to them, or because they are standard in my current environment.

My initial title for this article was "The tyranny of open" but I don't think we are at that point yet. We are surely on the path though.

I don't know the original source for the text. If you do, please leave a comment so that I can credit it properly. Update: source, Boletim de Julho de 2009, da Acreditar. Thx to José Augusto Macedo do Couto for the tip.

A importância de saber chegar a casa

Mário Cordeiro, pediatra, disse na semana passada numa conferência organizada pelo Departamento de Assuntos Sociais e Culturais da Câmara Municipal de Oeiras, que muitas birras e até problemas mais graves poderiam ser evitados se os pais conseguissem largar tudo quando chegam a casa para se dedicarem inteiramente aos seus filhos durante dez minutos.

Ao fim do dia os filhos têm tantas saudades dos pais e têm uma expectativa tão grande em relação ao momento da sua chegada a casa que bastava chegar, largar a pasta e o telemóvel e ficar exclusivamente disponível para eles, para os saciar.

Passados dez minutos eles próprios deixam os pais naturalmente e voltam para as suas brincadeiras. Estes dez minutos de atenção exclusiva servem para os tranquilizar, para eles sentirem que os pais também morrem de saudades deles e que são uma prioridade absoluta na sua vida. Claro que os dez minutos podem ser estendidos ou até encurtados conforme as circunstâncias do momento ou de cada dia.

A ideia é que haja um tempo suficiente e de grande qualidade para estar com os filhos e dedicar-lhes toda a atenção. Por incrível que pareça, esta atitude de largar tudo e desligar o telemóvel tem efeitos imediatos e facilmente verificáveis no dia-a-dia.

Todos os pais sabem por experiência própria que o cansaço do fim de dia, os nervos e stress acumulados e ainda a falta de atenção ou disponibilidade para estar com os filhos, dão origem a uma espiral negativa de sentimentos, impaciências e birras.

Por outras palavras, uma criança que espera pelos pais o dia inteiro e, quando os vê chegar, não os sente disponíveis para ela, acaba fatalmente por chamar a sua atenção da pior forma.

Por tudo isto e pelo que fica dito no início sobre a importância fundamental que os pais-homem têm no desenvolvimento dos seus filhos, é bom não perder de vista os timings e perceber que está nas nossas mãos fazer o tempo correr a nosso favor.

Some big examples are Gillete, in the business of selling razor blades; HP, in the business of selling ink and toner cartridges; Or even McDonalds, which lives of the rents it collects from his tenants, the restaurant franchisors.

I'm a very happy user of the Dropbox service, but I don't get their business model.

Their core technology is the sync engine, the best I ever used, not because of its speed, but because of the transparency and simplicity of the entire process: you drop your files in that folder, and all your devices will get it.

But all their marketing for both free and payed accounts is based on storage space, a commodity.

I pay for a lot of online services. Some, like Github and Flickr, I don't even needed to do so, because their free versions would be enough for my usage. But I keep paying because I want those companies that are providing me with a valued service to prosper and succeed.

I cannot do the same with Dropbox, although I probably use them more than any other service. Their focus on pricing their solution on something that is a commodity, and not their core technology, devalues my opinion on them, and prevent me from giving them my money.

I would gladly pay them for the sync service itself, without any server-side storage, because thats the part that nobody else was able to provide to me yet, in the same simple and efficient way.

But I can't: for some reason, they sell you storage, and you get this sync service for free. The only thing I can do is to use their votebox and hope to see a plan focus on sync, not storage space.

I really don't get their business model...

In the past few weeks the hot topic seems to be multi-tasking, or to be more precise, the ability to have several third-party apps running at the same time.

I do have an opinion on this but you should note that I don't own any iPhone OS-using devices. No iPhone, no iPod Touch, and still no iPad. So my view on this is purely theoretical.

Usually Apple doesn't give you what you want, or think you want. They look at a problem, and derive their true essence and solve that.

This 2nd derivative approach to problem solving allows for the creation of tools that work outside their primary target area.

I do think that the same will happen with multi-tasking. I do not expect to have the ability to run several UI-based applications at the same time. Personally I don't see a need for that if switching between apps is fast and easy, and each application keeps its state so that you can go back to where you were.

But there are some things that you want to keep doing when your app looses focus. A Pandora client would like to keep streaming music, a IM client would like to keep a connection open, and be notified when there is activity.

Those things are possible if Apple adopts one of Android features: services. Each app can delegate the small parts that it wants to keep on doing to Apple frameworks, and small non-GUI applications.

For example, we can imagine background services for:

• do a HTTP request, triggered by a push notification: the content would be stored locally on the device, and the app would have access to those blobs when it starts up the next time. Optionally a small non-GUI application could be asked to run to analyze those blobs and provide alerts, badges, or other types of on-the-device notifications;
• keep a TCP connection open: a SOCKS-style proxy that keeps the device-to-network TCP connection open, and accepts app-to-proxy connections and splices them together. You IM connection can ask this proxy to keep the persistent connection open to the IM server, and when there is activity, start the small non-UI helper to decide what to do next.

There are of course other type of background use cases. You might want to have your GPS position to be sent on a regular basis to a web service, or to be recorded locally. Again, this can be done by a trusted Apple-provided service, that POST's your current position to a app-provided URL, or stores the coordinates on a local log file that the app can later read.

In short, I do expect Apple to give developers ways to do more background tasks, but I don't expect multiple UI-based apps running at the same time.

I guess we will know more in a few hours.

The E5 project is going great but there is so much to do that I haven't had the chance to write here, something that I miss.

But the craziness is winding down and I expect to be back in a couple of weeks.

And it doesn't enter sleep mode. It just goes puf, and powers off.

Searching around for similar reports, it seems that my battery is kaput. Either a cell is damaged, or the battery is starting to swell. I'm guessing its the first one, since I don't notice any swelling.

This is the third battery I've burned through with this laptop. The first one lasted from May 2006 up-to sometime early 2008. The second one lasted a couple of months, then started to swell. And now this one, since Mon Oct 27 23:20:47 2008 until today.

Now you might wonder how can I know the exact date and time that I got the new battery. The answer is simple. Before I switched to the new battery, I wrote this script, x-apple-log-battery-status, a quick and very dirty way of keeping track of all the stats about my battery.

I run it every 300 seconds. Given that the script logs the data to disk, it has to spin-up the disk and therefore influence the data collected, but I use my laptop intensely anyway. It rarely sleeps.

So I've recorded 53605 samples of my battery life. The first record is this:

t:1225149647 # Mon Oct 27 23:20:47 2008
fully_charged: No
is_charging: Yes
external_connected: Yes
cycle_count: 1
time_remaining: 38
max_capacity: 6869
voltage: 12485
current_capacity: 6516
design_capacity: 6300
temperature: 3082

The last one so far:

t:1266581334 # Fri Feb 19 12:08:54 2010
fully_charged: Yes
is_charging: No
external_connected: Yes
cycle_count: 561
time_remaining: 25013
max_capacity: 3367
voltage: 12459
current_capacity: 3335
design_capacity: 6300
temperature: 2944

As you can see, I've still have about 50% of the original capacity, but with 561 cycles.

The entire file is here: battery_log.txt.gz, 736Kb of gzipped data. You can do whatever you want with it, I only ask that if you do something interesting, send me a link.

Someday I'll dump this into a spreadsheet and do some pretty graphics.

Gone are the days that you could actually believe the "Don't be evil" corporate motto. The reality came crushing down, and Buzz is only one of the last proofs that "Don't be evil" is much different proposition than "Do good".

I'm fortunate enough not to use a lot of Google services, but still the amount of information they have on me is scary. I knew some of it, and I was aware of the social graph efforts going on inside Google, in particular Brad Fitzpatrick work when he first joined The New Borg.

But the path they choose with Buzz, assuming that my address book or even my chat roster, has anything to do with my social network, is wrong on so many levels.

For the geeks at Google, it appears that one step in their quest to harness all of the human knowledge is to treat humans as a flat concept, without any levels in their social relations. I no longer have best friends, whom I trust almost everything about me but that rarely exchange electronic messages, but I keep in my address book, with some guy that asked me a couple of questions some months ago. All of them get put in the same basket. So the same level of trust and sharing privileges is attached to all of them by default.

What bothers me the most is that now, until I can find replacements for the services that I find useful at Google (address book and mail), I will have to create new empty random Google account, and use each one for parts of the data I have in there.

The Future

During my college days, I was specially interested on distributed systems. I intuitively believed that they were the only solution for resilient scalable network-oriented services. What I couldn't imagine was that just how much important they are for the little level of privacy that I expect from online services.

So one of my goals for 2010 is to split my online information across several online services. Also, study options to move them back to my control.

As for Google, I guess it is time to start working on active Google blocking tools. There is definitively a war coming.

Congrats to all the Saints fans, excellent second half, with a awesome starting play.

Drew Brees MVP. Pity about not breaking the pass record. Tied with Brady at 32.

But for those that share my mother tongue, here it is.

(para quem leu a versão na mailing-list, tem apenas pequenas alterações de estilo, e um "Resumindo:" lá pelo meio)

A discussão que está a acontecer sobre o iPad, o efeito do modelo fechado e controlado pela Apple, e o que ele trás à criatividade e ao espirito hacker (no sentido cosquinhas, curioso, e não no sentido cracker) está mesmo muito interessante.

Acho que a questão da criatividade, que pelo que percebi começa com o post do Tim Bray que termina com a frase "For creative people, this device is nothing" é uma falsa questão.

Sinceramente, o bias natural do Tim Bray para a plataforma Android baralhou-lhe as ideias temporariamente. Basta apontar o trabalho do nosso Jorge Colombo com as capas da New Yorker como contra exemplo do seu argumento.

Mas mesmo que ele estivesse a dizer criativo como programador/developer, seria falso. Ninguém programa para o Android no próprio device. Estes devices não são computadores que possam existir no vácuo, são apêndices aos computadores que vamos ter em casa, ou no emprego.

O único contexto em que se pode ver o iPhone/iPad como limitadores de criatividade é nas politicas de aprovação da AppStore. Tudo o resto está disponível.

A parte de discussão sobre hackers é muito mais perto do meu coração. Tanto quanto pude ver começa com esta frase do Alex Payne:

The thing that bothers me most about the iPad is this: if I had an iPad rather than a real computer as a kid, I’d never be a programmer today.

Apenas é possível concordar com isso se se pensar que o iPad será o nosso único computador, e claramente que não foi desenhado para isso. Mark Pilgrim parece concordar com esta visão negra do futuro dos hackers:

Once upon a time, Apple made the machines that made me who I am. I became who I am by tinkering. Now it seems they’re doing everything in their power to stop my kids from finding that sense of wonder. Apple has declared war on the tinkerers of the world.

O meu percurso (assim como de outros nesta lista, estou seguro) é semelhante ao do Mark: primeiro computador aos 11, começar a programar aos 12, 13 em BASIC e depois Assembler; começar a montar os seus próprios PCs; usar Pascal, C; usar minix, linux porque nos dava o controlo da nossa máquina; o querer saber como funciona, desmanchar, abrir, mudar. No meu caso sempre fugi um pouco da parte electrónica da coisa, não me dou com electrónica, e mais focado na parte de software low-level.

Mas respondendo à pergunta: será que o iPad vai limitar a próxima geração de hackers? Eu podia dizer que não e explicar as minhas razões, mas acho que a Gina Trapani tem uma resposta bem melhor que qq coisa que eu poderia escrever:

Because while we're all ranting about how closed the iPad will be, the jailbreak community is planning competitions to see who can crack it first. The sun isn't setting on tinkerers; their desire to crack things open intensifies when faced with something that's closed by design. The challenge is part of the appeal.

...

I'm of the mind that if someone wants to tinker, they will tinker, period. Because it's in their DNA, not because it's easy, and because by nature, tinkerers don't play by the rules.

A parte que eu posso acrescentar à discussão é esta: em 1991, na Universidade do Minho, a vasta maioria dos alunos não tinha acesso à internet. Não tinham mail, ainda não havia Web nem browsers, começava a aparecer usenet. Nesse tempo, quem andava pelos centros de informática, ou estava lá o menor tempo possível para fazer os seus trabalhos e ir embora, ou então pertencia ao grupo de pessoas que queria perceber tudo, como é que aquilo funcionava, como usar (e abusar) do sistema. Era uma "guerra" contra os administradores de sistema, porque era divertido. O que se fazia era programar, aprender e trocar conhecimento. Não havia outro sitio onde o fazer.

Durante os anos em que o acesso começou a aparecer, 92 e 93, apenas esse grupo lhe prestou atenção. Era uma fonte de conhecimento inesgotável e finalmente disponível. Despertava o nosso sentido de saber mais. Se tivesse de dar um nome a este período, seria a nossa explosão cambriana.

Esse aceso não nos transformou em colectores passivos de informação. Apenas nos deu mais energia para as próximas experiências.

Mas a partir de 94, altura em que os browsers Web começaram a ficar acessíveis às massas, começaram a aparecer uma nova tribo. Passavam horas e horas nos centro de informática a consumir informação, mas sem produzir informação. Lembro-me de discutir isto com a minha tribo e alguém (sinceramente não me lembro do nome dele) dizer "mas isto é o que se espera de pessoas normais". Sem dúvida que o nosso "hubris" tecnológico já estava presente e separava-nos das "pessoas normais" :).

Resumindo: não é possível julgar um device ou um comportamento que esse device facilita da mesma forma pela tribo dos hackers e pelo resto das pessoas normais.

A verdade é que para muitas pessoas, o que é importante não é se o sistema é aberto, às cores, hack-able, ou não. O que interessa é que com ele consigam fazer as tarefas simples do dia-a-dia sem as preocupações que apenas devem (ou deviam) existir na cabeça dos profissionais de IT.

Relembrando o artigo da Gina: ...someone wants to tinker, they will tinker, period. Because it's in their DNA, not because it's easy...

Penso em pessoas como a minha mãe, ou a minha tia, que por mais brilhantes e cultas que sejam (e claramente o são mais do que eu) não se sentem à vontade em usar um desktop clássico (Windows XP, Vista, 7, e Mac OS X) sem se baralharem.

Uma parte do artigo do Fraser Spiers:

Secretly, I suspect, we technologists quite liked the idea that Normals would be dependent on us for our technological shamanism. Those incantations that only we can perform to heal their computers, those oracular proclamations that we make over the future and the blessings we bestow on purchasing choices.

...

Those of us who patiently, day after day, explain to a child or colleague that the reason there's no Print item in the File menu is because, although the Pages document is filling the screen, Finder is actually the frontmost application and it doesn't have any windows open, understand what's happening here.

Ao olhar para os defeitos que a minha tribo atribui a esta nova plataforma, e seguindo um pouco a classificação dos mesmos que o artigo do Luke Wroblewski usa, podemos tentar vê-los do ponto de vista de uma "pessoa normal".

Quando se fala de multitasking, temos de clarificar exactamente o que queremos dizer com isso: correr várias aplicações ou partes delas ao mesmo tempo. De base, tanto o iPhone como o iPad, são um sistema multi-task, e apenas a limitação artificial da Apple em correr apenas uma aplicação ao mesmo tempo nos impede de o fazer.

Acho que é fácil de provar que a maioria das pessoas se baralha com multi-task. Basta pensar em todas as pessoas que já vimos a usar o Windows com todas as apps maximizadas, uma de cada vez. Ou de ver os problemas com o menu File > Print que o Fraser nos fala acima.

É claro que mudar rapidamente entre aplicações é importante, e essas mesmas pessoas o fazem, mas para tal não é necessário que a aplicação destino esteja a correr, desde que a ilusão criada pelo sistema operativo esteja bem feita.

Sem dúvida que várias features que a nossa tribo pensa em colocar nas suas aplicações passam por ter um processo em background a fazer qualquer coisa. Mas até as limitações actuais nas baterias estarem resolvidas, é possível argumentar que esses serviços devem ser feitos no nosso Mac/PC principal, e deixar a esse a tarefa de notificar o device que temos informação importante para ele, com a opção de enviar um link para a mesma informação que o device, transparentemente e sem a intervenção da app, obtem da rede para passar à app na próxima oportunidade.

Existem certas coisas que não são possíveis neste cenario, que normalmente envolvem informação que só o device pode ter (como o Bordalo nos diz, o Android dele adapta-se ao sitio onde está), mas até essas podem aparecer no futuro como um serviço do OS, disponível a todas as aplicações.

Quanto a não ter o plugin de Flash no Mobile Safari, já escrevi hoje sobre isso; resumindo a minha visão da coisa, era importante ter a opção de correr Flash no iPad, mas dispenso claramente de o ter dentro do browser, até porque as utilizações mais práticas que tenho para ele, estão (ou estarão) disponíveis via HTML5.

Todo o processo de aprovação da AppStore, e as barreiras à distribuição de aplicações são aspectos claramente negativos, apesar de dar origem a alguns episódios cómicos dignos de Kafka. Neste aspecto, acho que um compromisso pode surgir em que todas as apps tem de ser assinadas pela Apple (existem vantagens claras nisso), mas a distribuição pode ser via Web.

Uma coisa pouco discutida, é o facto de os i{Pad,Phone}'s não mostrarem o filesystem aos utilizadores. Segundo parece, no iPad, os ficheiros da aplicação ficam guardados numa pasta dentro da aplicação (que pode ser exportada para o Mac/PC via file sharing), de forma a que quando se remove uma App, tudo o que está associado a essa App desapareça (é giro cruzar esta funcionalidade com a alteração do Mac OS X relativamente a deixar cair o suporte a creator codes).

Cada app fica com a responsabilidade de mostrar os seus documentos, e se acham que isso é anormal, é porque não tem prestado atenção ao que o iTunes e iPhoto andam a fazer à anos: a esconder o file system com uma visão do mesmo ajustada à tarefa, ou tipo de documentos, que estão a ser manipulados.

Será que isso escala para muitos documentos? Olhando para o iTunes e iPhoto parece que sim, mas é possível argumentar que só funciona nesses casos porque apenas consumimos os documentos, não os alteramos profundamente, como numa aplicação do iWork.

É sem dúvida uma questão de volume de documentos, e faz mais uma vez lembrar que o iPad não pode ser um computador autónomo, sem o Mac/Pc para o acompanhar.

Mas olhando para as limitações como decisões conscientes de design, é importante ter uma coisa em conta: a nível de hardware, o iPad não nos limita em termos de user-inteface porque todo o seu UI é gerado por software (excepção única à ausência de camara que para mim é ainda hoje um grande mistério). Logo tudo o que nós hoje vemos como limitação, pode ser alterado, ou removido no futuro. Veja-se o que aconteceu com copy-paste nos iPhone's.

Como conclusão: para todos os efeitos, eu coloco-me claramente na tribo dos hackers. Ainda hoje continuo a abrir e fechar computadores como forma de hardware porn, principalmente máquinas antigas; ainda hoje continuo a continuar a ler e comprar livros para perceber como funcionam estes devices que nos intrigam; ainda hoje perco^Winvisto uma data de tempo todos os dias a estudar e a aprender com outros.

Por isso a minha posição sobre a politica da Apple não podia ser diferente daquela que a Gina Trapani tem:

First, know that I fundamentally agree with Alex and Mark: the closed nature of the iPad turns me off, and I wouldn't give one to my kid if I were encouraging her to learn about how computers work.

...

Even though I am critical of the iPad's closed nature and agree with Mark and Alex, I won't go as far as Alex did and say that it represents a dystopian future. I have more faith in our future tinkerers than that.

Mas a verdade é que eu não quero ser apenas hacker, e existem ocasiões em que é muito agradável desligar essa parte da minha vida para me concentrar no que está à minha volta, e nessas alturas, ter um device que me trate abaixo das minhas capacidades para o dominar, é uma trégua simpática.

Acho que a tribo está ao rubro porque este gadjet não é para eles, e isso é algo a que não estamos habituados.

As expected , the iPad will not support Flash, and this brought back all the discussion that went on when the iPhone appeared. I find it interesting that Gina Trapattani is seeing a reduction on Flash-enabled browsers (and thats not counting iPhones, just regular browsers) on her sites. Also Gruber (but he is counting iPhone's) and Alex Bayo.

As a end-user I would not mind see Flash burn in hell. On a regular day, I used to have regular Safari crashes. Since I installed ClickToFlash, I don't see them anymore. Maybe it is related, maybe not, but my suspicion is that the culprit is the Flash plugin.

But there are at least two features that I use several times per day that right now require Flash: multiple file upload, and clipboard access.

The first, multiple file upload, is solved by HTML5. Its in the spec, and implemented in Safari4 and Firefox 3.6+. I think that Chrome 2 already supports this too.

Regarding copy-to-clipboard, the situation is less clear. There is a section in the HTML5 spec about copy-and-paste, but it is inside the drag and drop chapter, and I don't know how it can be used today, if at all.

So the two last remaining useful (as in work-related useful) uses of Flash will soon have HTML5-based replacements, and that means that I will not miss Flash at all.

But that is not to say that Apple is right about this. As the person who bought (or will buy) one of this Apple devices that lack Flash support, it should be my decision to enable or not a technology that for better or for worst is still widely used. I do not want a Flash plugin, but I would not mind a Flash runner app. A dedicated browser, totally written and supported by Adobe, and available for free on the App store. With that, I would have a choice to click the blue legos and see the the Web as Adobe think it should be. (shudders)

First, you can look at the code of AnyEvent::Mojo::Server::Connection: no private Mojo APIs are being used.

They might have been with early releases, but as Sebastian says, I did work with him to improve Mojo so that I didn't have to depend on private APIs. That would be wrong.

In fact, given that I was able to write a AnyEvent-based Mojo server proves that the code was properly structured.

Second, I have no problems with changing state machines. I like state machines and expect them to change. I was eager to see Sebastian add a "pause" state, that would greatly simplify my own work regarding long-pooling. And the first break included changes in the state machine that I fixed on my side.

But the second break was caused not by the introduction of new states, but the abstraction of some code into the new Mojo::IOLoop. After the notification of no changes until 1.0.

Now, I totally accept that Sebastian needs to evolve his code base to support the new stuff he wants to support, like WebSockets. But it was nonetheless new code, and a re-factoring of all the Server interface.

I was pissed at having my code broken in that way after the "no changes" declaration, but only for a couple of days, but, and significantly more importantly, that has nothing to do with my deprecation of AnyEvent::Mojo.

I'm deprecating my module because there is a better way to do it now. I no longer need to keep this module up-to-date, I just have to work with the larger community of the PSGI/Plack, the Perl Web Server. It just makes sense to drop one-off module and switch to a system that is being reused by so many projects.

In a nutshell, I'm deprecating AnyEvent::Mojo because you have better solutions now. Solutions that I will move to and recommend to others. Solutions based on the PSGI/Plack stack, the Perl Web Server.

I am glad to see a clarification about the deprecation policy in the Changes and the http://github.com/kraih/mojo/commit/89fc6baf390d0065ad319abc540dc6a47e0bc812 for the Mojo project, I think it is a big step in the right direction; it can only give assurance to potential users of Mojolicious, and help the project.

I did learn a lot about HTTP that I hadn't pay attention before, the tiny details in the protocol. I do like Mojolicious MVC stuff, I just think that the Plack HTTP stack is better than the Mojo HTTP stack.

And I might get the cake and eat it too. I'll wait for Mojo 1.0 but it should be possible to use Mojolicious, the best part of the project, with Plack.

A long, long time ago...

When I wrote AE::M, I was looking for a way to do long-polling in Perl, with decent performance and cool stuff like epoll/kqueue support. At the time, the only solution might have been POE, but I have other problems with that module, so I decided I wanted a AnyEvent-based solution.

I went looking for a HTTP-stack that I could extend and found Mojo. The code was working and ready in a afternoon, and I was happy.

But then we had the famous 0.991250 release. A lot of backwards incompatible changes, and AE::M broke. I was caught at a bad time and could not update my own module for some weeks. In the Changes file, Sebastian explained why he broke backwards compatibility and that it would be the last time before 1.0; so I finally found the time and updated my code.

CPAN Testers was giving me green lights once more, and I was happy.

Until 2 months later that is, when we got the new Mojo::IOLoop code. AE::M broke again, and its broken since then. The place where I was using AE::M needed some fixes, and the work to adjust my code to the new Mojo is not as simple as it was the first time around.

So I took the dirty way out: included the last pre-Mojo::IOLoop code with my project, implemented the new features, and shipped. Not pretty. At all.

Meanwhile, in a distant planet, miyagawa was busy changing the Perl landscape with regards to HTTP integration. He wrote the PSGI spec, and the Plack toolkit. It is a wonderful piece of work, and the Perl Web Server of choice.

And after some API work, with the help of nothingmuch, it included a very decent support for asynchronous HTTP servers, including all the bits to make long-pooling work right.

Then in December, we got Mojo::Server::PSGI, native support for PSGI inside Mojo.

Next steps

Right now, I'm waiting for the next release of Mojo, and then I'll immediately release a new version of AnyEvent::Mojo, deprecating it.

With the native support for PSGI, there is no reason to maintain the AE::M interface, just use the Plack::Server::AnyEvent (or AnyEvent::HTTPD::PSGI when miyagawa finishes the cleanup he talked about).

But I will update the AE::M one last time to make it compatible with Mojo::IOLoop, as soon as I have the time. I don't want to leave it broken.

In the meantime, if you are using AE::M, I suggest that you bundle an earlier version of Mojo. I'll run a git-bisect to figure out which version is best and update this post.

Lessons learned

Not sure. On one hand, it is a version below 1.0 so we can expect some breakage.

On the other hand, the Changes file mentions that no more backwards incompatible changes would be made to Mojo before 1.0.

I still like the Mojolicious MVC framework and the dispatcher behind it, but I fear the Mojo stack and deprecation policy of the project now.

For now, all my async work is moving to Plack. I still need a asynchronous MVC framework, but I can wait a bit, or even write a very simple one if need be. Maybe steal code that I like from the others.

But I'm back to Catalyst (and I plan to talk about my feelings about the attribute-based dispatcher soon) because I feel safer. A lot safer. I know that there are other people out there that depend on Catalyst every day and in with big sites, much bigger than mine. And I trust the Cat developers not to break my code.

I do hope to see Mojo 1.0 soon, and I hope it goes well for all the gang at #mojo, but I will also look for some form of backwards compatibility policy before I even mention Mojo ever again in my presentations about Perl.

My analysis: I like the price, the battery life and the screen size. I would like to have a webcam for Skype, but that wasn't meant to be. Maybe next time.

I'll buy one, probably the 32G WiFi-only. It will replace my 17" Macbook for the non-work-related tasks that it does now.

I keep my iTunes, iPhoto, and other media stuff on my desktop Mac at work, and I travel around with the Macbook. Its a pain to keep these media libraries synchronized with the laptop, but its painless to do it with the iPad.

For most of my travel, I should be able to just leave the Macbook at home now.

For work related tasks, if I can get a Remote Desktop client on it, I should be able to connect back to the desktop at the office, via my IPSec VPN, and solve small problems.

I find the concept interesting only from a technical point-of-view. All the details about a good real-time web is what passes for tech-porn around here.

But from a human point-of-view, the concept is flawed. Or at least, my own brain finds the concept flawed.

I can't keep up with a constant barage of updates. For quite sometime, I don't have a Twitter client (I only get status updates that contain a link via RSS), I stopped using FriendFeed, and I've disabled or pushed to 8+ hours all of my checks for new mail or new articles on my feed reader. The only exception to the no-real-time updates I have is my instant messenger, and a lonely IRC client on the last virtual desktop of my system.

Not that it isn't helpful sometimes, its just not a good way to consume knowledge and information.

There are two very different concepts here: up-to-date information and real-time awareness.

I want the real-time web to succeed because it creates the necessary infrastructure to have up-to-date information in real-time, but I don't want to have the awareness of those updates shoved at me.

In an ideal future, you would have an application that sits on your mobile device, your smartphone or your tablet, that helps me manage the real-time stream. Some of my friends, Nuno and Pedro, call this application Exocortex.

Your Exocortex would subscribe to all your news feeds, your email, your Twitter/Facebook/Whatever stream. With the real-time web infrastructure in place, it would be constantly being updated with the latest information. He does not interrupt your concentration, he doesn't require your awarness; he just collects, organizes (preferably by learning what you like, and what is important to you), and indexes all. It fetches pages mentioned in the articles that you get and caches them (I find that I often click on the links but rarely follow to the second level).

Most of this happens locally on the device. It is a ecological crime not to use your local CPU. The cloud should not be CPU and storage outsourcer, but a meeting point and backup destination. Not the main point of consumption, but the backup destination. And yes, some business models would stop working.

Then, when you take your device, you get an interface into the latest trends, articles, web pages mentioned, all from the inside of your little information world, powered by your personal social network.

You can be offline and still see the latest information up to the moment you disconnected. Everything is cached locally, and if something is missing, you can pin it for later retrieval. I like to call this "close the laptop and go"-scenario. No need to sync the latest version.

So I welcome the real-time web, not because I want to be bothered every time someone farts on Twitter, but because I just want to take my iPad and go.

Giving thanks, like Christmas, doesn't need a special day, and for us perl hackers our favorite team, the perl-porters, giving thanks couldn't be easier.

So lets all, no matter where you are, run the perlthanks command line today, and celebrate our new schedule for Perl5 releases and happy days and long lives to our pumpkins.

On a more individual note, I can't forget three little big persons: cog, ambs and Magda. YAPC::EU raised the bar on so many levels.

But we don't live without CPAN, and so we acknowledge the PAUSE and CPAN teams, and all individuals that keep the mirror network up and running.

Finally, to those teams that make my day-to-day simpler and fun: the Moose cabal and the Plack gang (with a note of envy for miyagawa productivity...).

To all, a very big Muito Obrigado.

The small work I did was to start a script that converts a directory of files into a RSS or Atom feed.

For now it sits at the app-files2feed repository. I need to fix the last bugs with enclosure support and add documentation before releasing it to CPAN.

Non-perl work sucks.

I've added more commands to test (courtesy of code "borrowed" from SD, and the Launchy gem), and made the test suite more robust in case we don't find a suitable command.

<rant>
I'm amazed that something as simple as opening a URL is such a complicated task on most UNIX-based systems.

I have a single command to use on Mac OS X and on Windows based systems, but there seems to be no standard way of doing this simplest of things on UNIX systems.
</rant>

I need to tweak my server setup to be a little bit faster loading all this javascript, so you might notice a delay before the highlighter does its job.

Before each commit you should run the full test-suite of your code. If your test-suite has grown so much that it takes a long time to run, it makes more sense to run a smaller part of it, and let the continuous integration system (you do have one, right?) report back any problems with your tree later.

At the very least you should make sure that all your modules compile correctly. And that means checking the 00-compile.t test file.

To make the whole process automatic, I use the following pre-commit-hook:

#!/bin/sh

## Set any ENV vars that your test suite requires here

exec prove -l -Q t/00*

I usually write a script called my-git-pre-commit-hook and exec that one from the .git/hook/pre-commit file, this way I can run it by hand if I need to.

I also assume that all tests named 00-* are important enough to run before each commit, which lets me add more specific tests at any point.

This solves half of the problem. I also don't want to keep editing the 00-compile.t file each time I add another module to my project. So I wrote it like this:

#!perl

use strict;
use warnings;
use Test::More;
use Path::Class;
use File::Find;

my $lib = dir('lib')->absolute->resolve;
find({
  bydepth => 1,
  no_chdir => 1,
  wanted => sub {
    my $m = $_;
    return unless $m =~ s/[.]pm$//;

    $m =~ s{^.*/lib/}{};
    $m =~ s{/}{::}g;
    use_ok($m) || BAIL_OUT("***** PROBLEMS LOADING FILE '$m'");
  },
}, $lib);

done_testing();

This code uses File::Find to list all the .pm files in your lib/ directory, and then tries to use them. It also use submodules before using the main module, courtesy of the bydepth switch.

If a problem is found, it immediately bail out of the entire test suite, mentioning the file that has problems. On previous versions it would test all modules and only bail out at the end if any had problems. I found that version less productive because the same problem would be reported multiple times.

This works out fine for my projects. It adds at most a couple of seconds to each commit (in the current project it adds less than 4 seconds) but I find that acceptable.

It would be interesting to write a 00-test-commited-files.pl file that would look at the files being updated by the commit and run the tests that cover them. This could be done with the help from Devel::CoverX::Covered but I haven't done it yet.

I tried several times to enable CPAN::Reporter but I always found that it delayed just enough of my workflow that I found it a nuisance. After each test phase, it would start a SMTP connection and send the report. Those 3 or 4 seconds where a bit too much for me.

After a bit of reading I found a good compromise to report my test runs without affecting the performance. The setup is simple: make CPAN::Reporter write the test results to a directory and create a command to send them later.

To set this up, first you install CPAN::Reporter as usual and then you tweak the configuration to store the reports in a directory. My ~/.cpanreporter/config.ini looks like this:

email_from = "Pedro Melo" 
edit_report = no
send_report=unknown:yes fail:yes pass:yes na:no

transport=File /Users/melo/.cpan/reports

The trick is to use the File transport. You configure it with the directory where the test reports will be stored. In my case I choose /Users/melo/.cpan/reports. You need to make sure that directory exists.

From now on, every time you use the cpan shell to install a module, the test reports will be stored in you test report directory.

The final step is sending them. I wrote a simple script to take care of that, that you can find on my scripts repository: x-perl-send-test-reports (download).

How and when you run it is up to you. You can run it manually from time to time, or from a cron, or use something like folder actions to monitor the directory and start the script whenever a new file is placed there. I run it manually for now.

My command line to use the script is this:

x-perl-send-test-reports         \
    --from melo@simplicidade.org \
    --server smtp.gmail.com      \
    --transport 'Net::SMTP::TLS
         User melo@simplicidade.org
         Password my_password
         Port 587'               \
    ~/.cpan/reports

So one less excuse not to report your test results. If you are not using CPAN::Reporter, start now.

Almost 6 months later I can report that it works great, you don't notice it at all. Strongly recommended, if you need this sort of thing.

But there are no completely secure software-only solutions, and its good to know the limitations, like the "Evil Maid" Attacks on Encrypted Hard Drives.

The comments on the article are also worth a read. There are some proposals in there that might work and defend against this kind of attacks.

A former colleague of mine at PT had a small reporting problem, and he ended up comparing several languages for the job: C, Perl, PHP, and Python.

I was curious about the results, so I took the latest version of the Perl script that he was using and set off to work.

The first thing that you should be aware is where your bottleneck is. Take a look at this small script:

#!/usr/bin/env perl

use strict;
use warnings;

my $lines;
while () {
#  my @fields = split / /;
  $lines++;
}
print "$lines\n";

A basic line counter. Compare it to the system wc:

$ gzcat nfsd.gz | time wc -l
 12236390
       16.86 real        16.24 user         0.43 sys
$ gzcat nfsd.gz | time ./wc_simple.pl 
12236390
       10.13 real         8.51 user         0.84 sys

So, a bit faster than the C version, doing 1.2M lines per second on my laptop.

But if you remove the comment on the split(), we have:

$ gzcat nfsd.gz | time ./wc_simple_with_split.pl
12236390
       228.78 real       224.39 user         2.34 sys

A lot less: 53k lines per second.

So the first bottleneck is the split(). Lets improve on that. After some attempts I came up with this:

my $lines;
while () {
  my ($ts)    = /^(\d+)/gc;
  my ($type)  = /(\w)\sV/gc;
  my ($op)    = /\s(\d\d?)\s\w/gc;
  my ($bytes) = /\w\s(\d+)\s/gc;
  $lines++;
}

I make use of the gc flags to start the next match where the previous one ended. I also take advantage of patterns in the lines that I need to match, like the V in the NFS version.

With this version we get:

$ gzcat nfsd.gz | time ./wc_simple_with_regexps.pl
12236390
      109.62 real       107.42 user         1.55 sys

A bit better: 111k lines per second, a bit over 2x the previous result.

If we apply this gain to the reported times for the Perl script (419m11.267s), we get 200m49.97 which places Perl 1st, even above the C version by a minute or so.

My adjusted version of the stats_basic_optimized.pl script is melo_stats_basic.pl.

This was good enough I think, but I wanted to study the I/O gains that could be made. Stop reading here if you are already bored, it will only get worse. We did manage to get a few more minutes back, though... ;)

So I went after the I/O performance. Fist I wanted to rule out the pipe as the bottleneck.

$ cat /dev/zero | pv | cat > /dev/null
4.61GB 0:00:10 [ 498MB/s]

$ gzcat nfsd.gz | pv | cat > /dev/null 
1GB 0:00:04 [ 242MB/s]

So the pipe is not the bottleneck but we will never reach the full speed, gzcat will be our limitation.

I did try to read the gzip file directly into the Perl script and uncompressing it there, but it was very slow.

So assuming a limit of 242MB/s on the input side, how fast are we chopping lines? The size of the input is 1073743224 bytes, and our simple wc_simple.pl (no split, no regexp) took 10.13 seconds above, so we are chopping the input at a rate of 101MB/s.

So there is room to grow there (101MB/s to 242MB/s). I did some experiments:

$ gunzip nfsd.gz
$ time wc_simple.pl < nfsd
12236390

real    1m23.212s
user    0m7.434s
sys 0m2.166s

Yeah, the file doesn't fit in the cache like the gziped version, so there is real I/O, and the times go through the roof.

There is no point doing experiments with the big nfsd file. All of them will result in real I/O, and that is always slower than memory.

Lets try to do bigger reads and parse the results:

use strict;
use warnings;

my $size = shift || 2 ** 20; ## 1Mb default
my $offset = 0;
my $buf = '';
my $lines = 0;

while () {
  my $n = sysread(\*STDIN, $buf, $size, length($buf));

  while ($buf =~ /.+\n/gc) {
    $lines++;
  }
  last unless $n > 0;

  print "$lines $n\n" unless $lines & 0x1ffff;
  $buf = substr($buf, pos($buf));
}
print "$lines\n";

I tried several block sizes but with my OS the most I could read in one call was 64k. So asking 1MB and getting 64k reads we get:

$ gzcat nfsd.gz | time  ./wc_batch.pl
12236390
    8.37 real         7.52 user         0.58 sys

We get 1.46M lines per second, a 17% improvement. Lets adjust to retrieve our fields. The inner loop becomes:

while ($buf =~ /(.+)\n/gc) {
  $_ = $1;
  my ($ts)    = /^(\d+)/gco;
  my ($type)  = /(\w)\sV/gco;
  my ($op)    = /\s(\d\d?)\s\w/gco;
  my ($bytes) = /\w\s(\d+)\s/gco;

  $lines++;
}

and the runtime:

$ gzcat nfsd.gz | time  ./wc_batch_with_regexps.pl
12236390
   95.58 real        93.21 user         1.14 sys

So from 109.62s to 95.58s, 12% better (or comparing with our baseline wc_simple_with_split.pl at 228.78s, 58% better). Adjusting this to the reported results we would go from 419m11.267s down to 175m5.6842s.

I don't think I can improve on this unless we can have a bigger pipe reads. For example, forcing the reads to 8k:

$ gzcat nfsd.gz | time  ./wc_batch.pl 8192
12236390
       12.08 real        10.93 user         0.70 sys

A lot worse compared with the 8.37s we got with 64k reads. So the size of the pipe is the next factor we could explore, if that is even an option with your kernel.

But I'm happy now.

The next day

Or so I though. First there was doubts that the split() was faster than regexps. I wrote bench_splitters.pl (output on my laptop, download link) to compare split with my regexps. The regexps are a bit over twice as fast, but I found big differences between the Mac OS system perl (5.8.8 on my Leopard OS) and the 5.10.1 that I compiled: system perl was between 20 and 30% faster.

The same bench_splitters.pl gives you the max rate of extraction that you can expect from the global script. I also included timing of the bookkeeping parts of the original script. The only noteworthy detail is the fact than when you hit the second level condition, you pay the price of the modulus operator big time. I also think that something is wrong with the input. Those time stamps don't look like normal second-precision time stamps. They are too big. So I don't know if $ts % 3600 is the proper way to group performance by hour.

Second I wrote a max_line_rate.pl (output on my laptop, download link) that gives you the upper bound on the max rate that you can expect while parsing the required fields. You can run this script, and stop it at any point in time with ctrl-c, and it will print a performance report up to that point. Every 128k lines, a single line performance report is also printed.

You can use this max_line_rate.pl to compare your system perl with the 5.10.1 you compiled. I had much better performance with 5.8.8 in this particular application.

Finally I rewrote the statistics script. I did that to deal with the report that my previous version was consuming 7.5Gb of RAM. The reason is simple enough: I don't have access to the original input, only to a six-line excerpt that was posted. Therefore the regexps I use to extract the required fields might fail.

The new script, fast_stats.pl (output on my laptop, download link), is more robust, and should deal with lines that cannot be parsed: it will print the line that couldn't be matched and ignore it. Also: I've included the output of the ps command at the start and end of both fast_stats.pl and max_line_rate.pl to show that the RSS doesn't change that much.

To compare the original stats_basic_optimized.pl with my fast_stats.pl I wrote a small shell script bench.sh (output on my laptop). The nfsd.gz input file was generated with the build_source_file.pl script with the command:

build_source_file.pl 1073741824 | gzip --best > nfsd.gz

The new fast_stats.pl is almost twice as fast as the old one on my laptop.

On a final note (I wasted too much time already on this...), I'm not out to compare Perl with Python or Ruby or even PHPO. But I would like to know how we measure up against C though. The reason is simple: when Perl programmers feel that something is slow, they turn to C, not another scripting language.

This experiment is mostly to show that writing fast perl will sometimes take you to unexpected paths (like regexps beating a split), and that you should benchmark carefully if performance is critical to you.

It takes a single argument (can be a module, distribution, bundle or author name), checks the CPAN indexes to see which type it is, creates the proper URL for it at search.cpan.org and opens your browser with it.

The last bit, opening a browser with it, is very very immature. Right now it only works on Mac OS X. I'm hopping to get the experience to do it right from the Browser::Open distribution.

If no object is found, sends the user to the generic search interface.

The current hackish implementation can be found on my s_command topic branch (its a topic branch, I will rebase it on occasion onto master).

It does one simple thing: given a $url, it opens the default browser with it.

The difficult part is deciding how to open the "default browser". On Mac OS X, this is easy: just execute the open command.

On Windows, there is a start command that should do the trick, but I'm not a Windows user so I cannot test this. Any Windows users out there that can point me to the relevant information on how to open a URL with a simple command, I would appreciate it.

For Linux, you have too many choices it seems: you could use gnome-open but your user might be using KDE. There is a xdg-open command described at the FreeDesktop site that seems to do what I want. We can always fallback to firefox though. Fragmentation++!

Anyway, you can download the Java 1.6 update for Mac OS X Leopard from the Apple Software site, but its only 64-bit.

I do have a desktop that is 64-biT, unfortunately my Macbook Pro laptop is only a 32-bit Core Duo. Hence, I cannot run the new version of Java.

If you do have a 64-bit CPU, and after you install the update, you might want to switch the default version of Java to 1.6. To do that, the proper way to do that is to run the /Applications/Utilities/Java\ Preferences.app and drag the 1.6 version to the top of the list.

I just wanted to try out Jake... Guess I wont be able to, at least not until I can get my hands on the wife's laptop... hmms... she's sleeping already...

Let me just clarify some behaviors for non-background jobs:

• the worker will execute the job until it finishes even if the client that submitted it dies;
• if the client dies before the job is passed on to a worker, it will be removed and will never execute.

For background jobs, the behavior is different: a client submits a background job, the gearmand daemon queues this (on stable storage if you use the persistent queues of the latest versions of the C implementation), and sends it to a worker as soon as one is available.

This is basic stuff, I'm just making sure we are all on the same page regarding Gearman behavior.

Back to the the unique parameter. You can define a unique key for your jobs. Those keys should be unique per function, so you can have the same key on different functions and they will be two different jobs.

The fun part with unique is that you can have multiple clients listening to status and completion events of the same job if you share the key between them. Say that you start a job with a key alpha. If other clients submit a job with the same key, and (and this is an important) the job is still running, this second client will attach itself to the same job.

To test this I wrote a small shell script worker (save it as slow.sh and make it executable with chmod 755 slow.sh):

#!/bin/sh

echo "Starting a slow worker..." >>/dev/fd/2
for i in 5 4 3 2 1 ; do
  echo $i >>/dev/fd/2
  sleep 1
done
echo done $PID

This worker does nothing expect print to STDERR a count, and then sends to the client a done string with the worker process ID. But it takes 5 seconds to run so it allows us some time to switch between windows.

After you start your gearmand server (I used gearmand -vv just to have some debug) you can start a worker process like this:

gearman -w -f slow ./slow.sh

This worker registers the slow function and will execute the slow.sh per job.

Now open two new terminal windows and type on each one:

gearman -f slow -u alpha -s

This will submit a non-background job for the slow function, with the unique key alpha. The -s means that we won't be sending any data.

You'll see one execution of the worker, and then both clients will output something like this:

done 57998

You can experiment, and attach the second client only half-way through the worker run. Or you can stop the worker, start both clients, and then start the worker. The result will be the same: both clients receive the output for the same job.

This is a very very cool feature, and can be used easily for slow processing inside a web request.

First, if a web request requires a slow processing phase, we store all the relevant data and submit a background job to do the processing with some random key. The user receives a "Processing page" on his browser, that includes this key.

A small javascript program using AJAX, connects to our long-pooling server and submits a non-background job to the same function and using the key. This second client request will now wait until the processing is done, and can even receive status updates from the worker and send them back to the browser.

This attach-to-job-using-key is very reliable. I've tested several combinations with and without workers running, strange order of job submission, adding multiple clients to the same job, and all of them work as expected.

The only real problem with this is that the clients need to use the same API that is used to submit a job. Instead of a new API, like ATTACH_JOB for example, you use the same SUBMIT_JOB API that you use for new jobs. This works fine, until your second client attaches with a key for a job that has already ended. The gearmand server will fail to find it, and will dutifully create a new job.

My current workaround for this is to send a specific payload on the attach requests, to signal the worker that this is a attach request and not a new job. The worker, if he detects this signal, just ends the processing. For example, if your jobs require payload data, you can use an empty data field as the flag.

This is not optimal of course, you would be waking up workers just to return immediately, but it would work.

If you have multiple gearmand servers, you need to make sure that the clients that will be attaching to a job use the same server. A solution would be to pass the server identification (IP or even better hash-of-IP) along with the key.

The traditional way of doing this is have the workers store the completed result in a database somewhere. The solution presented here does not invalidate that, it only provides a very light notification of completion to background jobs. It beats pooling the database to see if the background job is completed.

What he wants is a way to tell CPAN this: "install the latest versions of my dependencies".

His solution wont work unfortunately. The code that he gives us will prevent the Task:: module from being installed but it will not guarantee that the latest version of the prereqs will be installed in the following runs.

the reason is simple: if you don't ask for a specific version of your prereqs, CPAN will accept any version, so it will only install each prereq once, the first time.

The perfect solution would be to create a marker on each prereq that would tell the CPAN tool chain that you want the latest version. This does not exist yet. You could probably standardize on version -1 meaning the last one (on a twisted parallel with the last index of Perl lists), but its all speculation. Its just not supported yet.

The next best thing would be to include the version required on each of your prereqs, and keep those values up-to-date to the latest available on CPAN whenever you rebuild your package.

This is a half-way solution. It wont guarantee the latest version at the time your package is installed but it would make sure you get the latest version at the time your package was built on your system before uploading to PAUSE.

This is actually a good compromise given that you are probably listing the versions of the prereqs that you tested your package with on your system.

And, better yet, this half-solution can be automatized. I wrote a Dist::Zilla plugin to do just that. The code is very simple and you can just adapt this into a Module::Install plugin or whatever you use to build your packages.

You can find the code for Dist::Zilla::Plugin::LatestPrereqs at my Github repository for Dist::Zilla tools. Its not on CPAN yet, and for now there are no plans to publish it. The reason is simple: it requires a small patch to the core Dist::Zilla. The patch is a single commit that you can find on Dist::Zilla fork. I've asked Ricardo Signes to accept the patch. If he likes the code, I'll release my plugin after the next Dist::Zilla release.

If you look at the LatestPrereqs code, you'll notice that it is very simple, but it does load the CPAN package and that is a big one. You could write this code directly on your Makefile.PL and have the very latest versions of your prereqs at install time, but this would assume that the system as a properly configured CPAN.

That was a risk that I'm not willing to take on my distributions. If you do it that way, ping me. I would like to follow your module CPAN Testers feed for a while.

Back to Marcel post, if you do need to prevent the install phase for your distribution, then I've also uploaded a Dist::Zilla plugin to do just that: Dist::Zilla::Plugin::MakeMaker::SkipInstall. It might be handy sometimes.

This was a long time coming unfortunately, and I accumulated FAIL test reports on CPANTS, but its here now.

Given that it uses the latest Mojo release, it supports HTTP keep-alive and pipelining, chunked-encoding and 100-Continue requests.

Although the test suite passes, I'm not fully confident on the pipelining code. My next step is to write a client with a slow network reader to exercise some corner cases of that part of the code. In particular, I'm concerned about the interaction of Mojo pipeline code with my request pause functionality that I use to implement long-polling servers.

It could be argued that long-polling and pipeline don't mix, but I think that the pause functionality could also be used on regular requests. For example, if one request needs an answer from a memcached server, the handler can start the memcached GET, pause the Mojo transaction, and resume it when the memcached response arrives. While that is going on, the server should be able to keep writing out previous requests, and reading the next ones.

Maybe thats a bit extreme, but I do hope to have this working, for complex pipelining situations.

But I'm a lazy bastard, and the lack of a bash completion script was getting on my nerves.

So here: you can get the new gh-completion.bash script from my repository. I've sent a pull request to Chris Wanstrath (defunkt)

Please note that this is still work-in-progress.

Right now, most of the command and options have completion, but I'm still trying to understand certain operations (like pull-request and track) that I never use. I don't really understand what they do, so I have no way to create a decent completion rule for them.

But using this bash completion is totally safe... Until you hit the ENTER key, that is.

On a regular basis, it indexes all our users, courses, teachers and other important tables. It works great, low install barrier, low maintenance, and it is very very fast. Perfect.

One of the problems that we found, that limited the usefulness of the full-text search engine, is that a lot of our text has accents, and it would be better to ignore those. Also we don't need case-sensitive-ness.

So I needed to generate a charset_table map, what Sphinx uses to normalize the text that you give him to index.

And being a (very) lazy person, I prefer to write a Perl script to do it. The result is the x-sphinx-charset-generator, now part of my script stash.

It takes an optional parameter, the charset that you are using on your text defaulting to 'utf8', the loose version of UTF-8, and generates a charset_table for the most common accented characters, mapping them to the lower-case version of the same letter without the accent.

I've only include the common Portuguese characters. Patches accepted for others characters that you might need.

The only part that I don't really like is that I need to apply the same logic to cleanup the strings that users use to search. I would prefer to have a module that would take the characters that I want to allow as valid, and have that module provide the charset_table and a function to clean search inputs. Interesting, but for now this will solve the important 80% of the problem.

The usual command you would use is something like this:

gunzip -c db-backup.sql.gz | mysql -udevuser -ppass db_dev

If your dump is big this can take a while and you wont have a clue about what is happening.

Instead, install pipe viewer and do:

gunzip -c db-backup.sql.gz | pv | mysql -udevuser -ppass db_dev

and you get a nice speed meter. For even better results:

dump=db-backup.sql.gz
size=`gunzip --list $dump | perl -ne 'print "$1\n" if /\d+\s+(\d+)\s+\d/'`
gunzip -c $dump | pv -s $size | mysql -udevuser -ppass db_env

and you'll get a speed meter and an ETA. The second command will get the uncompressed size of the dump and use that to teach pv how much data to expect.

Pipe viewer rockz.

At the time, we used HTML::Mason 1.05, and only after 1.30-ish (when the internal buffering changes introduced in the 1.10 release where reverted) did we upgraded to something more recent. To get an idea of the size of this Mason project, the current sites have a little over 450 different components across 7 sites (different layouts but same content) and 1 management site. A lot of those components are no longer in use, and where left there due to bad VCS practices. I would estimate about 200 actual useful components. And it makes heavy use of autohandlers, dhandlers, and multiple component roots, to implement site inheritance.

The E5 template discussion is also still on my mind, without a clear winner yet.

And finally, last week I read Jonathan Swartz article about what Mason 2.0 would look like.

So I took the time last night to re-evaluate Mason. As most powerful tools (and be sure that it is a very powerful tool), the problem with Mason is that you can easlly make a big mess of things. More than with other solutions like Catalyst and Mojo, that provide you with a clear separation of Controller, Model and View, Mason make is very easy to mix the three. You could end up with a lot of logic that should be in your models inside your templates.

But it has several advantages:

• its easy to start and add a new page. Just create the file and start typing: no need to jump between controller and template, and a restart (this alone makes for speedy development);
• it has decent wrapper functionality for skinning: autohandlers are great;
• the multiple component roots logic is very powerful, and its used both during the dispatch phase and component calls;
• the view logic is Perl: no need to learn a new language and be exasperated with their limitations like TT.

There are several downsides of course: for one, the split of Controller/View of modern frameworks allows you to reuse controller logic with multiple views. For example, you could output HTML, JSON and XML with the same controller code.

But the biggest downside is this: deployment is a bitch.

For production environments, deployment usually means mod_perl but I find FastCGI easier to deploy now-a-days. Yet, this option is only briefly mentioned on the MasonHQ site.

So I created a small experimental project (you can find all the files at the exp-mason-fcgi project on GitHub). It has a FastCGI startup script to power two virtual hosts. Each one shares a master component root, and has a local per-site component root to override the master site behavior when needed.

The setup works just fine under nginx+FastCGI (partial nginx.conf included), but I did get into some trouble. Mason usually delegates some stuff to Apache and without his big daddy around, he can get lost.

The first problem is directory index files. When you request a directory, Apache will help Mason out and point it to the proper index.html file. Without Apache, request to http://your-fastcgi-mason-site/ will just fail, because Mason cannot find the component for /. It has no logic to map / into /index.html for example.

You could implement this with a global dhandler, and it is probably the best solution, because it can also deal with 404 situations (another one that Apache could cleanup after).

I have a proof of concept hack in the repo that mimics the Apache DirectoryIndex directive. It is a hack, it should be in the Interp.pm and not in the Request. I'll clean it up later. But it does work, and it might be useful in some scenarios.

This patch makes a / request work just fine.

The second problem I have with Mason is the order of evaluation of templates. The current order for a request to /index.html is /autohandler which calls $m->call_next and that calls the /index.html component.

This makes it hard to influence the wrapper with content generated by the /index.html component.

The solution was to create a new HTML::Mason::Request method called scall_next(). It merges the $m->call_next() and the $m->scomp() calls into one, and allows me to use it in a autohandler like this. The $m->scall_next() will call the next component, get the generated HTML, and only then generate the HTML wrapper.

The last piece of the puzzle is a way for the /index.html and the parent autohandler to communicate, for example, to pass along the title for the page.

The Mason-recommended way is to use $m->notes() API, similar to the Catalyst stash concept. It works very well, but I prefer to take advantage of the fact that all components live inside the HTML::Mason::Commands namespace and just declare a shared %stash there and clean it up per request. With this, its easy to implement dynamic page titles.

All in all, I keep coming back to Mason. I do like most of what it provides, and for quick sites, it beats all the other alternatives in Perl-land.

A couple of months ago there was some discussion about a option to give web developers that was FTP friendly, in that you could upload your pages to a server and it would just work. Mason is the closest that Perl has to that goal.

But the deployment must be made simpler, and that I one of the things that Mason 2.0 should focus on: make it simpler to deploy.

So my laundry list for Mason 2.0 (and most of them can be implemented on 1.x):

• FastCGI support out-of-the-box;
• support for directory index files (without using dhandlers);
• $m->scall_next;
• better hooks for debugging and 404 errors.

I don't know. I'm strongly considering go Mason all the way for E5, and if that goes forward, I guess I'll have to write these four pieces myself.

I used the following test case to figure it out:

use Test::More tests => 1;
use Encode qw( encode decode );

sub u8l1 {
  return encode('iso-8859-1', @_);
}

my $ola_u8 = decode('utf8', 'Olá');
my $ola_l1 = encode('iso-8859-1', $ola_u8);
is(u8l1($ola_u8), $ola_l1);

The output of prove x.t is this:

t/x.t .. 1/1 
#   Failed test at t/x.t line 12.
#          got: '1'
#     expected: 'Ol?'
# Looks like you failed 1 test of 1.
t/x.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests

The got: '1' had me for quite some time. Until I changed the u8l1() helper to this:

sub u8l1 {
  return encode('iso-8859-1', $_[0]);
}

And it just works.

The problem is the definition of the Encode::encode() function. It has a prototype like this:

sub encode($$;$)

So our @_ is interpreted in scalar context, and so evaluates to the number of parameters, 1.

I don't like it at all because it changes the standard Perl behavior of expanding lists. Its action at the distance. The fact that you cannot pass a single element list is also not mentioned in the documentation.

The only really useful use of Perl prototypes is using a & as the initial char, that allows you to write a function that looks like some built-ins like sort or map, that take a anonymous sub as the first parameter.

I have only one question: who was the monster stupid that cancelled this show? What was he thinking?

One thing that we inherited from log4j was the notion that a message can match multiple loggers in your logging hierarchy.

The logic is simple and explained in detail on a Log::Log4perl FAQ entry. If you write something like this in your logger configuration file:

log4perl.logger.Cat        = ERROR, Screen
log4perl.logger.Cat.Subcat = WARN, Screen

which define two loggers. Cat and Cat.Subcat, the second a subcategory of the first, and then use:

my $logger = get_logger("Cat.Subcat");
$logger->warn("Warning!");

you'll get a duplicate message in your log file because it matches both loggers.

I knew that and I always added a line saying:

log4perl.additivity.Cat.Subcat = 0

that prevented this behavior, but this required a line like that per logger. Pain. Not lazy, at all.

But for some reason (stupidity comes to mind) I didn't read the FAQ completely, because at the end, there is a solution. Just put this in your logger configuration file:

log4perl.oneMessagePerAppender = 1

Bliss, pure bliss.

Mind you that oneMessagePerAppender is not compatible with log4j, something that Log::Log4perl tries very hard to be, and therefore this feature is not documented at all except on this FAQ entry.

But with my hands going about their bussiness, the corner of the paper starts to bend upwards.

The solution is not rocket science. A simple paper clip.

Problem solved.

It might not have a comparison between riding a manta ray or riding a air-powered submersible thingie, but it shouldn't be too bad, given the two names associated with it.