First, you can look at the code of AnyEvent::Mojo::Server::Connection: no private Mojo APIs are being used.

They might have been with early releases, but as Sebastian says, I did work with him to improve Mojo so that I didn't have to depend on private APIs. That would be wrong.

In fact, given that I was able to write a AnyEvent-based Mojo server proves that the code was properly structured.

Second, I have no problems with changing state machines. I like state machines and expect them to change. I was eager to see Sebastian add a "pause" state, that would greatly simplify my own work regarding long-pooling. And the first break included changes in the state machine that I fixed on my side.

But the second break was caused not by the introduction of new states, but the abstraction of some code into the new Mojo::IOLoop. After the notification of no changes until 1.0.

Now, I totally accept that Sebastian needs to evolve his code base to support the new stuff he wants to support, like WebSockets. But it was nonetheless new code, and a re-factoring of all the Server interface.

I was pissed at having my code broken in that way after the "no changes" declaration, but only for a couple of days, but, and significantly more importantly, that has nothing to do with my deprecation of AnyEvent::Mojo.

I'm deprecating my module because there is a better way to do it now. I no longer need to keep this module up-to-date, I just have to work with the larger community of the PSGI/Plack, the Perl Web Server. It just makes sense to drop one-off module and switch to a system that is being reused by so many projects.

In a nutshell, I'm deprecating AnyEvent::Mojo because you have better solutions now. Solutions that I will move to and recommend to others. Solutions based on the PSGI/Plack stack, the Perl Web Server.

I am glad to see a clarification about the deprecation policy in the Changes and the http://github.com/kraih/mojo/commit/89fc6baf390d0065ad319abc540dc6a47e0bc812 for the Mojo project, I think it is a big step in the right direction; it can only give assurance to potential users of Mojolicious, and help the project.

I did learn a lot about HTTP that I hadn't pay attention before, the tiny details in the protocol. I do like Mojolicious MVC stuff, I just think that the Plack HTTP stack is better than the Mojo HTTP stack.

And I might get the cake and eat it too. I'll wait for Mojo 1.0 but it should be possible to use Mojolicious, the best part of the project, with Plack.

A long, long time ago...

When I wrote AE::M, I was looking for a way to do long-polling in Perl, with decent performance and cool stuff like epoll/kqueue support. At the time, the only solution might have been POE, but I have other problems with that module, so I decided I wanted a AnyEvent-based solution.

I went looking for a HTTP-stack that I could extend and found Mojo. The code was working and ready in a afternoon, and I was happy.

But then we had the famous 0.991250 release. A lot of backwards incompatible changes, and AE::M broke. I was caught at a bad time and could not update my own module for some weeks. In the Changes file, Sebastian explained why he broke backwards compatibility and that it would be the last time before 1.0; so I finally found the time and updated my code.

CPAN Testers was giving me green lights once more, and I was happy.

Until 2 months later that is, when we got the new Mojo::IOLoop code. AE::M broke again, and its broken since then. The place where I was using AE::M needed some fixes, and the work to adjust my code to the new Mojo is not as simple as it was the first time around.

So I took the dirty way out: included the last pre-Mojo::IOLoop code with my project, implemented the new features, and shipped. Not pretty. At all.

Meanwhile, in a distant planet, miyagawa was busy changing the Perl landscape with regards to HTTP integration. He wrote the PSGI spec, and the Plack toolkit. It is a wonderful piece of work, and the Perl Web Server of choice.

And after some API work, with the help of nothingmuch, it included a very decent support for asynchronous HTTP servers, including all the bits to make long-pooling work right.

Then in December, we got Mojo::Server::PSGI, native support for PSGI inside Mojo.

Next steps

Right now, I'm waiting for the next release of Mojo, and then I'll immediately release a new version of AnyEvent::Mojo, deprecating it.

With the native support for PSGI, there is no reason to maintain the AE::M interface, just use the Plack::Server::AnyEvent (or AnyEvent::HTTPD::PSGI when miyagawa finishes the cleanup he talked about).

But I will update the AE::M one last time to make it compatible with Mojo::IOLoop, as soon as I have the time. I don't want to leave it broken.

In the meantime, if you are using AE::M, I suggest that you bundle an earlier version of Mojo. I'll run a git-bisect to figure out which version is best and update this post.

Lessons learned

Not sure. On one hand, it is a version below 1.0 so we can expect some breakage.

On the other hand, the Changes file mentions that no more backwards incompatible changes would be made to Mojo before 1.0.

I still like the Mojolicious MVC framework and the dispatcher behind it, but I fear the Mojo stack and deprecation policy of the project now.

For now, all my async work is moving to Plack. I still need a asynchronous MVC framework, but I can wait a bit, or even write a very simple one if need be. Maybe steal code that I like from the others.

But I'm back to Catalyst (and I plan to talk about my feelings about the attribute-based dispatcher soon) because I feel safer. A lot safer. I know that there are other people out there that depend on Catalyst every day and in with big sites, much bigger than mine. And I trust the Cat developers not to break my code.

I do hope to see Mojo 1.0 soon, and I hope it goes well for all the gang at #mojo, but I will also look for some form of backwards compatibility policy before I even mention Mojo ever again in my presentations about Perl.

Giving thanks, like Christmas, doesn't need a special day, and for us perl hackers our favorite team, the perl-porters, giving thanks couldn't be easier.

So lets all, no matter where you are, run the perlthanks command line today, and celebrate our new schedule for Perl5 releases and happy days and long lives to our pumpkins.

On a more individual note, I can't forget three little big persons: cog, ambs and Magda. YAPC::EU raised the bar on so many levels.

But we don't live without CPAN, and so we acknowledge the PAUSE and CPAN teams, and all individuals that keep the mirror network up and running.

Finally, to those teams that make my day-to-day simpler and fun: the Moose cabal and the Plack gang (with a note of envy for miyagawa productivity...).

To all, a very big Muito Obrigado.

The small work I did was to start a script that converts a directory of files into a RSS or Atom feed.

For now it sits at the app-files2feed repository. I need to fix the last bugs with enclosure support and add documentation before releasing it to CPAN.

Non-perl work sucks.

I've added more commands to test (courtesy of code "borrowed" from SD, and the Launchy gem), and made the test suite more robust in case we don't find a suitable command.

<rant>
I'm amazed that something as simple as opening a URL is such a complicated task on most UNIX-based systems.

I have a single command to use on Mac OS X and on Windows based systems, but there seems to be no standard way of doing this simplest of things on UNIX systems.
</rant>

Before each commit you should run the full test-suite of your code. If your test-suite has grown so much that it takes a long time to run, it makes more sense to run a smaller part of it, and let the continuous integration system (you do have one, right?) report back any problems with your tree later.

At the very least you should make sure that all your modules compile correctly. And that means checking the 00-compile.t test file.

To make the whole process automatic, I use the following pre-commit-hook:

#!/bin/sh

## Set any ENV vars that your test suite requires here

exec prove -l -Q t/00*

I usually write a script called my-git-pre-commit-hook and exec that one from the .git/hook/pre-commit file, this way I can run it by hand if I need to.

I also assume that all tests named 00-* are important enough to run before each commit, which lets me add more specific tests at any point.

This solves half of the problem. I also don't want to keep editing the 00-compile.t file each time I add another module to my project. So I wrote it like this:

#!perl

use strict;
use warnings;
use Test::More;
use Path::Class;
use File::Find;

my $lib = dir('lib')->absolute->resolve;
find({
  bydepth => 1,
  no_chdir => 1,
  wanted => sub {
    my $m = $_;
    return unless $m =~ s/[.]pm$//;

    $m =~ s{^.*/lib/}{};
    $m =~ s{/}{::}g;
    use_ok($m) || BAIL_OUT("***** PROBLEMS LOADING FILE '$m'");
  },
}, $lib);

done_testing();

This code uses File::Find to list all the .pm files in your lib/ directory, and then tries to use them. It also use submodules before using the main module, courtesy of the bydepth switch.

If a problem is found, it immediately bail out of the entire test suite, mentioning the file that has problems. On previous versions it would test all modules and only bail out at the end if any had problems. I found that version less productive because the same problem would be reported multiple times.

This works out fine for my projects. It adds at most a couple of seconds to each commit (in the current project it adds less than 4 seconds) but I find that acceptable.

It would be interesting to write a 00-test-commited-files.pl file that would look at the files being updated by the commit and run the tests that cover them. This could be done with the help from Devel::CoverX::Covered but I haven't done it yet.

I tried several times to enable CPAN::Reporter but I always found that it delayed just enough of my workflow that I found it a nuisance. After each test phase, it would start a SMTP connection and send the report. Those 3 or 4 seconds where a bit too much for me.

After a bit of reading I found a good compromise to report my test runs without affecting the performance. The setup is simple: make CPAN::Reporter write the test results to a directory and create a command to send them later.

To set this up, first you install CPAN::Reporter as usual and then you tweak the configuration to store the reports in a directory. My ~/.cpanreporter/config.ini looks like this:

email_from = "Pedro Melo" 
edit_report = no
send_report=unknown:yes fail:yes pass:yes na:no

transport=File /Users/melo/.cpan/reports

The trick is to use the File transport. You configure it with the directory where the test reports will be stored. In my case I choose /Users/melo/.cpan/reports. You need to make sure that directory exists.

From now on, every time you use the cpan shell to install a module, the test reports will be stored in you test report directory.

The final step is sending them. I wrote a simple script to take care of that, that you can find on my scripts repository: x-perl-send-test-reports (download).

How and when you run it is up to you. You can run it manually from time to time, or from a cron, or use something like folder actions to monitor the directory and start the script whenever a new file is placed there. I run it manually for now.

My command line to use the script is this:

x-perl-send-test-reports         \
    --from melo@simplicidade.org \
    --server smtp.gmail.com      \
    --transport 'Net::SMTP::TLS
         User melo@simplicidade.org
         Password my_password
         Port 587'               \
    ~/.cpan/reports

So one less excuse not to report your test results. If you are not using CPAN::Reporter, start now.

A former colleague of mine at PT had a small reporting problem, and he ended up comparing several languages for the job: C, Perl, PHP, and Python.

I was curious about the results, so I took the latest version of the Perl script that he was using and set off to work.

The first thing that you should be aware is where your bottleneck is. Take a look at this small script:

#!/usr/bin/env perl

use strict;
use warnings;

my $lines;
while () {
#  my @fields = split / /;
  $lines++;
}
print "$lines\n";

A basic line counter. Compare it to the system wc:

$ gzcat nfsd.gz | time wc -l
 12236390
       16.86 real        16.24 user         0.43 sys
$ gzcat nfsd.gz | time ./wc_simple.pl 
12236390
       10.13 real         8.51 user         0.84 sys

So, a bit faster than the C version, doing 1.2M lines per second on my laptop.

But if you remove the comment on the split(), we have:

$ gzcat nfsd.gz | time ./wc_simple_with_split.pl
12236390
       228.78 real       224.39 user         2.34 sys

A lot less: 53k lines per second.

So the first bottleneck is the split(). Lets improve on that. After some attempts I came up with this:

my $lines;
while () {
  my ($ts)    = /^(\d+)/gc;
  my ($type)  = /(\w)\sV/gc;
  my ($op)    = /\s(\d\d?)\s\w/gc;
  my ($bytes) = /\w\s(\d+)\s/gc;
  $lines++;
}

I make use of the gc flags to start the next match where the previous one ended. I also take advantage of patterns in the lines that I need to match, like the V in the NFS version.

With this version we get:

$ gzcat nfsd.gz | time ./wc_simple_with_regexps.pl
12236390
      109.62 real       107.42 user         1.55 sys

A bit better: 111k lines per second, a bit over 2x the previous result.

If we apply this gain to the reported times for the Perl script (419m11.267s), we get 200m49.97 which places Perl 1st, even above the C version by a minute or so.

My adjusted version of the stats_basic_optimized.pl script is melo_stats_basic.pl.

This was good enough I think, but I wanted to study the I/O gains that could be made. Stop reading here if you are already bored, it will only get worse. We did manage to get a few more minutes back, though... ;)

So I went after the I/O performance. Fist I wanted to rule out the pipe as the bottleneck.

$ cat /dev/zero | pv | cat > /dev/null
4.61GB 0:00:10 [ 498MB/s]

$ gzcat nfsd.gz | pv | cat > /dev/null 
1GB 0:00:04 [ 242MB/s]

So the pipe is not the bottleneck but we will never reach the full speed, gzcat will be our limitation.

I did try to read the gzip file directly into the Perl script and uncompressing it there, but it was very slow.

So assuming a limit of 242MB/s on the input side, how fast are we chopping lines? The size of the input is 1073743224 bytes, and our simple wc_simple.pl (no split, no regexp) took 10.13 seconds above, so we are chopping the input at a rate of 101MB/s.

So there is room to grow there (101MB/s to 242MB/s). I did some experiments:

$ gunzip nfsd.gz
$ time wc_simple.pl < nfsd
12236390

real    1m23.212s
user    0m7.434s
sys 0m2.166s

Yeah, the file doesn't fit in the cache like the gziped version, so there is real I/O, and the times go through the roof.

There is no point doing experiments with the big nfsd file. All of them will result in real I/O, and that is always slower than memory.

Lets try to do bigger reads and parse the results:

use strict;
use warnings;

my $size = shift || 2 ** 20; ## 1Mb default
my $offset = 0;
my $buf = '';
my $lines = 0;

while () {
  my $n = sysread(\*STDIN, $buf, $size, length($buf));

  while ($buf =~ /.+\n/gc) {
    $lines++;
  }
  last unless $n > 0;

  print "$lines $n\n" unless $lines & 0x1ffff;
  $buf = substr($buf, pos($buf));
}
print "$lines\n";

I tried several block sizes but with my OS the most I could read in one call was 64k. So asking 1MB and getting 64k reads we get:

$ gzcat nfsd.gz | time  ./wc_batch.pl
12236390
    8.37 real         7.52 user         0.58 sys

We get 1.46M lines per second, a 17% improvement. Lets adjust to retrieve our fields. The inner loop becomes:

while ($buf =~ /(.+)\n/gc) {
  $_ = $1;
  my ($ts)    = /^(\d+)/gco;
  my ($type)  = /(\w)\sV/gco;
  my ($op)    = /\s(\d\d?)\s\w/gco;
  my ($bytes) = /\w\s(\d+)\s/gco;

  $lines++;
}

and the runtime:

$ gzcat nfsd.gz | time  ./wc_batch_with_regexps.pl
12236390
   95.58 real        93.21 user         1.14 sys

So from 109.62s to 95.58s, 12% better (or comparing with our baseline wc_simple_with_split.pl at 228.78s, 58% better). Adjusting this to the reported results we would go from 419m11.267s down to 175m5.6842s.

I don't think I can improve on this unless we can have a bigger pipe reads. For example, forcing the reads to 8k:

$ gzcat nfsd.gz | time  ./wc_batch.pl 8192
12236390
       12.08 real        10.93 user         0.70 sys

A lot worse compared with the 8.37s we got with 64k reads. So the size of the pipe is the next factor we could explore, if that is even an option with your kernel.

But I'm happy now.

The next day

Or so I though. First there was doubts that the split() was faster than regexps. I wrote bench_splitters.pl (output on my laptop, download link) to compare split with my regexps. The regexps are a bit over twice as fast, but I found big differences between the Mac OS system perl (5.8.8 on my Leopard OS) and the 5.10.1 that I compiled: system perl was between 20 and 30% faster.

The same bench_splitters.pl gives you the max rate of extraction that you can expect from the global script. I also included timing of the bookkeeping parts of the original script. The only noteworthy detail is the fact than when you hit the second level condition, you pay the price of the modulus operator big time. I also think that something is wrong with the input. Those time stamps don't look like normal second-precision time stamps. They are too big. So I don't know if $ts % 3600 is the proper way to group performance by hour.

Second I wrote a max_line_rate.pl (output on my laptop, download link) that gives you the upper bound on the max rate that you can expect while parsing the required fields. You can run this script, and stop it at any point in time with ctrl-c, and it will print a performance report up to that point. Every 128k lines, a single line performance report is also printed.

You can use this max_line_rate.pl to compare your system perl with the 5.10.1 you compiled. I had much better performance with 5.8.8 in this particular application.

Finally I rewrote the statistics script. I did that to deal with the report that my previous version was consuming 7.5Gb of RAM. The reason is simple enough: I don't have access to the original input, only to a six-line excerpt that was posted. Therefore the regexps I use to extract the required fields might fail.

The new script, fast_stats.pl (output on my laptop, download link), is more robust, and should deal with lines that cannot be parsed: it will print the line that couldn't be matched and ignore it. Also: I've included the output of the ps command at the start and end of both fast_stats.pl and max_line_rate.pl to show that the RSS doesn't change that much.

To compare the original stats_basic_optimized.pl with my fast_stats.pl I wrote a small shell script bench.sh (output on my laptop). The nfsd.gz input file was generated with the build_source_file.pl script with the command:

build_source_file.pl 1073741824 | gzip --best > nfsd.gz

The new fast_stats.pl is almost twice as fast as the old one on my laptop.

On a final note (I wasted too much time already on this...), I'm not out to compare Perl with Python or Ruby or even PHPO. But I would like to know how we measure up against C though. The reason is simple: when Perl programmers feel that something is slow, they turn to C, not another scripting language.

This experiment is mostly to show that writing fast perl will sometimes take you to unexpected paths (like regexps beating a split), and that you should benchmark carefully if performance is critical to you.

It takes a single argument (can be a module, distribution, bundle or author name), checks the CPAN indexes to see which type it is, creates the proper URL for it at search.cpan.org and opens your browser with it.

The last bit, opening a browser with it, is very very immature. Right now it only works on Mac OS X. I'm hopping to get the experience to do it right from the Browser::Open distribution.

If no object is found, sends the user to the generic search interface.

The current hackish implementation can be found on my s_command topic branch (its a topic branch, I will rebase it on occasion onto master).

It does one simple thing: given a $url, it opens the default browser with it.

The difficult part is deciding how to open the "default browser". On Mac OS X, this is easy: just execute the open command.

On Windows, there is a start command that should do the trick, but I'm not a Windows user so I cannot test this. Any Windows users out there that can point me to the relevant information on how to open a URL with a simple command, I would appreciate it.

For Linux, you have too many choices it seems: you could use gnome-open but your user might be using KDE. There is a xdg-open command described at the FreeDesktop site that seems to do what I want. We can always fallback to firefox though. Fragmentation++!

What he wants is a way to tell CPAN this: "install the latest versions of my dependencies".

His solution wont work unfortunately. The code that he gives us will prevent the Task:: module from being installed but it will not guarantee that the latest version of the prereqs will be installed in the following runs.

the reason is simple: if you don't ask for a specific version of your prereqs, CPAN will accept any version, so it will only install each prereq once, the first time.

The perfect solution would be to create a marker on each prereq that would tell the CPAN tool chain that you want the latest version. This does not exist yet. You could probably standardize on version -1 meaning the last one (on a twisted parallel with the last index of Perl lists), but its all speculation. Its just not supported yet.

The next best thing would be to include the version required on each of your prereqs, and keep those values up-to-date to the latest available on CPAN whenever you rebuild your package.

This is a half-way solution. It wont guarantee the latest version at the time your package is installed but it would make sure you get the latest version at the time your package was built on your system before uploading to PAUSE.

This is actually a good compromise given that you are probably listing the versions of the prereqs that you tested your package with on your system.

And, better yet, this half-solution can be automatized. I wrote a Dist::Zilla plugin to do just that. The code is very simple and you can just adapt this into a Module::Install plugin or whatever you use to build your packages.

You can find the code for Dist::Zilla::Plugin::LatestPrereqs at my Github repository for Dist::Zilla tools. Its not on CPAN yet, and for now there are no plans to publish it. The reason is simple: it requires a small patch to the core Dist::Zilla. The patch is a single commit that you can find on Dist::Zilla fork. I've asked Ricardo Signes to accept the patch. If he likes the code, I'll release my plugin after the next Dist::Zilla release.

If you look at the LatestPrereqs code, you'll notice that it is very simple, but it does load the CPAN package and that is a big one. You could write this code directly on your Makefile.PL and have the very latest versions of your prereqs at install time, but this would assume that the system as a properly configured CPAN.

That was a risk that I'm not willing to take on my distributions. If you do it that way, ping me. I would like to follow your module CPAN Testers feed for a while.

Back to Marcel post, if you do need to prevent the install phase for your distribution, then I've also uploaded a Dist::Zilla plugin to do just that: Dist::Zilla::Plugin::MakeMaker::SkipInstall. It might be handy sometimes.

This was a long time coming unfortunately, and I accumulated FAIL test reports on CPANTS, but its here now.

Given that it uses the latest Mojo release, it supports HTTP keep-alive and pipelining, chunked-encoding and 100-Continue requests.

Although the test suite passes, I'm not fully confident on the pipelining code. My next step is to write a client with a slow network reader to exercise some corner cases of that part of the code. In particular, I'm concerned about the interaction of Mojo pipeline code with my request pause functionality that I use to implement long-polling servers.

It could be argued that long-polling and pipeline don't mix, but I think that the pause functionality could also be used on regular requests. For example, if one request needs an answer from a memcached server, the handler can start the memcached GET, pause the Mojo transaction, and resume it when the memcached response arrives. While that is going on, the server should be able to keep writing out previous requests, and reading the next ones.

Maybe thats a bit extreme, but I do hope to have this working, for complex pipelining situations.

On a regular basis, it indexes all our users, courses, teachers and other important tables. It works great, low install barrier, low maintenance, and it is very very fast. Perfect.

One of the problems that we found, that limited the usefulness of the full-text search engine, is that a lot of our text has accents, and it would be better to ignore those. Also we don't need case-sensitive-ness.

So I needed to generate a charset_table map, what Sphinx uses to normalize the text that you give him to index.

And being a (very) lazy person, I prefer to write a Perl script to do it. The result is the x-sphinx-charset-generator, now part of my script stash.

It takes an optional parameter, the charset that you are using on your text defaulting to 'utf8', the loose version of UTF-8, and generates a charset_table for the most common accented characters, mapping them to the lower-case version of the same letter without the accent.

I've only include the common Portuguese characters. Patches accepted for others characters that you might need.

The only part that I don't really like is that I need to apply the same logic to cleanup the strings that users use to search. I would prefer to have a module that would take the characters that I want to allow as valid, and have that module provide the charset_table and a function to clean search inputs. Interesting, but for now this will solve the important 80% of the problem.

At the time, we used HTML::Mason 1.05, and only after 1.30-ish (when the internal buffering changes introduced in the 1.10 release where reverted) did we upgraded to something more recent. To get an idea of the size of this Mason project, the current sites have a little over 450 different components across 7 sites (different layouts but same content) and 1 management site. A lot of those components are no longer in use, and where left there due to bad VCS practices. I would estimate about 200 actual useful components. And it makes heavy use of autohandlers, dhandlers, and multiple component roots, to implement site inheritance.

The E5 template discussion is also still on my mind, without a clear winner yet.

And finally, last week I read Jonathan Swartz article about what Mason 2.0 would look like.

So I took the time last night to re-evaluate Mason. As most powerful tools (and be sure that it is a very powerful tool), the problem with Mason is that you can easlly make a big mess of things. More than with other solutions like Catalyst and Mojo, that provide you with a clear separation of Controller, Model and View, Mason make is very easy to mix the three. You could end up with a lot of logic that should be in your models inside your templates.

But it has several advantages:

• its easy to start and add a new page. Just create the file and start typing: no need to jump between controller and template, and a restart (this alone makes for speedy development);
• it has decent wrapper functionality for skinning: autohandlers are great;
• the multiple component roots logic is very powerful, and its used both during the dispatch phase and component calls;
• the view logic is Perl: no need to learn a new language and be exasperated with their limitations like TT.

There are several downsides of course: for one, the split of Controller/View of modern frameworks allows you to reuse controller logic with multiple views. For example, you could output HTML, JSON and XML with the same controller code.

But the biggest downside is this: deployment is a bitch.

For production environments, deployment usually means mod_perl but I find FastCGI easier to deploy now-a-days. Yet, this option is only briefly mentioned on the MasonHQ site.

So I created a small experimental project (you can find all the files at the exp-mason-fcgi project on GitHub). It has a FastCGI startup script to power two virtual hosts. Each one shares a master component root, and has a local per-site component root to override the master site behavior when needed.

The setup works just fine under nginx+FastCGI (partial nginx.conf included), but I did get into some trouble. Mason usually delegates some stuff to Apache and without his big daddy around, he can get lost.

The first problem is directory index files. When you request a directory, Apache will help Mason out and point it to the proper index.html file. Without Apache, request to http://your-fastcgi-mason-site/ will just fail, because Mason cannot find the component for /. It has no logic to map / into /index.html for example.

You could implement this with a global dhandler, and it is probably the best solution, because it can also deal with 404 situations (another one that Apache could cleanup after).

I have a proof of concept hack in the repo that mimics the Apache DirectoryIndex directive. It is a hack, it should be in the Interp.pm and not in the Request. I'll clean it up later. But it does work, and it might be useful in some scenarios.

This patch makes a / request work just fine.

The second problem I have with Mason is the order of evaluation of templates. The current order for a request to /index.html is /autohandler which calls $m->call_next and that calls the /index.html component.

This makes it hard to influence the wrapper with content generated by the /index.html component.

The solution was to create a new HTML::Mason::Request method called scall_next(). It merges the $m->call_next() and the $m->scomp() calls into one, and allows me to use it in a autohandler like this. The $m->scall_next() will call the next component, get the generated HTML, and only then generate the HTML wrapper.

The last piece of the puzzle is a way for the /index.html and the parent autohandler to communicate, for example, to pass along the title for the page.

The Mason-recommended way is to use $m->notes() API, similar to the Catalyst stash concept. It works very well, but I prefer to take advantage of the fact that all components live inside the HTML::Mason::Commands namespace and just declare a shared %stash there and clean it up per request. With this, its easy to implement dynamic page titles.

All in all, I keep coming back to Mason. I do like most of what it provides, and for quick sites, it beats all the other alternatives in Perl-land.

A couple of months ago there was some discussion about a option to give web developers that was FTP friendly, in that you could upload your pages to a server and it would just work. Mason is the closest that Perl has to that goal.

But the deployment must be made simpler, and that I one of the things that Mason 2.0 should focus on: make it simpler to deploy.

So my laundry list for Mason 2.0 (and most of them can be implemented on 1.x):

• FastCGI support out-of-the-box;
• support for directory index files (without using dhandlers);
• $m->scall_next;
• better hooks for debugging and 404 errors.

I don't know. I'm strongly considering go Mason all the way for E5, and if that goes forward, I guess I'll have to write these four pieces myself.

I used the following test case to figure it out:

use Test::More tests => 1;
use Encode qw( encode decode );

sub u8l1 {
  return encode('iso-8859-1', @_);
}

my $ola_u8 = decode('utf8', 'Olá');
my $ola_l1 = encode('iso-8859-1', $ola_u8);
is(u8l1($ola_u8), $ola_l1);

The output of prove x.t is this:

t/x.t .. 1/1 
#   Failed test at t/x.t line 12.
#          got: '1'
#     expected: 'Ol?'
# Looks like you failed 1 test of 1.
t/x.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests

The got: '1' had me for quite some time. Until I changed the u8l1() helper to this:

sub u8l1 {
  return encode('iso-8859-1', $_[0]);
}

And it just works.

The problem is the definition of the Encode::encode() function. It has a prototype like this:

sub encode($$;$)

So our @_ is interpreted in scalar context, and so evaluates to the number of parameters, 1.

I don't like it at all because it changes the standard Perl behavior of expanding lists. Its action at the distance. The fact that you cannot pass a single element list is also not mentioned in the documentation.

The only really useful use of Perl prototypes is using a & as the initial char, that allows you to write a function that looks like some built-ins like sort or map, that take a anonymous sub as the first parameter.

One thing that we inherited from log4j was the notion that a message can match multiple loggers in your logging hierarchy.

The logic is simple and explained in detail on a Log::Log4perl FAQ entry. If you write something like this in your logger configuration file:

log4perl.logger.Cat        = ERROR, Screen
log4perl.logger.Cat.Subcat = WARN, Screen

which define two loggers. Cat and Cat.Subcat, the second a subcategory of the first, and then use:

my $logger = get_logger("Cat.Subcat");
$logger->warn("Warning!");

you'll get a duplicate message in your log file because it matches both loggers.

I knew that and I always added a line saying:

log4perl.additivity.Cat.Subcat = 0

that prevented this behavior, but this required a line like that per logger. Pain. Not lazy, at all.

But for some reason (stupidity comes to mind) I didn't read the FAQ completely, because at the end, there is a solution. Just put this in your logger configuration file:

log4perl.oneMessagePerAppender = 1

Bliss, pure bliss.

Mind you that oneMessagePerAppender is not compatible with log4j, something that Log::Log4perl tries very hard to be, and therefore this feature is not documented at all except on this FAQ entry.

This takes a lot of time, but I had most of the process already in auto-pilot.

But still it was a hack, so I decided to take the opportunity of the 5.10.1 release and make something more pretty and reliable.

The result is my Perl bootstrap repo.

There are two scripts. The first, bootstrap.sh, will install the local::lib module and prepare the environment. Its still not finished, it doesn't alter the .bashrc file, but it will get there.

The second, install_deps.sh will use the cpan shell to install a local Task::Bootstrap module. This Task has all the modules that I want installed.

There are still some problems. I still lack some distro prefs for a couple of them that pause the process and ask for user input. And some of the modules won't install without force (Mac::Carbon is the one that fails the most).

Other modules just don't install correctly on Mac OS X. Danga::Socket for example, requires Sys::Syscall, but this one fails the tests because Mac OS X lies about sendfile support: the sys/syscalls.ph includes the SYS_sendfile constant, but when you actually call it, we get a Function not implemented. I'm sure I could work around it, and probably fix it, but I no longer use Danga::Socket so I'll probably just remove that dependency.

The other was Mac::AppleEvents::Simple. Finder.app has a different naming scheme for its windows, and t/simple.t was failing. I've send a patch to the module RT Queue.

I still have small failures, but right now, I can mostly use this two scripts to setup a Perl environment from bare metal.

Update: I removed my ~/.perl5/5.10.1/ directory and ran time ./bootstrap.sh. The results:

real 56m4.598s
user 37m17.724s
sys  7m51.923s

So about an hour on a MacBook Pro 2.16Ghz Core Duo, running Leo.

When I started using them at work (circa 1998 or 99) the goal was simple: separate programmers logic from the layout so that designers can tweak the templates without breaking the code.

It was a simple and very worthwhile goal. And for the most part, it worked. You still need to coordinate which templates will be created, and how will the page be broken up into pieces to enable reuse of common parts, but it mostly worked.

The template systems where designed to be used by web designers, and given that those where accustomed to HTML tags, the more powerful template system that needed loops and decision constructs opted to create those with a simple syntax. The template systems became domain specific languages (DSL).

But, with the improvement of the HTML, CSS and JS capabilities, I don't know if the designers are still the target audience of template systems. As I split a set of HTML pages into a set of reusable templates, I try to create blocks that make sense in terms of caching, data-store life cycles, and other subtle factors that a web designer just doesn't care about. The page breakout is becoming a engineering exercise.

As programmers/engineers, we don't need the comfy world of a DSL. We want to work in the same language that we use to code. Templates are, at most, a small macro language that gets compiled into source code, in the same language that we have the rest of the system.

So I'm starting to move away from Template Toolkit, my preferred template system, into something that uses native Perl as the DSL. I don't know where I'll end up yet. I'm looking at Tenjin, Text::MicroMason, and Mojo::Template. There aren't many template systems that use Perl as the control language, actually.

The first two should be the most stable ones, older ones. Tenjin is also the fastest template system I've tested, although speed wouldn't be a deal breaker.

Controller-centric or View-centric render process

But we can't judge template systems without considering where and how they will be used. In my case they will the last step of a web request, handled by a web framework.

I'll use a BBC site page to illustrate my current dilemma, an article about Wikipedia page controls. Things of interest to me:

• the central part of the page is the main content, and directly related the the URL. This is the content that you came here to get. It has a lot of HTML before and after that are present to add value to the site (we hope), and provide exit destinations (the navigation and related stories);
• the title of the page, early in the HTML, is also directly related to the URL and the story: this is important because the template system must support inside-out rendering (first the central part, and then the wrappers), or have all the information in memory before starting the render process;
• some parts (the header, footer, and left-hand sidebar) are mostly static, and with low or no relation to the main content: in this case only the category of the article influences the navigation. It might also influence the related BBC sites;
• the right-hand side shows related stories. The "related" part might be highly dynamic, or static per category or a mix in between. The related external links are probably part of the story.

The path for a typical request (like /2/hi/technology/8220220.stm) looks like this:

1. collect all the important data from the URL: in this case the it seems to react to the hi/low part to pick the version of the master template, the category (controls the right-hand side column) and the article ID;
2. fetch the correct article and related informations (author, publish date, picture URL, caption...);
3. fetch the related articles for the right-hand sidebar;
4. call the view renderer with all of this.

Pretty basic and straight forward. Steps 1, 2 and 4 don't pose to many questions, they are required on any pipeline I can think of.

But the proper way to do 3 is a matter of discussion. On one hand you can put enough information on the URL to decide what the right-hand sidebar needs and fetch it in with some calls inside the Controller (I'm assuming here a MVC-style framework). In this model the controller prepares everything that we need to display the page.

But this makes the view a bit inflexible because if we need to had another box the top or bottom of the page, we would need to tweak the controller code.

On the other hand if our template system is powerful enough, we can have a small API that we can call back into Perl land, and have these little reusable boxes generated for us. The view controls the entire process of display.

The advantage of having a decent view system, even a object oriented view framework, is the extensibility that you get. Imagine a Moose-based template system: view are classes that you can extend, override, add before and after processing. Rendering is just creating an instance and calling a small method, as_string() or even stream_to_socket().

A system that brings the full power of Perl to the template world is Template::Declare. It treats templates as classes that you can subclass and extend. It seems very similar to what I like but I have zero experience with it so far.

No decisions have been made yet. I need to evaluate Template::Declare and see if it really fits my brain.

A beginning is the time for taking the most delicate care that the balances are correct.

from Manual of Muad'Dib by the Princess Irulan

extracted from Dune, by Frank Herbert

In the next weeks I'll be publishing a series of articles about a new project that I'm starting. Actually, I started this project once already, but I didn't like the path it was taking, so I'm restarting it.

The new system will replace the current EVOLUI.COM code base. Why rewrite a system that is in production for 8/9 years and that works reasonably well? There are several reasons actually.

The first reason is code rot: the current system is spread across 3 code bases (called internally E1 through E3). E1 was the first stab (who would guess, right?), and grew organically inside the company, without much planning, and coded by several programmers. E2 added a better management interface, and E3 added a new LMS and forum system.

A lot has changed in the last 8 years. And my inexperience in certain topics shows by the lack of proper tests and documentation. Also, it has grown to 850 Perl packages. Yes, eight hundred and fifty packages. I expect that more than half of those are no longer in active use.

The second, and more important reason, is that the new business requirements require a deep restructuring of the code, but given that we don't have a decent test suite, it would be madness to change stuff in one of those 850 packages.

The way I see it, I can write the new system properly, using a test-driven methodology, making sure that all the business logic is properly tested.

I'm not starting from scratch, I plan to reuse certain small but critical parts of the old system, but going over them to add tests and documentation.

The failed attempt was called E4, so this new version will be called E5.

I plan to cover the following topics (not sure of the order yet):

• source control, and ticketing systems;
• project directory layout;
• documentation techniques;
• business layer techniques;
• databases used and ORM layers;
• web frameworks;
• template systems;
• logging;
• smoke testing;
• deployment.

Hope you enjoy the ride.

Today, a new article describes the answer he got. In the end he decided to put the validation inside the model. I can only say: good for you.

But one of the reasons against the validation-inside-the-model that he received baffled me: Models don’t know about the current user (or other higher level information).

If your models don't know who is the person authenticated (and please read Yuval article about modelling identity to understand how complex it can become), then you are doing it wrong.

Models must know who is the authenticated user. They must know it because that is the only sane way to implement authorization inside your methods, and to generate a audit log of operations.

What I usually do is to create a Session class. For each request (be it Catalyst request, incoming email message, or XMPP request), I create a new Session instance initialized with the current user information plus some tidbits like channel (Web, email, XMPP) and IP address (if available).

Then all accesses to my API are either via the session object (create methods to access most important parts of your API), or by passing this session object to the API you are calling.

You should also use your Session as the access point to your logging and auditing capabilities.

The article mentions other issues like error messages. I haven't add the necessity of developing an application with multiple language support yet, but I think that the current throw-exception-objects strategy, with all the information (including the authorized user to figure our preferred language) will be able to deal with the problems that I can think of now.

In the end, I'm actually really curious about why would people think that putting your validation code in the controllers is a good idea.

Update: a lot of stuff in the comments if you care about this topic.