Rss on Notes

P2P? Think NNTP...

melo@simplicidade.org (Pedro Melo) — Fri, 06 May 2005 11:41:50 +0000

Celso wrote about using P2P technology to distribute content, in the context of blogging.

There are two types of content that you might want to distribute: the syndication feeds, and the HTML pages.

The syndication feeds is a hot topic right now, specially in the Feedmesh group. Feedmesh already has notification distribution and it seems to be working great. In the two previous weeks there where some messages about full content distribution of feeds in the Feedmesh mailing list.

My personal view about syndication feeds distribution is well known to Rui and Nuno who are forced to listen to me every time the subject comes up. I’m a firm believer that NNTP is the way to go regarding the distribution of RSS/Atom feeds. Notice that I’m not talking about Usenet, but the NNTP protocol itself.

If you use NNTP to flood your content network, and write the files in a easy-to-export layout so that you can put any HTTP server over it and use that as a syndication proxy, you have a basic Content Distribution Network (CDN) for small files.

There are a lot of small issues that can be improved. There is no need for each node to keep a copy of every feed. If the logs of the HTTP server are used as a lazy-subscription mechanism, falling back to plain HTTP Proxy when the feed is not found locally, but signaling the NNTP server that he should accept that feed from now on, you have a organic beast that follows the trends of your client base. You can also expire old feeds that nobody seems to be reading, or expire only the feed itself but keep the meta data of feeds that are not updated (so you can answer 304 to our clients). We can (and should) store the feed data as GZiped files, so you don’t need to GZip-on-the-fly.

You can point out a lot of drawbacks to this system: the publisher looses statistics of readership, you can’t do password protected feeds (at least not in the traditional basic HTTP authentication sense), among others. But those are the tradeoffs that we make to save yourself a lot of traffic.

My personal side-effect is that if this kind of infrastructure was available, we could extend Feedmesh to the syndication clients. The current Feedmesh is only useful to be used between the major syndication aggregation services because it sends all the feeds that where modified. But if each node in this CDN also provided a filtered Feedmesh feed based on subscriptions of clients, then they could remove all the pooling and switch to a trigger-based content retrieval if they so wished (care should be taken to prevent a swarm of request, but that could be done by controlling the stream of updates sent by this service).

The question is who will support this. Well, the optimistic person inside me, would like to believe that the ISPs would be interested to provide a better service to it’s customers, and if they had a simple system that just worked, they would install it on their own networks, and give it to their clients. The advantages are that they would save some bandwidth (negligible in a world of Bittorrent, I know) and give better service to its customers.

But the main push, the main collecting of feeds seems to be a perfect fit to the current syndication aggregation services. They already need to do it because their business model is based on having the most fresh content, so each one of them fetches the changed feeds whenever we ping them. If they start pushing the content via NNTP, we just might have the seed we need to implement something like this.

The HTML distribution part is different, and I think we are starting to see the solution: Google Web Cache.

Google is happy to do that service for you, to be your Coral Network. It seems obvious to me that Google value is related directly to the freshness of their own index. So they want your HTML as soon as it changes. So if Google taps the Feedmesh notification stream, they can see two things: that a specific page has changed, so they can fetch it, index it and cache it (and feed their own GWA), and (as a bonus) mark that page as more than likely being a blog and with that improve PageRank by removing the problems that the blogs are causing them.

Akamai is probably going to be relegated to DNS and static content CDN from now on.

RSS your yum updates

melo@simplicidade.org (Pedro Melo) — Sat, 22 Jan 2005 18:04:48 +0000

If you like RSS and use Fedora Core 3, then you can have the nightly yum update reports delivered to you via RSS.

The following setup requires: * using fedora core 3, or having the command rss-generate in yum (check man yum); * your mail is delivered via qmail; * you can install .qmail files.

You probably could workaround the last two. I might do that someday. For now, consider this as a quick (took me less than 20 minutes after I noticed rss-generate in yum manpage :) ) hack.

The setup is simple: each night, a cron runs on each server, and generates the RSS file. The file is then sent to a special mailbox, and there we filter the message and place it in a public HTTP site. You then configure your favorite RSS reader with that URL.

First I created a email address with .qmail file in my homedir:

cat > ~/.qmail-yum <<EOF'
|/home/melo/bin/convert_yum_rss_mails_to_http.pl
& youraddress@yourdomain.top
EOF

The second line should be a good email address. If something goes wrong with the script, it will fallback to the email address. At least you’ll know that something has gone bad.

Then you place the perl script in the correct place. You can download the convert\_yum\_rss\_mails\_to\_http.pl script, or paste it:

#!/usr/bin/perl -w

# Where to place RSS files? Should be available via Web
my $dir_for_rss_files = "/var/httpd/htdocs/yum_rss";
my $server;

while (<>) {
  $server = $1 if /^Subject: Yum changes for (.+)/;
  last if /^$/;
}

fallback('no_server') unless $server;

my $filename = "$dir_for_rss_files/$server.xml";

umask(022);
fallback("could not create file '$filename.new'") unless open(RSS, ">$filename.new");
print RSS <>;
close(RSS); 

sucess() if rename("$filename.new", $filename);

unlink("$filename.new");
fallback("could not rename file '$filename.new'");

# qmail exit codes, see qmail-command man page
sub sucess   { exit(99) }
sub fallback { my $m = shift; print STDERR "Err: $m\\n"; exit(0)  }

That’s it. If something goes wrong, check your qmail-send logs. A message is written there.

Now you only need to generate the mails on each server. To do that download the script yum-rss-report.cron. It looks like this:

#!/bin/sh

/usr/bin/yum --rss-filename=/dev/fd/1 -R 120 -d 0 -e 0 generate-rss |  \\
/bin/mail -s "Yum changes for `hostname`" youraddress-yum@yourdomain.top

This command will generate the RSS (the -R 120 will make sure that each server will wait a random number of seconds), and mail it to your address that you set up before. Make sure you change the address in this script, and make sure you can receive mail at that address.

Then, copy the edited script to all the servers that you want to report to you. On each one, copy the script to /etc/cron.daily and make sure you make it executable:

cp yum-rss-report.cron /etc/cron.daily
chmod 555 /etc/cron.daily/yum-rss-report.cron

That’s it! Just subscribe each server in you favorite RSS reader and each morning you’ll know which servers got upgraded and to which versions.

I have some improvements to make to these scripts, but they are good enough for now.

Enjoy!

darcs2rss now at 0.5

melo@simplicidade.org (Pedro Melo) — Thu, 30 Dec 2004 17:47:49 +0000

Real life has kept me from posting what’s going on at simplicidade.

I just sent a mail to darcs mailing list announcing version 0.5. I switched to a proper XML::Parser and added command line options to change RSS metadata.

In my personal copy I have two big changes: - generating RSS 2.0; - filtering comments: current filters are ‘simple’ and ‘markdown’.

Both these features are optional, of course.

I really like markdown and I’m using it more and more everywhere, so this is mostly a feature that I use a lot and I needed fast. But to use HTML formating in your item descriptions, you need to move to RSS 2.0.

In order to do that I’ll have to deal with meta-information. Using command line parameters is not such a good idea. My current script to generate a feed for darcs2rss itself is 4 lines long! I’ll have to code support for a config file.

Will see.

About darcs, it’s been really great. Much more simple to use than arch, my previous favorite CVS replacement, and fast enough. I’ve been lurking the mailing list and there seem to be some situations where darcs can be very slow, but my projects are very small, so it’s not an issue for me. The darcs developers are working on this, but I admit that I don’t even try to follow those discussions, that usually evolve into mathematical discussions :).

darcs2rss

melo@simplicidade.org (Pedro Melo) — Wed, 08 Dec 2004 13:09:05 +0000

I’ve been using darcs for some weeks now, and I like it a lot. I’ll write more about darcs and why I like it in a future post, but I’m switching from CVS to darcs for my current and future pet projects.

Last friday, Bennett Todd was asking in the darcs-users mailing list for a script to generate a RSS feed of darcs changelog. I send him a quick perl script to do just that. I also got some sugestions and improvements.

So I created a darcs repo for this project. You can find it at http://darcs.simplicidade.org/repos/darcs2rss/. You can browse all my repos at http://darcs.simplicidade.org/.

Next, I have to make a small page about it. Not today, though. Real life is waiting. Meanwhile, you can use the readme to know more, specially the roadmap.

Episode IV: a new hope

melo@simplicidade.org (Pedro Melo) — Tue, 28 Sep 2004 13:23:51 +0000

Well, if your bandwidth bill from you RSS feeds is starting to eat your income, then you should thank Bloglines and some desktop aggregation software companies.

Check out the new service from Bloglines. Basically they will develop a new Web service API so that applications can pull the feeds from them, instead of pooling your site directly. FeedDemon, NetNewsWire and Blogbot will have support (apparently there is a beta of FeedDemon already out with support for it).

See also Mark excitement over the new service :).

Although I don’t like pooling (see a previous post regarding mod_speedyfeed), I see this one as a necessary step towards a real pub-sub network. Bloglines, as others who already have a lot of feeds locally, are the best players to start, or to feed a pub-sub network.

Update: there is a Perl module already!
Update 2: some comments from Brent give you some idea about the possibilities of this setup.

Very good stuff, congrats to Bloglines.

Reading RSS

melo@simplicidade.org (Pedro Melo) — Tue, 31 Aug 2004 07:56:05 +0000

I’ve been using NetNewsWire to read RSS feeds since I started doing it, little over two years ago. So far it has been good enough and the features in the to-be-released version 2.0 seem great.

Yet, I don’t see some features that I would like to see, like aggregating several items into a single one if they mention the same URL.

So I need to implement some sort of filter between the feeds I subscribe to and the feeds I read, that transforms the content into something more manageable.

I don’t know if the end result is something that can be read via something like a RSS reader, or if I need some other application. I think I’ll also look into reading my feeds by converting them to mail messages. Rui is using newspipe and I noticed Hep to (although I don’t know if it does the same thing yet).

I really need to waste less time (30 to 40 minutes each day) catching up on RSS feeds, or become more efficient reading them.

Coral your RSS

melo@simplicidade.org (Pedro Melo) — Sun, 29 Aug 2004 12:19:22 +0000

Now that Coral is in public beta, what about telling your RSS subscribers to use it? It will reduce your bandwidth bill.

If you subscribe to this blog, change your subscription to http://www.simplicidade.org.nyud.net:8090/notes/42.xml (notice the nyud.net:8090). You should start to get this feed from your nearest cache of Coral.

Seems to work.

If Coral is here to stay, you can even add to your config:

RedirectPermanent /notes/42.xml http://www.simplicidade.org.nyud.net:8090/notes/42.xml

Update: Rui popped up on IM and told me that this does not work, and of course he’s right. It will cause a loop. I’ll have to check if Coral sends a specific user agent that I can filter. Rui tells me that it sends a HTTP_VIA and he has the configuration to solve this. Now I’ll wait for him to post it and copy it :).

I wonder if the choice of port (8090) was a good one. Some corporate firewalls don’t allow nothing unless is on port 80.

By the way, Microsoft DNS servers have a problem with Coral. The developers are thinking about a workaround for the fact that MS DNS chokes on DNAME records (I confess I didn’t know about them either, but they look pretty cool).