Notes on Notes

New Mac

melo@simplicidade.org (Pedro Melo) — Thu, 30 Mar 2017 22:37:08 +0000

I got a new Mac at work a couple of weeks back and I’ve been slowly moving my life to it. It is the new Macbook Pro 13” with a 2.9GHz i5 and Touch Bar. The only upgrade I did was bumping the memory to 16GB.

For comparison, my previous Mac was still a Late 2008 Macbook, fully upgraded with the unofficially supported 8GB and after market SSD (two in fact, replaced the DVD a year back with a second SSD). And yes, apart from the slowness (a 2.0GHz Core 2 Duo can take only so far these days…) it is still a perfectly good Mac, and I still use it on occasion.

The build quality on the new Mac is excellent with one exception. I don’t like the keyboard at all, it is too shallow for me. On the other hand, 99% of the time I use an external keyboard, so I can live with it. Still, I would trade a thicker laptop for a better keyboard. This is a “feeling”-thing. In actual use the keyboard is precise enough for my casual needs, I just don’t like shallow keyboards.

The screen is wonderful, but don’t get to enjoy it much because I mostly use the Mac with non-retina external displays. Compared to the old Macbook the difference is striking.

The SSD is wicked fast, it feels like magic, and I love having 8 hours of battery. This are actual real life number for my usage: text editors, Perl, a bit of Go, and a light Docker - heavy Docker cuts this to 5 to 6 hours.

The Touch Bar is useless to me. Even when I’m using the laptop keyboard, there is very little that the Touch Bar provides that makes my life simpler or more productive. As productivity enhancer, I don’t have many needs for an analog input mechanism. The only thing analog that I use in there is the sound and brightness controls, and I’m perfectly happy with the keys for those on my external keyboard.

I do really miss the Esc key…

I’ve thought about of exchanging this one with the model without a Touch Bar, but that one is inferior in at least two ways:

only has two USB-C ports: yes, this is relevant, see below;
the CPU is 30% slower, and even with the upgrade you don’t reach the less expensive Touch Bar model.

I also remember reading that the internal Thunderbolt is half the capacity of the Touch Bar models, but I don’t have the reference at hand at the moment.

So I’m keeping the four-port Touch Bar model.

Regarding dongles, yes, they are unfortunate, but I like the fact that I now have a laptop with USB-C 3.1 and TB3. I can plug some really nice external devices with this, and I don’t mind this dongle iteration. Over the years I’ve owned and used:

serial DB9 mice, DIN-5 keyboards, and parallel port printers;
DIN-6, PS/2 style, keyboards and mice;
USB-A, USB-B, and all the mini and micro variants;
SCSI-I, -II cables;
Firewire 400 and Firewire 800;
VGA, DVI, HDMI, and too many display port variants.

I’m not naive enough to believe that this will be the last interface type, but I’m hopeful that it lasts as long as USB-A did. If he can do that, at the physical connector level, I’ll be happy enough.

I mentioned above that I needed more than 2 ports. This is true. I currently have the following adapters:

a Multi-AV VGA adapter from Apple: VGA and one USB-A, plus power passthrough - this is a decent mini-docking station for the office;
a Gigabit ethernet adapter from Belkin: due to my work, a physical network connection is a requirement;
a external SSD USB-C 3.1 Gen2 enclosure: really really fast - and the best way to keep a set of large git repos and build tools between two computers - the speed is very good over USB 3.1 Gen2;
a Apple USB-C to USB-C connector: this one works, there are a lot of really bad dongles out there.

So I need at least three ports at the office, I can live with two during the weekend but I can’t connect to the external display at home. I might need to buy another AV connector there, but I’m looking into other options.

My hope is to see a decent 27” or 30” display, TB3, with two USB-A, a ethernet port, and hopefully one or two USB-C or TB3 ports. That would be the perfect docking station for me. The idiotic LG 5K is a TB3 monitor but has no extra ports!

One final warning: I bought a USB-C => 3x USB-A + Ethernet adapter at a local retailer (identical to this one but from another brand). This is a low-quality dongle. The USB-A and Ethernet ports work fine, but the moment you plug it in, the Wifi connection dies. This is most likely related to poor shielding somewhere.

In terms of build to order upgrades, I would recommend that you at the very least max out the RAM: you won’t be able to do it later, and a Mac with 16GB of RAM will last a couple of years more. If you do any type of virtualisation or Docker, you will see a big difference.

After the memory, it depends. I don’t value CPU power that much, the i5 I have is more than enough, but others might. I think my second upgrade would be bumping the SSD to 512GB if you have big iTunes libraries like me.

Now the big question: would I buy the new Macbook Pro again, or would I go back to the previous generation without USB-C ports? I think this is easy to answer:

if you are a true road warrior, moving from place to place, and with a myriad of connectivity scenarios, you will probably be better served with the previous model, without USB-C - you will save a lot on dongles to cary around and loose. I might change my mind if a small USB-C dock, with video (VGA + HDMI), USB and ethernet shows up soon;
if not, if you are mostly stuck to a desk or move between one or two places, I recommend the new one.

We’ll talk again in 8 years.

User interaction is not allowed

melo@simplicidade.org (Pedro Melo) — Fri, 24 Mar 2017 18:23:03 +0000

This errors happened to me sometime ago, but today I saw it again on a coworker Mac.

You’ll get the error on the Terminal, with commands that want to use MacOS Keychain passwords. I’m sure there is another element at work here, but I could not find it.

In this case, the error on docker login was this (you’ll get a similar error with git fetch or git pull):

Error saving credentials: error storing credentials - err: exit status 1, out: `User interaction is not allowed`

The command line tool is trying to access a Keychain Item, and this item is protected to only be used by some applications. On normal conditions, this would trigger a graphical pop-up asking for permission for this new application to access said item, but in this case he fails to launch the pop-up and therefore, rejects the action.

To fix, open the Keychain Access application, search for the domain you were trying to access (in the case of docker login it would be the private registry address, or hub.docker.com), edit the entry, select the tab “Access Control”.

You can then add permission to the application you need by taping the “+” button on the lower left, and picking the correct file, but I don’t have a easy way to discover what is the correct file to add. If the problem is docker login you could first try to add the command /Applications/Docker.app/Contents/Resources/bin/docker-credential-osxkeychain.bin. For git pull or git fetch issues, try to add /Library/Developer/CommandLineTools/usr/libexec/git-core/git-credential-osxkeychain.

If you don’t know which command line application is the culprit, you can just disable access control for this password item by making sure that the “Allow all applications to access this item” option is selected and “Save Changes”.

You should try to find the correct application if possible, but the second options will work. No security, but it would work.

Question mark dock icons

melo@simplicidade.org (Pedro Melo) — Fri, 17 Mar 2017 08:55:02 +0000

On my new Mac (more on that later), I’ve noticed that some apps (in my case Slack and Mattermost) would revert to big gray question mark icons when I restarted them.

The key piece of information that set me on the right path was a comment on the Vandyke support forum.

While that solution worked for me, it seemed a bit too geeky. When you copied the app from the DMG or Download folder into Applications, Finder should have taken care of this for you.

My simpler solution: quit the application, drag it out of the Applications folder (to the desktop or another place, not important), and then copy it back to Applications.

This worked for me.

brew info

melo@simplicidade.org (Pedro Melo) — Fri, 17 Mar 2017 08:40:15 +0000

Small, quick, tip: before installing the latest wow app with brew install wow-app, take a minute to scan brew info wow-app in a separate terminal.

Two main reasons:

some of the time, it will tell you that you have a lot of options to enable or disable some features. You might need some of them, and usually this requires a new compilation;
most of the time this includes what next steps are required of you to properly configure, and you can read them while the app installs.

Either way, this will save you some time, and you might learn something new.

Where to eat: Figueira da Foz edition

melo@simplicidade.org (Pedro Melo) — Thu, 22 Sep 2016 00:36:51 +0100

A friend asked for good places to eat around Figueira da Foz, where I live.

He was not the first, it is a common request, so I wrote a page with some of the places I go to on a regular basis.

So enjoy, Eating in Figueira da Foz.

iOS (null) Applications

melo@simplicidade.org (Pedro Melo) — Sat, 27 Aug 2016 07:21:48 +0100

Over the past weeks, I wasn’t able to complete any of my iPhone backups, and without a backup (actually two, iCloud and iTunes, just because) I didn’t want to upgrade to the latest iOS. Last night I decided to investigate why.

The Console.app error I got was this:

26/8/16 10:28:09.285 PM AppleMobileBackup[42056]: ERROR: Backup message response:
     101 Unable to open domain directory: No such file or directory (2) at path
     "/var/mobile/Containers/Shared/AppGroup/453DA9D8-45A4-4B88-9D43-57A5772B6C57"
     (MBErrorDomain/101)

Not very useful. I guessed that if I could find which app it was, I could delete it, but googling for 453DA9D8-45A4-4B88-9D43-57A5772B6C57 or parts of it didn’t give me any hints.

Eventually, while browsing Definitions > General > Storage and iCloud Usage > Manage Storage I found a application without an icon:

Selecting it I was able to see just two things: the version and the size, nothing more.

I searched a bit about these (null) applications and there are a lot of tips and tricks to delete them. In my case, deleting it by long-press and using the small x didn’t work, neither going to the Manage Storage section of the definitions and deleting it from there.

The other solution was finding out what the application was, using the size as a guide, and installing it again.

It turns out that there is a simple way of figuring out which application it is based on size. I opened iTunes, and inspected the iPhone apps, sorted by size, and it was very simple to identify it:

I was now pretty sure that this was the application that got stuck. Unfortunately, re-installing it did not solve the issue, the application was now stuck on installation, and removing it was still impossible.

I decided to use iTunes to try and do it. I clicked the Remove button in the image above, and sync’ed my iPhone. This worked, and the application was removed.

With this application out of the way, I was able to complete my iPhone backups and upgrade to the latest iOS.

Crossroads

melo@simplicidade.org (Pedro Melo) — Wed, 10 Aug 2016 11:39:31 +0100

I’ve finished reading A Generation Lost in the Bazaar by Poul-Henning Kamp (of Varnish fame).

The key takeaway is simple enough: quality happens only when someone is responsible for it.

Right now, I’m working on my personal/family project, modernising the infrastructure of a 16 years old Perl application, and moving some parts to other languages to make it easier for us to hire resources. I love Perl, and still is my favourite language, but getting people to work on it in Portugal is a very hard proposition.

But the future is uncertain. What should I do after I finish this task? So I go back to what I’ve been doing over the past three years, managing a team of developers, nurturing young talent and give freedom and cover to senior developers to do its thing?

Or should I keep it small, build the hand-full of services I would like to build, not something that is brand spanking new, but my own personal view how those systems should be work? Be one more SaaS out there, selling my work directly to companies?

Should I go back to the factories, or go for the boutique?

I honestly don’t know yet… There are still somethings I would like to do, that are only available at a factory. But the thing is, one of the major decisive points to leave my previous employer was that I could not longer control the quality of the work my team was doing, and that is a position where I never want to be again.

Quality happens only when someone is responsible for it. And I want to be that someone, somewhere. Just not sure where that will be.

Phoenix, take 2

melo@simplicidade.org (Pedro Melo) — Tue, 26 Jul 2016 13:06:25 +0100

In Greek mythology, a phoenix or phenix (Greek: φοῖνιξ phoinix; Latin: phoenix, phœnix, fenix) is a long-lived bird that is cyclically regenerated or reborn. – via Wikipedia, Phoenix (Mythology) entry

A couple months back, the server where this site was hosted died in a horrible disk crash. I had (more or less) decent backups so nothing was lost, but the time to bring it back to life didn’t materialize. For the past 3+ years, I was deeply involved with a couple of eCommerce sites that took all my free time.

I left my previous employer on June 30th, and took a couple weeks of holidays to be with the family. And now I’m back to figure out what to do next.

For the past years, I felt the need to write about current events and technical issues that I found interesting, but lacked the place to do it. So one of the first priorities I had was to recover this site. The previous setup was still using a Movable Type 3.x installation, but I wanted some sort of modern setup, with the same “burn to static files” approach that MT had. I decided to use Hugo as it seems a very sensible and straightforward solution to the problem, and it is written in Go, a language that I’ve been learning over the past year or so. I also picked the Hyde-y theme for Hugo as a simple enough layout. Not sure if it will stick, but good enough for now.

To avoid any further hosting snafu’s, I’m publishing the site using GitHub pages. This unfortunately means that I have to update the RSS feed URL. You should use https://www.simplicidade.org/index.xml to subscribe to updates.

Next steps was to convert/copy the old MT posts to this new site, and add a couple of alias to make the old URLs work properly. This is still work in progress, I’ve placed the scripts I’m using in a new repository, movable_type2hugo.

It was not ready yet, still needs tweaking, but I got everything back. I had to migrate some posts from DB (with all the original metadata), some by scraping the static HTML pages that MT generated, and others from MarsEdit, with only some of the metadata. For some reason, my backup has all the HTML pages for the site, but the DB that was backed up at the same time, only has some of the pages. Weird.

Most of the previous Phoenix post is still valid, but now I’m back for more.

Fat scripts

melo@simplicidade.org (Pedro Melo) — Wed, 12 Nov 2014 21:53:19 +0000

Recently we installed HipChat at the office, and I want to write a small script to send messages to it. To make it easier for others to use I wanted to distributed it as a single standalone script, no dependencies.

I wrote the script, x-hipchat, with all the bells and whistles I felt were important, and then turned to the process of pack it into a single no-dependencies script. The simplest method I could find is App::fatten.

For best results, you should create a configuration file somewhere, and use the --config-path to point to it. In fact, if you don’t create it, you keep getting “ERROR 412: Profile ‘your-script-name’ not found in configuration file”. In the configuration file, the section name is the base name of your script. My config file looks like this:

[profile=x-hipchat]
overwrite=1
exclude_core=1

The first attempts ran fine, but the fat script failed to find JSON::backportedPP and later HTTP::Tiny. And if I manually included them with include=MODULE on my config file (oh, there is a bug in 0.13, if put a single include statement, it borks - issue open), HTTP::Tiny will die later on, I’m guessing because some of the internal modules that HTTP::Tiny could not be found.

I tried to use trace_method=prereqscanner_lite_recurse configuration option and fatten found more modules, but too many to include, and some were Core, which caused problems with a different version of perl.

The final solution for this was to install with perlbrew a older version of perl (I choose 5.10.1), and create a perlbrew lib just for packing:

perlbrew install -j 4 perl-5.10.1
perlbrew lib create perl-5.10.1@fatten
perlbrew use perl-5.10.1@fatten
cpanm App::fatten Module::CoreList

(we force Module::CoreList because App::tracepm requires a more recent version that she declares.)

This gives me a sufficiently old perl that lacks most of the more recent core modules, and will force them into the script. This actually helped, but I still got “599 Internal Exception” from HTTP::Tiny.

I switched to trace_method=require that actually runs your code. This is much better because a lot of modules are only loaded later, when needed. Here App::fatten could make my life easier. My script requires parameters but I could not find a way of pass them. A command line option to give extra parameters to the program would be nice. Fortunately I had environment variables for most options, so I used those.

Eventually I found the culprit: HipChat API is HTTPS-only, and HTTP::Tiny and perl in general require XS modules to use HTTPS, and XS modules cannot be fat packed.

So I could not create a standalone script for this because of my HTTPS dependency. Oh well…

I think this is a good example for me to dust up my Go knowledge…

Messaging

melo@simplicidade.org (Pedro Melo) — Sat, 02 Aug 2014 10:27:23 +0000

A thread showed up on Twitter about messaging software options. My story is not 140-charaters material, so here we are.

If you do insist on a tldr; version: use NSQ if you need something that works out of the box, use Redis if you want to build your own.

At work, we use a lot of message queues to plug different components together. They are a great tool to have on your belt but the landscape is filled with solutions so picking your own solution can be a daunting task.

I’ve also gone through several solutions over the past 19 years working on ISP and Web-related businesses, I have scars to prove this, so what you’ll read below is the summary, obviously biased, of my experiences.

I divided this into four sections:

Defining the problem;
Solutions out there;
What do I use?;
Tips and tricks I collected over the years.

Defining the problem

To help us understand one another, I’ll give you the mental model I use to think about this problem.

There are three types of actors involved:

First you have producers. You have two main types: some generate messages that represent work tasks that someone must perform, others just emit messages that represent significant events that might be worth reacting to on another part of your system. A producer can do both during its lifetime. The big difference is that the work tasks usually have higher requirements of persistency. You should not loose a message that triggers a email to a client on a successful order for example.

Then you have consumers. They collect the messages from producers and do “stuff” with them. You imagination is the limit on “stuff” here. You can perform the task you got, collect statistics, log events, you name it. Consumers are usually producers too, generating further messages to other consumers in line.

For CS geeks, producers and consumers usually form a acyclic directed graph.

And then you have you queue software, the message broker. I usually like to divide them into three main categories:

centralised: you have one central message broker that producers and consumers talk to. Given it’s SPOF nature, some provide you with clustering solutions;
distributed: you have message brokers everywhere, producers can connect to any of them, consumers may need to connect to all of them or only to some if the broker provides forwarding services (typical tradeoff, where do you put the complexity, on the broker or the client). The solutions here vary a lot on how producers and consumers discover message brokers or each other for a direct connection, and what kind of consistent view of the network is provided. Assume there is some warm up period during which messages could be lost;
embedded: the queue software runs on your producers/consumers process space and connect to each other directly or via tiny dumb message routers. They tend to produce distributed topologies, but you get to solve you problem at a lower level in the stack.

The message broker is there to provide you one main service: decoupling. Two dimensions of decoupling are important to have:

location: producers should not know or need to know who the consumers are, or how many there are, or where they are, and you should be able to add/drop consumers at any point in time;
time: something produced right now should be available to a consumer that connects at a latter point in time.

You also should look at what type of message routing options and fan-out strategies are provided. The two main ones are pub/sub and round robin respectively. Check what kind of options are available to use them together inside the broker, usually a powerful combination.

Pub/sub will broadcast messages to all consumers, some solutions will provide a routing only on topic, others hierarchical topics (you can subscribe to subtrees), and some even complex filtering solutions. Round robin will distribute incoming messages over all of the consumers, each message getting to only one consumer. One thing to keep an eye out is how good the broker is at allowing you to cluster multiple consumer connections as a single one, for load balancing purposes.

Then you need to consider message delivery guarantees, a key phrase to look for when you are shopping for this systems. Personally I tend to look for “at least once” semantics, as I believe them to be the better trade-off. “Only once” is usually too resource intensive, and all the others are too unreliable for my use cases. Removing duplicates on the consumer side is easy enough with either a memory cache with decent expire policies or if you store the message ID on the destination system as a unique key.

Finally you should think how the system deals with the two main problems that you’ll see:

slow consumers: how can you avoid or deal with a message broker holding hundreds of thousands of messages;
dead consumers: what happens to the messages the consumer received just before his life was over.

Fun stuff, and I even had to mention byzantine failures yet!

Solutions out there

There are lots. I believe that we have very good generic tools now, providing you the required building blocks for this type of systems, and this caused an cambrian explosion. It seems almost all startups out there decided to roll their own, and publish it as the best thing since sliced bread. Guilty as charged…

I have a borderline fetish interest on this stuff, so I always look at any solution that crosses my radar. Below I’ll list the main candidates I think you should look into.

First for rolling your own, you have two generic tools: ZeroMQ and Redis.

On the mostly built, just use it section, you have AMQP-based solutions, SAPO Broker, Gearman and NSQ.

ZeroMQ

ZeroMQ is fast, very fast, but also very very low-level. It is an embedded system, but also includes a small message router - 0MQ Api”) that you can spin up to provide you location decoupling, and some basic translation services, from Queue to Broadcast for example. You can build basically any topology, but you do it by hand, using the building blocks. It’s basically for custom-made solutions.

You have pub/sub and you can also load balance messages between different workers.

Previous version had disk-based storage for overflow messages, but this was deprecated and removed in more recent versions.

There some higher level systems that use ZeroMQ as the backbone, but none ever caught my eye as interesting enough.

Basically, a fast low-level embedded library where the sky is the limit on what you can do as long as you know you get to keep this baby all for yourself.

Redis

Redis is a memory-only system (it had a complex VM system before, but it was dropped; there is a unofficial VM branch, I wouldn’t trust it), so all your queued messages will be in memory. You need to understand that you can fill up your entire memory and crash or stop accepting commands. It has protections agains this, look into them.

So, either you have (as in “you build one) a serious flow control system to pause producers, or you need an escape valve, a special “consumer” that goes into big queues, dumps messages to disk and puts them back later, or just drop them and let your producer or bean counter system inject them again later.

Publish and subscribe works, even in the future cluster version, but consumers need to be subscribed before the message is published. There is no queueing for subscribers, none.

To work around this, most systems use the BLPOPRPUSH command which reliably takes a message from a inbox queue and inserts into your own private queue for processing. As long as you have a system to look at those private queues and inject messages from the private queues back into the inbox, you can build a pretty fast and reliable queue with this.

A solid centralised system, that you can use to build whatever semantics you need.

AMQP

AMQP-based systems are centralised, with a very flexible routing solution, that allows you to build almost any topology you need.

I started a pure-perl implementation of the client, that would take the XML protocol specification (the best part of AMQP actually) and generate the Perl code for it, but while doing that I was always thinking “do you really need all this protocol complexity?”

I moved to simpler things. I really didn’t like the centralised aspects of it, and most clustering solutions look to me as a fix for a design decision, not a good symptom.

Still, it has it’s place if you need a central system for environments with strong requirements of auditing for compliance purposes.

SAPO Broker

I used this a lot when I worked at SAPO. It is a distributed system, very fast, with a rich set of semantics, and clients in multiple languages.

It includes a forwarding service, so you can subscribe to topics locally and the network reacts and starts forwarding relevant messages to your node.

The node can become the bottleneck on big topics. I haven’t looked at it recently, but you cannot setup your own node to receive only a subset of the entire traffic, so that you could distribute a single topic set of consumers over a set of nodes.

Again, I haven’t looked at it recently, it might already have a solution for this, but I do know the author and the code is solid. In heavy use at SAPO for quite some time.

Look into it.

Gearman

One of Brad Fitzpatrick (of memcached, perlbal, Open ID fame…) pearls.

A distributed system focused on work tasks, doesn’t support pub/sub at all.

Pretty good semantics to guarantee you don’t loose messages, although persistence was only added later.

Also includes a feedback system to report work status back to the client, including progress messages, something you don’t see often.

It just works.

NSQ

Built from scratch in Go by the Bitly people, processing huge amounts of messages there.

Very easy to install, has three main components: the message broker nsqd, the topology/discovery service nsqlookupd and a decent admin interface with nsqadmin.

Each nsqd works standalone, reports topology to a set of nsqlookupd servers. Logic is on the client side that aggregates information received from all the discovery services to form a coherent picture of the network, and finds out to which brokers he needs to connect to to collect all messages on the topic he is interested on.

Allows you to stream messages to clients, keeping a per-client configuration number of in-flight unacknowledged messages, works wonderfully well for asynchronous consumers, or for consumers that need to batch messages and process them in a single operation.

My recommendation at the moment.

What do I use?

I gone through several solutions: AMQP, Gearman, ZeroMQ, Redis and now NSQ.

AMQP

I dropped AMQP altogether. The centralised nature doesn’t provide me enough benefits to compensate for a complex protocol. It is very flexible, but I find it a heavy complex solution.

Having said that, if you want/need to use AMQP, use RabbitMQ.

ZeroMQ

Some of the AMQP authors agreed with my complexity assessment, they dropped AMQP and created ZeroMQ (and one of them is even doing nanomsg, an even simpler system).

I still keep ZeroMQ at hand for very specialised solutions, with very stable topologies, where speed is the number one concern. I don’t expect any kind of reliability of systems build from this, so I usually wrap them into a end-to-end reliability blanket like a opportunistic transactional queuing system (see [Tips and tricks][] section).

For speed, you cannot beat ZeroMQ. nanomsg seems saner but I don’t think is ready yet.

Redis

I absolutely adore Redis… It is my favourite swiss army knife. You can do just about anything with it, and it has all the tools you need to write you own queuing system. At least, that’s what 99% of all startups think, because queuing systems using Redis is probably the major software category on Github at the moment.

It was my go to system for quite some time, and I still use it if I need something quickly, very simple to get it right, but it reacts badly to slow consumers, you have to be very careful with that.

You can use use several out-of-the box solutions. Resque is one of the most popular, and it is available on most languages, but logic is on the consumer side, so each version might not be compatible with the others.

My favorite is qless, a Resque inspired solution, but that uses server side Lua for all the logic, which means that all languages can share the exact same logic (see the qless-core repository).

Gearman

This is rock solid stuff, Etsy processes a huge amount of tasks with it (I believe I saw the number 6000 per second over 5 servers on the mailing list).

The discovery part is the non-existing bit. If you add a new broker, you need to tell all your producers and consumers.

If you want to use this, look at Consul for the discovery part.

NQS

My current favourite, and the one I use now. It gets so many things right:

dead easy installation: you can just use the provided binaries, but there is no GPG signature on those at the moment, something that I hope to be fixed soon;
decent “at least one” semantics;
decent buffer to disk semantics, although we try to avoid this situations with producer flow control (see tips and tricks section);
the pubsub + channels with round-robin is my favourite topology;
the nsqlookupd solves the discovery problem nicely: we use three of them, they share the same AWS micro instance that hosts the Consul cluster.

We run a central cluster of 2 servers for general use, but we also keep one nsqd per instance. We use it for system integration, monitoring, and logging (we have a couple of consumers that take batches of messages and publish to Loggly).

We also have a couple of consumer gateways between NQS and nginx-push-stream-module, allows some messages to reach client browsers in real-time.

Still, I would like to see:

nsqlookupd should publish topology changes over a nsq topic: this would allow consumers to react faster to new producers;
nsqd to nsqlookupd protocol should be over a public channel, that other systems could leverage for cluster monitoring and control.

I’m still learning all the details of NSQ and getting into the community, so maybe some of this is already there and I just haven’t found it.

For Perl mongers out there, I’m writing a NSQ client using AnyEvent, basically extracting the internal working code into a public distribution. My goal is to release it on August 14th, CPAN day, but I can only work on it during weekends.

Our code deals keeping track of the topology using nsqlookupd and connecting to multiple nsqd, although the in-flight message management might still need some TLC. Currently, on my CPU laptop, I can do a bit more than 1M message per minute (no processing of course), but the benchmark is clearly CPU bound (sum of broker, producer and consumer give 180% CPU usage on my dual core CPU… well, I was listening to iTunes and browsing the web at the moment…), a faster CPU would give you better numbers.

Tips and tricks

Opportunistic transactional queuing

My motto is “trust but verify”. I usually build the queuing solution assuming it will loose messages, but I keep a checks and balances system on the side that will re-inject a message if they doesn’t show up on the queue system.

The main ingredient I use is something I call oportunistic transactional queueing. This derives from an observation: the messages that need the highest reliability usually happen at the end of a database transaction.

The common scenario is a client that does something over the web or API that changes the state on your system and you need to perform further work on it, preferably asynchronous. If you have a work log, a place to keep stuff that you need to do later, on the same database, then you can insert a row there in the same transaction that updated the user state. The expensive part is the transaction commit, so adding a new row is marginal. At the same time you send the message to your queueing system.

You also need a bean counter software. It will scan those work log tables from time to time (once a minute is fine, unless you have a lot of them), and make sure they are either done (and in this case you can archive them), or lost (and in this case re-inject them).

This will scale with your application because you can shard this work log. You can have as many work logs as you want.

As long as you don’t scan this tables often and as long as you archive them aggressively, you should avoid most of the problems usually associated with “database as a queue” problem. The main thing you should remember is the work log is just to check the reliability of your queue, not as your queue.

Topology discovery messages

Mapping the topology and monitoring it can be a hard problem to solve with this systems, specially if they are distributed.

Engineer your consumers and producers to support discovery messages. When a discovery message is received by a consumer, he needs to do two things:

report that you got it: the message should include a URL, consumers post a small JSON blob saying who he is, where he is, where did he receive the message from, which topics is he subscribed to, which topics does he publish to, and optional statistic information (total messages processed, uptime, messages processed since last tracer, time since last discovery message, whatever else you fancy);
publish the message to all topics that you would normally publish to.

This simple system allows you to collect those reports somewhere (I use Redis myself), and show a unified view of your network (with pretty load graphs even) on a browser.

End-to-end message tracing

Include a UUID on the message you generate, and make sure this UUID is passed along, from publisher to consumer.

By searching your logs for this UUID you can see all the consumers the message travelled through and even show it as a timeline, with parallel tasks in separate lines.

I usually publish a log message on a separate topic, include the consumer identification, the UUID of the task, and the time it took to process, and then have a Redis-based collector that keeps track of the last couple of hours of messages, to see how much they took end-to-end, and where are they spending their time.

I can quickly spot hot spots in the network with this system.

Have at least three priorities

No matter the technology I use, I try very hard to have three levels of priority:

urgent: this message should take precedence over all others;
normal: basic stuff, normal traffic;
bulk: lowest priority, consume this if you have nothing better to do. It can introduce delays on bulk message processing, but you can detect that, and spin up a new consumer (even a dedicated consumer just for bulk).

Sometimes I split normal into “normal normal” and “normal but generated by clients”, but unless you have other problems with some slow components on your system that need this type of split to give priority of your clients messages, I don’t bother.

On NSQ and Redis this is usually implemented by using the topic name and a prefix (_bulk, _urgent prefixes for the two non-normal queues.)

Producers flow control

With slow consumers, you have to be careful not to flood your broker network with messages that they will have to spend time writing to disk and reading back. I find it easier to pause producers.

This of course assume that you can do that. On my uses cases, I’m fortunate enough to have producers that can limit their output rate. If you deal with end-user traffic, you might get spikes you can’t control. Hopefully you’ll be able to launch more instances to cope with them.

I do this in several ways, depending if I’m using Redis or NSQ. Some tricks you can try:

on very simple situations where you are using a Redis list a small queue between two systems, when you push into the queue, Redis returns the number of messages in there. If this number is larger than you high-water threshold, just pause, check from time to time until it reaches the low-water threshold and resume;
on NSQ you can periodically query the broker you are publishing to and find out the size of the outgoing channels backlog. Use that information to manage your output rate. Works well, supports multiple consumers per topic, but it’s only hop-by-bop;
include on your messages a “pause” URL: if a consumer is felling overwhelmed, you can use the URL to pause and resume the producer. Works throughout your network, no matter how many levels it has;
if you use the topology discovery messages above, the system that collects them can periodically generate traffic report messages on a separate channel with the recommended rate of input on each topic, by analysing each flow report he gets. Producers subscribe to this report channel and use it as a guideline. This is a complex system to get right, but when it does, it is a beauty to see.

I’m sure there are other ways, but these four so far allowed me to solve this problem whenever it showed up.

I love the last one, and I’m currently toying with a system like this, I’ll write more about it later, when I have enough experience to know if it is working or not.

The third is a good compromise. It is very nice to have a callback into the producer to tell him to shut up for a bit, and works with any system.

Final words

Enough words already. Go and start having fun with this stuff. Send me links about your work, love to read about real world problems solved with this systems.

800

melo@simplicidade.org (Pedro Melo) — Mon, 02 Jun 2014 17:46:34 +0000

WWDC is a couple of hours from starting. Some thoughts about it.

800-pound gorilla

I was at Imagine conference a couple of weeks back, and one presentation clicked a lot of different articles I’ve read recently.

You can read this Wired article to get the basic business logic behind this. Basically, Apple is the company with the largest number of credit cards on file, by far. Almost 4 times Amanzon.

So my WWDC prediction is that they will take advantage of this by doing two things:

creating a iOS/Mac OS X framework that allows you to integrate direct payments on your apps: that part that I don’t like of this prediction is that I don’t thing they would be able to charge the same 30% they charge on other payments via the Apple Store;
exposing this framework as a global JavaScript API on their browsers: this would allow checkout processes through out the world to immediately taking advantage of this, at least on all iOS-native browsers.

With 800 million users, and most of them with a credit card, this would make them the largest mobile payment service overnight.

One more thing

The twist of this is that, by having the same customer base, and using the same trick, a JavaScript-based API built-in into the browsers they control, they could also become a leading identity provider.

One-click/swipe/button-press login on all your accounts online.

What I would like to see

Apple TV 4, or at least an announcement that we would be able to install Apps on a future Apple TV hardware.

I think this WWDC will be very interesting to watch indeed!

Phoenix

melo@simplicidade.org (Pedro Melo) — Tue, 13 May 2014 09:03:48 +0000

This blog should be renamed to Phoenix…

Since we last spoke, a lot has changed but most remained the same.

I still have three kids, and a dog. I’m still married to the same smart beautiful woman.

I still use Perl as my main language, although I’ve been using a lot of JavaScript and Go (my favourite new language; in 3 years it will be a major contender, mark my words) whenever I can get away with it. I still truly completely hate significant white-space.

I still use daemontools supervise everywhere, but I’ve been playing with Docker more and more, and I might finally retire my decades-old deployment setup, as soon as I figure out a decent way of linking containers across different hosts securely (geard of Project Atomic looks good for this). Also OpenShift Origin is nice. And with version 1.6, Vagrant will finally become my preferred developer environment.

I still use qmail and ezmlm-idx for all my mail and newsletters needs.

I still find that, for small companies, hosting in a cloud like AWS is much more expensive than dedicated hosting, but I’ve learned to appreciate the cosy environment they provide if you have enough scale and some money. I love that AWS BeanStalk supports Docker containers now.

I still buy books, in their natural dead tree format, although I do have a lot of them in eBook format on my iPad. Yes, I finally caved in last Christmas and bought myself a iPad Air. I could not find a decent Moveable Type client though. Yes, I still use MT too, although my ageing setup is no longer CGI-powered, I’ve switched to a Plack-based setup around the original CGIs with Starman.

I still have my (now) 14 years old startup, I still have a lot to do there. But I also needed a new challenge, so I took a new job (well, a year ago). I’m responsible for a couple of eCommerce sites in Portugal, doing very fun stuff with Node.JS, AngularJS, Redis and Perl of course. BTW, we are hiring, ping me if you like any of the previous stuff. It is nice to have sites that make money.

Yes, I still think Redis is the best thing since sliced bread. Seriously. The only thing better than Redis is Git. And possible Camlistore.

I still use MySQL. And I use it as a document-oriented DB, with either compressed JSON or Sereal documents. I do keep extra tables with some of the document attributes for decent reporting though. I’m curious about ArangoDB. And the 9.4 Postgres release, with JSON indexes, looks awesome. Can I haz it in MySQL soon?

I still search with SphinxSearch, although I find ElasticSearch cute.

I still like XMPP, but I haven’t had the time to work with it recently.

I still use Mac’s, they still last 6+ years each on my hands (currently using late 2008 Macbook). I still use my iPhone4, and I don’t plan to switch for at least one year more. I like hardware that lasts that long.

I still think people who believe iOS or Android will ever dominate the other should search history for cases of zero-sum games.

I still love Portugal, although I miss a lot of friends that left us for greener pastures over the last two years.

Health-wise I had a kidney stone. I’ll just say that you should ask for the real drugs from the start. Probably the most painful experience ever…

Best of all, I’m still happy.

Californication

melo@simplicidade.org (Pedro Melo) — Mon, 15 Apr 2013 06:54:56 +0000

There will always be extreme in Californication, from the initial minutes of a blasphemous moment in a church by a agreeable nun, to the moment that dreams do come true.

Maybe boys will be boys and we just enjoy lewdness, or maybe there is a grain of truth in it. But 6 years ago when I started watching the show, I never though the roller coast of emotions it would trigger.

Based on a set of never ending love stories, some peaceful, other just normal and a big sad one, it keeps rocking my socks off with the impossible real situations, and the fulsome dialogs.

You get to live an alternative universe in a even dozen instalments every year, and the only thing left after you are done, is a longing for the next fix.

Thank you, Hank Moody, you degenerate impossible lucky bastard. See you next year.

Dreaming in sync

melo@simplicidade.org (Pedro Melo) — Mon, 07 Jan 2013 11:32:10 +0000

If you ever get me talking about applications and synchronisation of data, you might notice that I'm very passionate about sync.

You'd be right. My favourite topic in college was distributed operating systems. Today, when I design a system, I always envision it as a set of cooperating processes, working together and in parallel for a common goal.

Over the past 15 years, I've kept refining a set of rules of what I think is the ideal features any application that does sync should have.

I've narrowed down to three.

1. Close your laptop and go

If you see a "sync now" button, they blew it.

To paraphrase Vince Lombardi:

Sync is not a sometime thing; it's an all time thing. You don't sync once in a while, you don't do things right once in a while, you do them right all the time. Sync is habit.

If you share application state with your co-workers (say for example a project management app, with issues, goals, notes, design documents, whatever), when you are sitting in your office, your own version of the application is running on your laptop. It has a copy of the state. Maybe not all of the state, maybe just the part that relates to you, but for now let's assume that it has all of the state.

When you close the laptop lid and move to another place, the information on your local copy should be the update version of the shared state up to the moment you lost network connectivity.

There should be any "oh I'll just wait a bit and sync before I disconnect" though crossing your mind. Sync should not be part of your thinking process. It should just be the normal world view, it should be subconscious.

One of the best outcomes of this always-in-sync subliminal state? Given that you'll be always seeing, manipulating, the latest version of something, the opportunity for conflicts based on mis-information is rare.

2. Cherish thy conflicts

Which brings us to the big bad wolf of synchronisation: conflicts. This is the primary fear most people who think about implementing a sync solution share.

What I'm here to tell you, is that conflicts are you friends. They are the tether that binds you to humanity, because they only come up when someone has a different world view from your own.

Having someone who disagrees with you is glorious! It gives you the opportunity to get out of your bubble and interact. And you get to choose how to do so. Maybe you just IM or mail him, or maybe you call him, or even share a few minutes of his time in a hall near a whiteboard.

Conflicts are nature way of telling you that you need to get out a bit and talk to someone.

3. Be a packrat, collect it all

The most important lesson I got from git was not the graph of objects on which it was built, but the fact that git never stores diffs. It will always store a complete version of the new state.

The main advantage of having the entire state blob for each version is simple: you can always improve your conflict resolution, or your diff algorithm, or any part of the UX of both, because you have the raw data available. If you store diff's, you'll be using the state of the art diff algorithm at that moment, and going back to any version means replaying all the diffs since the last full version.

You might think that this is a huge waste of space, and it could be if you have big state blobs on which only a small percentage change between versions, but I posit that this is not the most common case. In fact, big state blobs should sound warning bells in your head, and start to break them up into smaller concepts.

No, you want small state blobs, connected together. They are faster to sync individually, and they provide a smaller surface for conflict resolution when those blissful events are casted upon you.

And with small state blobs, when state changes, you should always store

Let's be about it

So you are hyped now, you want your next application to have sync. What is your next step?

The Dropbox generation

The sync experience of Dropbox is comparable to the automobile experience of a Model T. Or the the sexual pleasure of masturbation. It gets the job done, barely, but you end up thinking that should be more to life than this.

It should, and there is.

Dropbox is a fine product, I depend on it daily for a lot of things, but most of them are files.

To sync files between devices and across several operating systems, there aren't many other solutions out there with the same proven track record. So in recent time, a lot of applications that deal with files have added builtin Dropbox support as their solution for the sync problem. And it works fine, for files, and single user scenarios.

But when you start talking about application state, with multiple users, if you plan on using Dropbox, you better start modelling your data store as a series of independent files. And forget about data protection of any kind.

But it can be done. For the single user case, it is more than enough.

The best example I know (and use, and recommend by the way) is 1Password. A couple of versions back, they switched from a single file to a file bundle as their storage system. You can read all about the Agile keychain design, good stuff in there.

What they did was break a large state blob into several smaller ones.

But the Dropbox API lacks any way to be notified in real-time of changes others may have made to your files. Sure, the desktop client uses a private Dropbox API to be notified of new stuff, but that is not available to you.

So, if you plan on using Dropbox as the sync service for your app, remember that always-in-sync scenario requires the Dropbox desktop client, and you have to monitor (fsevents on Mac OS X, or inotify on Linux, kqueue on FreeBSD...) the filesystem yourself to detect changes.

But, but, but git is magical!

Yes it is. But only for some things. Files, source code, text articles.

If you plan on basing your sync solution on git, thats fine. It is a viable option now that libgit2 is stable.

Just remember that source code conflicts are easier to solve because they happen on something that has structure, something that can be validated,at syntax level by your language compiler or interpreter, and at semantic level by your test suite. You do have one of those, right?

There is no possible test suite that covers your application data semantics. You can have some business rules that must be checked before accepting changes to application state, but they will never be comprehensive.

Also, git lacks any way to notify you in real-time of new commits on remote branches, so you have to roll your own too.

Natives need not apply

You might think that all of this is my subtle way of pushing you from your friendly web to the evil seductive native apps embrace, but no.

Yes, I do believe that the vast majority of your app logic should be executed on the client side. It's not only my distributed systems engineer wet dream, but also your ecological duty not to wast all those processor cycles sitting on your lap or nicely wrapped in your hand.

I also believe that no matter how much bandwidth you have, how blazing fast your servers are, you can't beat the latency of a local app working on local data.

And besides, even if you have always-hopefully-on internet connectivity, isn't it better to hide all those wishful thoughts about "when the user presses this button I will have network access, my server will be up, and everything is peachy" in the smallest part of your application as you can manage it? If your entire app uses local data, and then, in the background, those changes are sent to the remote peers, the user experience is much faster.

So keep you mad Web skills, they are your best bet it this the brave new continent you're about to explore. Just think about bringing along a simple HTTPS server, sitting right there on your local device, holding a copy of your app, and your state.

Interactive sync

There is a crop of products that focus on real-time collaboration. Software like SubEthaEdit and recently Ex-Google Wave.

They are very specific niches of the whole synchronisation topic, and I don't think they are a good solution for most apps. A bit of overkill.

Action!

Finally, my brain dump from having done some synchronisation work.

Divide and conquer. Start with a simple problem. Don't make sync a feature in a future version, plan it from the start. It is very hard to bold sync later on, believe me.

Cherish immutable content, or single-owner content, like comments. No conflicts there. A post has a set of comments, this set is conflict free.

Embrace that conflicts are a human problem, not a technical one, and instead of wasting your time dreading them, focus on the best UX you can provide to make the work of two or more people involved easier to do: how to pick the new world view?

Start with a central meeting place, where all changes flow through. It is easier to do at first, just make sure this central place never accepts conflicted information. All conflicts must be solved on the client side befor any changes are sent.

Start with "rolling sync", the simplest approach to merges: if you have been offline for a bit of, first undo all your offline changes, apply all remote changes done since your last sync, and then roll your changes over that to catch any conflicts. If you know git, think rebase, not merge.

Use UUID's or SHA1 of content as identifiers: there is no clear cut rule on which one to pick, it really depends on the situation.

Remember the truth about clocks: everybody has one, and each one keeps his own perfect time. If only they could agree on what that perfect time is...

Don't ask permission to collect all of what you can from your peers, just do it. Later, your person can always check why was something changed, and who did it.

Its a brave new world

Sync is not just a feature anymore, it must be something that its just there, unquestioned, subliminal.

Do your part, and start designing your apps with "Sync first!" mentality.

tidyall: TextMate command, and other tidbits

melo@simplicidade.org (Pedro Melo) — Fri, 21 Sep 2012 10:41:11 +0000

I've started using Jonathan Swartz excellent tidyall for all my tidy needs. This is TextMate command that I use to replace the Tidy command distributed with the Perl bundle.

require_cmd tidyall 'Requires tidyall, install from CPAN'

### Have a default tidyall.ini on your home, override per project
config="$HOME/.tidyall.ini"
for ini in tidyall.ini .tidyall.ini ; do
  if [ -e "${TM_PROJECT_DIRECTORY}/${ini}" ] ; then
    config="${TM_PROJECT_DIRECTORY}/${ini}"
  fi
done

## Override target file if the current scope is perl: useful when
## tidying a selection of perl code inside a Mason document
target="$TM_FILEPATH"
if [ "${TM_SCOPE:0:11}" == "source.perl" ] ; then
  target="${TM_PROJECT_DIRECTORY}/perl.pl"
fi

tidyall --root "$TM_PROJECT_DIRECTORY" \
        --conf-file "${config}"        \
        --pipe "$target"

To use it, create a new command in your own personal bundle, and set:

Save: Nothing;
Command(s): the above code;
Input: Selected Text or Document
Output: Replace Selected Text
Activation: Key Equivalent Ctrl+Shift+H (the same as Tidy);
Source: source.perl, comment.documentation.perl

The last one, Source, you should also add other scopes that your tidyall config covers.

My current tidyall.ini is this:

[PerlTidy]
select = **/*.{pl,pm,t}

[PodTidy]
argv = --column=100
select = **/*.{pm,pod}

[Perl::AlignMooseAttributes]
select = **/*.pm

[Perl::IgnoreMethodSignaturesSimple]
select = **/*.{pl,pm,t}

[MasonTidy]
select = **/*.{mc,mi,html}
select = **/{autohandler,dhandler}

The MasonTidy section is still new to me, I'm not sure if I like it or not. The rest is very good, specially the clean integration to tidy Method::Signatures declarations provided by the Perl::IgnoreMethodSignaturesSimple plugin.

Although this plugin targets the Method::Signatures::Simple, I'm using it with Method::Signatures, my preferred version of the concept, without problems

I've also used the recent git pre-commit hooks code for a bit, but I'm having problems with it and haven't had the time to really look into it. The code stashes stuff that is not in the index before running, to make sure it is only tidying the parts you are going to commit, and then pops the rest from the stash. This last part causes me a lot of merge failures.

If you don't use hunk or line-based commit building, you should be fine though.