node-seq Revisited (About Asynchrony, Again)

I'm late to the party. While the new Tesla Roadster drove upside down and did mid-air flips at its reveal, and the most impatient people are already using ES6 features in their projects, I've brought you another library to make an async developer's life easier.

There have long been a million promise implementations, and async for anyone who wants it. There's also Seq, widely known in narrow circles, by the not entirely unknown Substack. I started using it almost from my first days with JavaScript and used it wherever I could. Its approach to taming asynchronous spaghetti seems clearer and more logical to me than the one in async, for example. See for yourself:

var fs = require('fs');
var Hash = require('hashish');
var Seq = require('seq');

Seq()
  .seq(function () {
    fs.readdir(__dirname, this);
  })
  .flatten()
  .parEach(function (file) {
    fs.stat(__dirname + '/' + file, this.into(file));
  })
  .seq(function () {
    var sizes = Hash.map(this.vars, function (s) { return s.size })
    console.dir(sizes);
  });

It's all so simple and clear that I'm too lazy to explain it. Unfortunately, the library hasn't been developed or maintained for a long time. On top of that, I accumulated a list of bugs I kept running into, a list of features I wanted, a list of complaints — because not everything worked as well as I'd imagined. One day I ran into the world's imperfection once again and decided that This Moment had arrived. Time to fork and fix. I do that often, but what this library did seemed like real Magic.

So, happy and full of determination, I do git clone, cd, gvim, and try to understand what's going on. I don't. The author uses a couple more of his own libraries, so to reach enlightenment I first have to figure those out. After a couple of hours I get bored and discover a Fatal Flaw. Said and done. I sit down and write the Library of Dreams from scratch. Strangely enough, there was no magic in any of it. The prototype was ready in an evening. Then I spent some more time polishing it while using it in a real project, replacing Seq completely. And here's what came out. Let's get acquainted.

YAFF — Yet Another Flow Framework.

Overall, I tried to make it compatible with Seq. Most code migrated by simply replacing the import (from var Seq = require('seq'); to var Seq = require('yaff');). Of course, a few other things had to change too. Seq uses .catch() to catch bugs. For example, the snippet above could be changed like this:

var fs = require('fs');
var Hash = require('hashish');
var Seq = require('seq');

Seq()
  .seq(function () {
    fs.readdir(__dirname, this);
  })
  .flatten()
  .parEach(function (file) {
    fs.stat(__dirname + '/' + file, this.into(file));
  })
  .catch(function (err) {
    console.error(err);
  })
  .seq(function () {
    var sizes = Hash.map(this.vars, function (s) { return s.size })
    console.dir(sizes);
  });

This construct is terrible because after we've “caught” an error we can keep going. That can't be allowed. First, it's unclear what to do when parEach (or another similar method) throws several errors. Catch only the first? Catch all? What if we've already gone far down the chain and an error pops up on a timer in some parEach above? What if there's no catch below us anymore? What if the catches below aren't prepared to handle errors that crawl out of a forEach on a timer? Many questions, no answers.

So I decided that every YAFF chain should have exactly one error-handling construct, and it should be at the end. And to keep with Node.js tradition, let it handle the result too. The result is beauty. See for yourself:

var fs = require('fs');
YAFF(['./', '../'])
  .par(function (path) {
    fs.readdir(path, this);
  })
  .par(function (path) {
    fs.readdir(path, this);
  })
  .flatten()
  .parMap(function (file) {
    fs.stat(__dirname + '/' + file, this);
  })
  .map(function (stat) {
    return stat.size;
  })
  .unflatten()
  .finally(function (e, sizes) {
    if (e)
      throw e;
    log(sizes);
  });

If we assume this is all inside an asynchronous function, in finally we can simply pass in the callback we were given, like this:

var listDirs = function (dirs, cb) {
  YAFF(dirs)
    // magic bubbles
    .finally(cb);
};

Convenient? I like it too.

Actually, the library is built around the concept of an argument stack array (I should have written about that at the very beginning). Every method here changes that stack by applying to it or to its individual elements. Say a flock of functions wrapped in par takes one element each from the stack, in the order the par calls are written. Only after all the par callbacks have fired — the callback inside every method is this — does YAFF move on to what's next in line. Suppose next we have several functions wrapped in seq. YAFF will apply the whole stack to them and replace it with whatever the wrapped function returns in its callback. Here's the code, so it's clear:

YAFF(['one', 'two', 'three'])
  .par(function (one) {
    this(null, one);
  })
  .par(function (two) {
    this(null, two);
  })
  .par(function (three) {
    this(null, three);
  })
  .seq(function (one, two, three) {
    this(null, one, three);
  })
  .seq(function (one, three) {
    this(null, null);
  })
  .seq(function () {
    this(null, 'and so on and so forth');
  })

Of course, if someone calls their callback with a non-null first argument — an error — YAFF immediately spits on all the functions still waiting there. It goes off to execute what's written in finally. If you forgot to write finally, or decided that your code definitely can't have errors, YAFF will unceremoniously throw an exception when one occurs. So it's better to have a finally.

There are also all sorts of synchronous functions for working with the argument stack as an array: push, pop, filter, set, reduce, flatten, extend, unflatten, empty, shift, unshift, splice, reverse, save(name), load(fromName). Phew, I think that's all. The names speak for themselves, but if anything — don't hesitate to ask and look at the source. There's one file (main.js) and the whole thing is very simple.

And, of course, the reason all this was started: the asynchronous functions for processing arrays of data — forEach (YAFF won't wait for this block to finish; its results won't affect the stack; YAFF moves straight to the next handler in the chain), seqEach, parEach (YAFF will wait for all the functions to fire, but their results won't affect the stack), seqMap, parMap, seqFilter, parFilter — they do what you expect. YAFF waits for them to finish, and their results replace the values that were on the stack before.

Also, every method with a par prefix accepts a number after the function. That number is the limit of simultaneously running asynchronous functions. Something like this:

var resizePhotos = function (photos, cb) {
  YAFF(photos)
    .parMap(function (photo) {
      asyncResize(photo.image, photo.params, this);
    }, 10)
    .unflatten()
    .finally(cb);
};

In this example we resize a batch of photos asynchronously. To avoid overloading the server, we resize no more than 10 pictures at once for each client. unflatten is needed to gather the photos spread across the stack into an array that will be a single argument for the callback.

YAFF also has mseq and mpar — a curtsy toward async users. These methods take an array of functions to be executed sequentially or in parallel. We could just as well write a bunch of seq() and par(), but sometimes you want to generate functions dynamically. We're a functional language, after all, aren't we?

To confuse you completely, I came up with the following example and drew a picture (in the desperate hope that it will clarify everything):

YAFF(['./'])
  .mseq([
    function (path1) {
      fs.readdir(path1, this);
    },
    function (arg) {
      this(null, arg);
    }
  ])
  .flatten()
  .parMap(function (file) {
    fs.stat(__dirname + '/' + file, this);
  })
  .map(function (stat) {
    return stat.size;
  })
  .unflatten()
  .finally(function (e, sizes) {
    log(sizes);
  });

I hope I managed to explain what I wanted clearly. You've got the idea — and I'm not the only one who doesn't understand why anyone needs async.

All concrete wishes and suggestions are best filed as issues (in English — don't be shy, I'm bad at it too; we'll practice together) on GitHub, or even as pull requests.

Oh, right — the library is on npmjs.org.

P.S. Just now, in a fit of passion, I added a synchronous apply method — so all the other synchronous methods can be thrown away. But I'll keep them for convenience and compatibility.