Extending Callback APIs With Closures

Closures are hard to understand for many beginners. If you encounter a callback API that is missing parameters you have an excellent chance to get your feet wet with closures.

Closures in JavaScript

I will use JavaScript here for illustrating the problem and its solution. Perl hackers will find a Perl version at the end of this post but should still read the explanations given for JavaScript.

The Problem

We will start looking at the following little script:

'use strict';

function logMessage(prefix, msg) {
    console.log(new Date() + ': [' + prefix + ']: ' + msg);
}       

function runService(logger) {
    logger('starting service');

    // Do something.

    logger('stopping service');
}       

var services = ['cruncher', 'crasher'];
for (var i = 0; i < services.length; ++i) {
            runService(logMessage, services[i]);
}

The function logMessage() in line 3 is a typical logging function that prints out a log message preceded by a timestamp and a log prefix.

Next we have a stub example for a micro service runService() in line 7. It accepts a logging function as a callback and then does its job true to the old Unix adage "Do one thing and do it well". In this case its job is to do nothing and it is particularly good at it. Well, it logs the start and stop of its activity with the provided logging function.

Finally, starting in line 15, two such services named "cruncher" and "crasher" are started.

Now run this script either in a browser or on the command line with node.js:

Mon May 22 2017 19:14:07 GMT+0300 (EEST): [starting service]: undefined
Mon May 22 2017 19:14:07 GMT+0300 (EEST): [stopping service]: undefined
Mon May 22 2017 19:14:07 GMT+0300 (EEST): [starting service]: undefined
Mon May 22 2017 19:14:07 GMT+0300 (EEST): [stopping service]: undefined

Umh, no, not exactly what we intended. The log message is used as the prefix in square brackets, and the actual log message reads "undefined". Why?

The error is actually easy to spot. Look at line 15:

runService(logMessage, services[i])

runService() expects one single argument, the logging function, and we give it two, in the naïve hope to inject the prefix into the function call. Let's fix that:

runService(logMessage(services[i]))

What? You must be kidding! Now we no longer pass the function logMessage() as the callback argument but the return value of its invocation which is not only not a function but also undefined. That can't be the solution!

Let's look closer at the "microservice" starting in line 8.

function runService(logger) {
    logger('starting service');

    // Do something.

    logger('stopping service');
}

We see that the logging function is always invoked with just one single argument and that is actually our problem here. What options do we have?

We can rewrite our logger so that it accepts only one single argument, the message, and gets the prefix value from the outside, for example from a variable outside of its scope, something like this:

var prefix;

function logMessage(msg) {
    console.log(new Date + ': [' + prefix + ']: ' + msg);
}
// ...
for (var i = 0; i < services.length; ++i) {
    prefix = services[i];
    runService(logMessage);
}

That is not only an ugly hack but would fail as soon as the services run asynchronously.

The solution is to use a closure. Leave logMessage() and runService() as they initially were, and pass a logger as a callback that "remembers" its context when passed around:

for (var i = 0; i < services.length; ++i) {
    var prefix = services[i];
    function innerLogger(msg) {
        logMessage(prefix, msg);
    }
    
    runService(innerLogger);
}

The closure is function object, a reference to a function, that gives its surrounding variable scope a piggyback into its invocation.

Run the new version:

Mon May 22 2017 19:23:04 GMT+0300 (EEST): [cruncher]: starting service
Mon May 22 2017 19:23:04 GMT+0300 (EEST): [cruncher]: stopping service
Mon May 22 2017 19:23:04 GMT+0300 (EEST): [crasher]: starting service
Mon May 22 2017 19:23:04 GMT+0300 (EEST): [crasher]: stopping service

Voilà! It works! And Voilà again! You have finally used a closure for fixing a real-world problem.

The closure now acts as glue code between the calling API and the callback itself. It translates the invocation from the signature that the API expects to the signature of the actual callback. In the same way you can not only add arguments but also remove arguments or change their order.

Passing Objects To Non-OO APIs

A very common case is that a callback API expects a simple function reference when you actually want to make it call a method of an object.

Assume that the logger is now a JavaScript object:

function Logger(prefix) {
    this.prefix = prefix;
}

Logger.prototype.logMessage = function(msg) {
    console.log(new Date + ': [' + this.prefix + ']: ' + msg);
};

The logMessage() method now gets the logging prefix from an instance variable prefix so that it no longer needs to be passed in every invocation.

But, alas, the "microservice" is still as dumb as it was before. How can we lure it into our now object-oriented approach for logging? You can use the same solution as described above, a closure as a wrapper around the method invocation:

var services = ['cruncher', 'crasher'];
for (var i = 0; i < services.length; ++i) {
    var prefix = services[i],
        logger = new Logger(prefix);
    
    function innerLogger(msg) {
        logger.logMessage(msg);
    }
 
    runService(innerLogger);
}

Works like a charm!

But have a closer look again at our logger object:

Logger.prototype.logMessage = function(msg) {
    console.log(new Date + ': [' + this.prefix + ']: ' + msg);
};

It uses the instance variable this.prefix and this leads us to a much more concise and more idiomatic solution to the problem:

var services = ['cruncher', 'crasher'];
for (var i = 0; i < services.length; ++i) {
    var prefix = services[i],
        logger = new Logger(prefix);
    
    runService(logger.logMessage.bind(logger));
}

Now what is this??? Try it out! It works!

Remember, every function in JavaScript is an object. And all function objects have a method bind() that returns the function object itself but with the keyword this bound to whatever you pass as the first argument to bind(). Finding out what happens, when you pass more than one argument to bind() is left as an exercise to the reader.

You should also remember that this is not necessarily an object but can be any JavaScript variable. The original problem could have been solved like this with the help of bind():

function logMessage(msg) {
    console.log(new Date + ': [' + this.prefix + ']: ' + msg);
};
//...
runService(logMessage.bind("cruncher"))

Now this is just a plain string but that works just fine.

Not interested in Perl? No problem! Now that you successfully tackled bind(), why not checking out what the related methods call() and apply() do?

Perl

The direct translation of the initial JavaScript solution would look like this in Perl:

use strict;

sub log_message {
    my ($prefix, $msg) = @_;

    my $now = localtime;
    print STDERR "[$now][$prefix]: $msg\n";
}

sub run_service {
    my ($logger) = @_;

    $logger->("starting service");

    # Do something.

    $logger->("stopping service");
}

foreach my $service ("cruncher", "crasher") {
    my $prefix = $service;

    sub inner_logger {
        my ($msg) = @_;

        log_message $prefix, $msg;
    }

    run_service \&inner_logger;
}

However, this does not work as expected:

[Tue May 23 08:08:46 2017][cruncher]: starting service
[Tue May 23 08:08:46 2017][cruncher]: stopping service
[Tue May 23 08:08:46 2017][cruncher]: starting service
[Tue May 23 08:08:46 2017][cruncher]: stopping service

The prefix is always "cruncher". Obviously, the variable $prefix keeps the value that it had, when the inner function was defined. The solution is to use an anonymous subroutine instead:

#! /usr/bin/env perl

use strict;

sub log_message {
    my ($prefix, $msg) = @_;

    my $now = localtime;
    print STDERR "[$now][$prefix]: $msg\n";
}

sub run_service {
    my ($logger) = @_;

    $logger->("starting service");

    # Do something.

    $logger->("stopping service");
}

foreach my $service ("cruncher", "crasher") {
    my $prefix = $service;

    my $logger = sub {
        my ($msg) = @_;

        log_message $prefix, $msg;
    };

    run_service $logger;
}

Now it works like expected. In general, closures in Perl, are always anonymous subroutines. Anonymous subroutines are almost always what you want, and they are also the fix for the notorious warning "variable $xyz will not stay shared" (or its sibling warning "subroutine xyz will not stay shared").

There are two pitfalls, when using closures in Perl: The variable definition in line 23 above looks redundant and in JavaScript it actually is redundant. In Perl it is required. You cannot use the loop variable $service directly inside the closure. You have to assign it to another variable, here $prefix. The loop variable exists inside the closure but is undefined. This behavior is a little bit surprising.

Another very common error is to forget the semicolon in line 25. Doing so produces a compile-time error:

Bareword found where operator expected at microservice.pl line 31, near "run_service"
    (Missing semicolon on previous line?)
syntax error at microservice.pl line 31, near "run_service "
Global symbol "$logger" requires explicit package name (did you forget to declare "my $logger"?) at microservice.pl line 31.
Execution of microservice.pl aborted due to compilation errors.

You assign something (the closure) to a variable ($logger) and that is just a regular statement and as such has to be separated by a semi-colon. This is easy to forget because you are not used to putting a semi-colon after a function definition.

Using Object Methods As Callbacks

Like in JavaScript, you can also make the calling API invoke a method of an object. Try this:

#! /usr/bin/env perl

package Logger;

use strict;

sub new {
    my ($class, $prefix) = @_;

    bless \$prefix, $class;
}

sub logMessage {
    my ($self, $msg) = @_;

    my $now = localtime;
    print STDERR "[$now][$$self]: $msg\n";
}

package main;

use strict;

sub run_service {
    my ($logger) = @_;

    $logger->("starting service");

    # Do something.

    $logger->("stopping service");
}

foreach my $service ("cruncher", "crasher") {
    my $prefix = $service;
    
    my $logger = Logger->new($prefix);
    
    my $callback = sub {
        my ($msg) = @_;

        $logger->logMessage($msg);
    };

    run_service $callback;
}

Now you can use the old-school callback API with an object-oriented Perl program.

And what about the JavaScript bind() solution described above? Isn't this possible in Perl? Well, it wouldn't be Perl if it wasn't possible to implement a perlish equivalent to JavaScript's bind() in a completely transparent and idiomatic fashion with just a few lines of code. Read how in a future blog post!