Platform-Independent Asynchronous Child Process IPC

Spawning subprocesses and catching their standard output and standard error channels sounds like a trivial task but it is somewhat tricky to make it also work on Windows because of the platform's numerous quirks. It is, however, feasible, as the instructions below show. Code is provided for both C and Perl. You should be able to port it to other scripting languages without major problems.

The Task

The development version of Qgoda generally works on recent versions of Windows. One exception is helper processes.

Qgoda allows configuring an arbitrary number of helper processes that run in the background. Typical helpers are bundlers like webpack, Parcel, or Vite, and Development web servers like http-server or Browsersync. Qgoda catches their standard output and standard error channels and integrates the messages into its own logging with the prefixes of info or warning respectively.

In parallel, Qgoda polls the file system for changes and rebuilds the web site if a source file has been modified, deleted, or created. True to one of the fundamental principals of Un*x "Everything is a file" polling the file system for changes means that you are doing a select() on a file descriptor for a special device file that can be used to consume file system changes. This is true for the Linux inotify API, for macOS FSEvents, and for the BSD kqueue(2) system call.

In other words, Qgoda in watch mode simply does a select(2) on various file descriptors and processes the events for them inside its main loop. Actually, Qgoda does not call select(2) directly but uses Marc A. Lehmann's excellent AnyEvent library that provides a uniform interface to various event loop implementations. But under the hood, AnyEvent is a glorified version of select(2). That holds true, at least for the pure Perl event loop implementation.

GitHub Repository with Example Source Code

If you want to see the complete code examples in C and Perl, clone the git repository at https://github.com/gflohr/platform-independent-ipc. Its main branch compiles and works both on POSIX systems and Windows. The branch "unix" shows the normal approach that works only on POSIX systems.

Traditional Child Process IPC

The traditional way of asynchronous inter process communication with child processes consists of a combination of pipe(2), dup2(2), fork(2), exec*(2), and select(2). This pattern can be seen in countless network daemons and similar software.

For multiple reasons that doesn't work on Windows:

1. Windows doesn't have fork(2).
2. Windows doesn't have execve(2) and friends.
3. Windows supports select(2) only on sockets, but not other file descriptors, like pipes.

One recipe to work around these Windows quirks and create a communcation channel between the parent and child process goes like this:

1. Create a pair of connected sockets and enable non-blocking I/O on it.
2. Create an anonymous pipe.
3. Set the "handle" for corresponding system file descriptor (standard input, standard output, or standard error) in the STARTUPINFO structure to the write ends of the pipe from the previous step.
4. Spawn the child process with the CreateProcess() system call, and pass a pointer to the STARTUPINFO from the previous step.
5. Create a new thread for each system file descriptor that you want to connect.
6. Inside these threads, synchronously read from one of the anonymous pipes and copy everything read to one of the sockets. Both reading and writing blocks, which is okay because the main thread is not blocked.
7. In the main thread do a normal select(2) on the read ends of the socket pairs.

As it turns out, the intermediate thread is not necessary, neither in C nor in Perl.

In C, it is sufficient to do this:

1. Create a pair of connected sockets and enable non-blocking I/O on it.
2. Set the "handle" for corresponding system file descriptor (standard input, standard output, or standard error) in the STARTUPINFO structure to the write ends of the socket from the previous step.
3. Spawn the child process with the CreateProcess() system call, and pass a pointer to the STARTUPINFO from the previous step.
4. Do a normal select(2) on the read ends of the socket pairs.

In Perl, you have no control over the STARTUPINFO argument to CreateProcess(). You therefore have to use a slightly different technique:

1. Use the Perl socketpair(2) emulation to create a pair of connected sockets and enable non-blocking I/O on its read end.
2. Duplicate the file handles for STDOUT and STDERR and remember them.
3. Duplicate the socket pairs for the sockets from the first step overwriting STDOUT and STDERR.
4. Spawn the child process with the Win32::Process::Create() method.
5. Restore STDOUT and STDERR.
6. Do a normal select(2) on the read ends of the socket pairs.

Note: If for whatever reason you prefer the traditional approach with the helper threads doing the blocking I/O on anonymous pipes, you can check out the example repository and go to commit e9c71ac. It contains the version with the pipes and helper threads.

Detailed Description

Let's now go into more detail about the solution. If you haven't done so already, you should now clone the example git repository https://github.com/gflohr/platform-independent-ipc. The code assumes that you want to catch standard output and standard error of three simple commandline applications.

Only the Windows version of the code is explained. The POSIX version is pretty much standard and not worth documenting.

The Child Code

The example program writes one line to standard out and standard error in an endless loop, and sleeps a random amount of time between each message. See child.c and child.pl in the example repository.

For cosmetic reasons, standard output and standard error of the child processes should be unbuffered or line-buffered. Otherwise, the output will be collected into large chunks of usually 4096 or 8192 bytes, and it will take quite some time until the first buffer is flushed and you can see output.

In Perl, this is done as follows:

autoflush STDOUT, 1;
autoflush STDERR, 1;

For older Perl versions, you may have to add a line use IO::Handle; to make autoflush available.

In C, you would normally use line-buffering because we always write out one line at a time:

setvbuf(stdout, NULL, _IOLBF, 0);
setvbuf(stderr, NULL, _IOLBF, 0);

The libc will allocate a buffer of optimal size and enable line-buffering for stdout and stderr. But - surprise, surprise - that does not work on Windows (setvbuf(3) returns EOF for failure). Instead you have to resort to unbuffered output on the streams:

setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stderr, NULL, _IONBF, 0);

Printing gibberish to the console is easy. In Perl:

while (1) {
    print STDOUT "Perl child pid $$ writing to stdout.\n";
    sleep 1;
    print STDERR "Perl child pid $$ writing to stderr.\n";
    sleep 1;
}

The C version looks like this:

pid_t pid = getpid();

while (1) {
    fprintf(stdout,
            "C child pid %d writing to stdout.\n",
            pid);
    sleep(1);
    fprintf(stdout,
            "C child pid %d writing to stderr.\n",
            pid);
    sleep(1);
}

The real version sleeps a random amount of time of up to three seconds. See the git repository for details.

Create a Pair of Connected Sockets

This task is easy in Perl because the interpreter emulates socketpair(2) for systems that lack the call. However, enabling non-blocking read on the socket is a little bit tricky because the necessary ioctl constant FIONBIO is not available in Perl. You have to hardcode its value of 0x8004667e and hope that it will not change anytime soon.

You will find the following code in parent.c and parent.pl in the example repository.

For brevity, only the code for standard output is described here. The code for standard error is the same except for variable names and constants:

use constant FIONBIO => 0x8004667e;

socketpair my $stdout_read, my $stdout_write,
        AF_UNIX, SOCK_STREAM, PF_UNSPEC
    or die "cannot create socketpair: $!\n";

ioctl $child_stdout, FIONBIO, \$true
    or die "cannot set child stdout to non-blocking: $!";

In C, we have to roll our own version of socketpair(2):

#if IS_MS_DOS
# define socketpair(domain, type, protocol, sockets) \
    win32_socketpair(sockets)
static int win32_socketpair(SOCKET socks[2]);
static int convert_wsa_error_to_errno(int wsaerr);
#endif

See the source of parent.c for the implementation of convert_wsa_error_to_errno. Windows usually does not set errno in case of errors but you have to retrieve a proprietary error code with WSAGetLastError(). The function convert_wsa_error_to_errno() converts the relevant Windows error codes to standard error codes.

Let's now look at the socketpair(2) emulation win32_socketpair():

static int
win32_socketpair(SOCKET socks[2])
{
    SOCKET listener = SOCKET_ERROR;

    listener = WSASocket(AF_INET, SOCK_STREAM, PF_UNSPEC, NULL, 0, 0);
    if (listener < 0) {
        return SOCKET_ERROR;
    }

    ...
}

As you can see, the protocol is forced to AF_INET, the type to SOCK_STREAM and the protocol is unspecified (PF_UNSPEC). This differs from the regular version of socketpair(2) because that flexibility is not needed here.

It is crucial to use WSASocket() here, and not the standard BSD socket() function that is also availble for Windows. However, there is a subtle and seemingly undocumented difference. Only sockets created with WSASocket() can be used as system file descriptors of child processes.

The next step is to set the socket address to the loopback interface 127.0.0.1 and put the socket into listen mode. This is all standard network programming:

struct sockaddr_in listener_addr;
memset(&listener_addr, 0, sizeof listener_addr);
listener_addr.sin_family = AF_INET;
listener_addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
listener_addr.sin_port = 0;

errno = 0;
if (bind(listener, (struct sockaddr *) &listener_addr,
    sizeof listener_addr) == -1) {
    goto fail_win32_socketpair;
}

if (listen(listener, 1) < 0) {
    goto fail_win32_socketpair;
}

Setting the port number to 0 in line 5 has the effect that the operating system will pick a random free port.

The code at the lable fail_win32_socketpair just cleans up and frees the resources that have already been created. See the source code for details.

Next we have to create the read end called connector of the socket pair:

SOCKET connector = socket(AF_INET, SOCK_STREAM, 0);
if (connector == -1) {
    goto fail_win32_socketpair;
}

struct sockaddr_in connector_addr;
addr_size = sizeof connector_addr;
if (getsockname(listener, (struct sockaddr *) &connector_addr,
                &addr_size) < 0) {
    goto fail_win32_socketpair;
}
if (addr_size != sizeof connector_addr) {
    goto abort_win32_socketpair;
}

if (connect(connector, (struct sockaddr *) &connector_addr,
            addr_size) < 0) {
    goto fail_win32_socketpair;
}

Here, you are free to use standard BSD socket(2) and not WSASocket(), although it does not hurt to use the latter.

With the getsockname(2) call in line 8, we copy the connection info from the listener socket into that of the connector. This is necessary because we have to know the port number that the operating system had picked.

The call to connect(2) initiates the connection on the socket.

Now for the write end of the socket pair:

socklen_t addr_size;
SOCKET acceptor = accept(listener, (struct sockaddr *) &listener_addr, &addr_size);
if (acceptor < 0) {
    goto fail_win32_socketpair;
}
if (addr_size != sizeof listener_addr) {
    goto abort_win32_socketpair;
}

closesocket(listener);

if (getsockname(connector, (struct sockaddr *) &connector_addr,
                &addr_size) < 0) {
    goto fail_win32_socketpair;
}

if (addr_size != sizeof connector_addr
    || listener_addr.sin_family != connector_addr.sin_family
    || listener_addr.sin_addr.s_addr != connector_addr.sin_addr.s_addr
    || listener_addr.sin_port != connector_addr.sin_port) {
    goto abort_win32_socketpair;
}

The write end socket acceptor accepts the connection with accept(2).

The code at label abort_win32_socketpair sets errno to the closest equivalent of ECONNABORTED available on the system and then does the same clean-up as for fail_win32_socketpair.

The socket listener is now no longer needed. Note that you have to close it with closesocket() not close() because sockets are not files for Windows (which is pretty odd).

Lastly, we need to compare the connection info of the listener and connector socket in order to make sure that the IP address and port number are identical.

The last step is to return the two connected sockets to the caller:

sockets[0] = connector;
sockets[1] = acceptor;

return 0;

This code is inspired both by the socketpair(2) emulation present in Perl and example code for a selectable socket pair by Nathan Myers at https://github.com/ncm/selectable-socketpair.

Pass System Descriptors to the Child Process

This step differs between Perl and C.

Perl - Redirect Socket Pair to STDOUT/STDERR

Apparently, child processes created with Win32::Process::Create() automatically inherit the system file descriptors from the parent process. It is therefore necessary to temporarily replace STDOUT/STDERR with the write end of the respective socket pairs:

open SAVED_OUT, '>&STDOUT' or die "cannot dup STDOUT: $!\n";

open STDOUT, '>&' . $stdout_write->fileno
    or die "cannot redirect STDOUT to pipe: $!\n";

First, the regular standard out file descriptor is dup()ed to a new handle SAVED_OUT, and then the write end of the socketpair $stdout_write is dup() to STDOUT.

If at this point you want to print something to regular standard output you must use the copy SAVED_OUT!

C - Set STARTUPINFO

The C code does not have to dup(2) but write the corresponding descriptors into a structure STARTUPINFO that is passed to CreateProcess() later:

STARTUPINFO si;

memset(&si, 0, sizeof(si)); 
si.cb = sizeof(si);
si.hStdOutput = (HANDLE) stdout_sockets[1];
si.dwFlags = STARTF_USESTDHANDLES;

The corresponding fields in STARTUPINFO for standard error and standard input are hStdError and hStdInput respectively.

Finding out whether setting the dwFlags to STARTF_USESTDHANDLES is really necessary is left as an exercise to the reader until somebody generously leaves the answer in the comments section. For the time being, that line is just copied from a Microsoft code example.

The cast to a HANDLE in line 5 looks a little bit suspicious but is correct.

Apparently, a HANDLE was originally just an integer for a file descriptor. At one point the Microsoft developers must have decided that they want to store more information than just the descriptor number and declared a HANDLE to a pointer to void. Interestingly, these "pointers" point to very low addresses, something like 0x10a or so. That suggests that they are not really pointers but rather offsets into a table with open file descriptors, but this is just an educated guess.

Spawn the Child Process

Spawning the child process is quite similar in Perl and C:

Perl - Use Win32::Process::Create()

require Win32::Process;

my $process;
Win32::Process::Create(
    $process,
    $^X,
    "perl child.pl",
    0,
    0,
    '.',
) or die "cannot exec: ", Win32::FormatMessage(Win32::GetLastError());

my $child_pid = $process->GetProcessID;

The Perl binding of Win32::Process::Create() has a somewhat odd design in that it does not just return the created process object, but writes it into its first argument.

Arguments 2 and 3 in line 6-7 need a little explanation. Argument 2, which can be undef is defined as the absolute path to the actual executable (the special Perl variable $^X holds the absolute path to the Perl interpreter), whereas argument 3 is the complete command line.

If argument 2 is undef, then the first token of argument 3 is searched in $PATH and the rest is taken as arguments to the executable. However, I actually had to pass both argument 2 and argument 3 in order to execute a Perl script.

Be aware that you have to do the escaping of the command line in argument 3 yourself. The escaping mechanism is simple: You enclose arguments that contain spaces in double quotes. And double quotes (") are escaped as two double quotes (""). The same applies to the name of the executable (the first token in the command line).

The last argument '.' in line 10 is the current working directory of the child.

By the way, do not step into the trap of using Perl's exec() emulation! Win32::Process::Create is not like exec() but like a combination of fork() and exec() pretty much like posix_spawn. Just using a plain exec() will simply replace your current process with the child which is not what you want.

C - Use CreateProcess()

In C, things look very similar:

const char *cmd = "child.exe";
PROCESSINFO pi;

memset(&pi, 0, sizeof pi);

if (!CreateProcess(NULL, cmd, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi)) {
    printf("CreateProcess failed: %s.\n", strerror(errno));
    goto create_process_failed;
}

CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);

The first and second argument to CreateProcess() have the same semantics as arguments 2 and 3 in Perl. The first argument is optional, and contains the absolute path to the executable image on disk. The second one contains the complete command line (you must escape it yourself) and the first token is taken as the executable which is searched in $PATH.

The pointer to si is a pointer to a STARTUPINFO which contains the information about the file descriptors to inherit (see above).

Information about the process, like its pid, is returned in the PROCESSINFO structure.

Restore STDOUT and STDERR

Remember that the Perl version currently uses the write ends of the sockets for standard output and standard error. You have to restore that to the original descriptors:

if (!open STDERR, '>&SAVED_ERR') {
    print SAVED_ERR "cannot restore STDERR: $!\n";
    exit 1;
}

open STDOUT, '>&SAVED_OUT' or die "cannot restore STDOUT: $!\n";

This step is not necessary in C because no redirection takes place in the parent process.

Read Asynchronously from Children with Select

See the source files parent.c and parent.pl for instructions on how to asynchronously read the child process output. Just search for "select".

For Perl, when using select(), it is wise to bypass Perl's buffered I/O and use sysread() instead of read()!

Alternative Approach with an Intermediate Thread

As mentioned above, an alternative way to achieve the desired behaviour is to use an intermediate thread that reads from a pipe in blocking mode, and copies everything into the write end of the socket pair. This version is available at commit e9c71ac of the example repository.

Although, the intermediate thread is not necessary, that technique can still prove useful, when you want to turn an arbitrary asynchronous event source into a selectable file descriptor. You just let the thread wait for the event to get fired, and then write the relevant information into the write end of the socket pair so that it triggers an "data readable" on the read end of it.

I have used that technique in the Perl module AnyEvent::Filesys::Watcher. In the implementation for Windows which uses Filesys::Notify::Win32::ReadDirectoryChanges. This module creates a thread that synchronously waits for change events of the file system and communicates them via a Thread::Queue object to the main thread. My implementation passes a wrapper around Thread::Queue to the Windows implementation that is decorated with an additional socket pair and hooks into Thread::Queue's enqueue() and dequeue() for copying all communication into the socket.

Perl's fork() and exec() Emulations

Perl emulates fork() and exec() on Windows systems but you cannot use that for asynchronously reading and writing from child processes.

In fact, Perl's fork() for a Windows is a glorified CreateThread(). On the parent side it does not return the child process id but a (negative) thread id because there is no child process, just a thread. And when you try to naively kill that pseudo child process you are presented an unpleasant surprise: You normally kill yourself because you have sent the signal to the process group which includes the current process.

Because fork() does not create a new process but a new thread under Windows, exec() consequently does not replace the current process but instead just spawns the child process and terminates the thread created by the fork() invocation before.

However, you can benefit from this in situations where you want to communicate with the child process via an intermediate thread. If you do a fork(), you know that its just a thread but with its own private copy of the system file descriptors. Consequently, there is no need to save and restore the system file descriptors before spawning the child processes.

Usability of the Example Code

The example code has been tested on macOS with Perl 5.34 and Windows 10 with Strawberry Perl 5.32.1.1. You can use it as a starting point for your own applications, but you will probably improve the error handling and error reporting. Depending on your requirements you can also add code that kills child process on demand and frees resources unless you terminate immediately after the communication with the child processes.

If you have suggestions for improvements, please send me a pull request.

Summary

To do asynchronous I/O with child processes under Windows, follow these rules:

1. Communicate over the loopback interface with sockets instead of pipes.
2. When creating the socket pair (or using a socketpair(2) emulation), make sure that you use WSASocket() and not BSD socket(2) for creating the listener socket.
3. Do not use fork() or execvp() emulations but spawn the child process with the CreateProcess() family of functions.