1. Technology
You can opt-out at any time. Please refer to our privacy policy for contact information.

Discuss in my forum

Process Substitution and Named Pipes

Redirecting Multiple Streams

By

One of the advantages of the command line is that is provides flexible ways to feed output from one program to another.

The "pipe" and "redirection" methods enable this for the cases where there is just one data stream. For example, the output stream of a wc command may be passed on to a grep command, the output of which may then be passed on to another process, etc. The basic pipe and redirection methods connect the stdout of one process to the stdin of another process.

For example, if you use the "cat" command to concatenate two files "table1" and "table2", you can send the result to a file instead of stdout (the screen) as follows:

cat table1.csv table2.csv > table12.csv

The ">" redirection operator can also be used to empty a file or create a new file by simply not specifying the source:

> all.csv

This remove all content from the file "all.csv" if it exists, otherwise will create a new file with that name.

The pipe operator "|" is used to pass a stream of data from process to another, that is, the stdout of the first process is used as stdin of the second. In this example the concatenation of the two file table1.csv and table2.csv is sent to the word count program "wc":

cat table1.csv table2.csv | wc

The Bash shell provides more general methods to handle cases where a program requires multiple inputs, for example when comparing two files/streams, or when a program has multiple outputs that need to be "piped" to other processes.

The process substitution method enables you to substitute program parameters with processes, such that the output of these processes are considered the parameter values.

For example the command diff3 compares the contents of three files and outputs desriptions the differences between them. If you want to compare the contents of three directories, that is, listings of the files and directories in each, you can use the process substitution method as follows:

diff3 <(ls $dir1) <(ls $dir2) <(ls $dir3)

Where $dir1, $dir2, and $dir3 are variables that have been set to the specific directories being compared. Each process substitution expression is of the form <(...), where the less-than symbol immediately precedes the parenthesis term. There must be no space between the < and the left parenthesis.

If a command parameter specifies an output file, process substitution can be applied as well. In this case the output parameter is replaced with an expression of the form >(...), which contains a command that uses the output being generated for this (virtual) file as input. As an example consider the following command line:

tar cfv >(bzip2 -c > dir1.tar.bz2) $dir1

It executes the program tar with the options cfv. The c stands for create, the f stands for file, and the v stands for verbose. The command expects two arguments, the first one specifies the output file, and the second one the directory is being "archived". The first argument has been replaced with a process substitution expression. The command inside the parenthesis is applied to the (virtual) file being written by the tar command. That is, it is used as stdin for the bzip2 command, which compresses it and output the result to a new file named "dir1.tar.bz2".

A cool feature of Linux, you may not have heard about yet, is the "Named Pipe" mechanism. It can be used to connect programs that were not necessarily designed to work together. Oftentimes programmers use files to accomplish that, which is kind of clunky and inefficient. Named pipes look similar to files but provide an elegant method to pass data directly from one process to another without the overhead of writing the data to disk and then reading the data from disk.

Named pipes are also called FIFO special files, where FIFO stands for First In First Out. It provides a location in the file system where two or more processes can exchange data without actually using files. For example, with the command mkfifo fifo1 you can create a pipe called fifo1. Then you can have a program or command write to that pipe as if it were a file. However, nothing is going to happen until another process tries to read from that pipe.

For example, the command ps > fifo1 will try to send its output to fifo1. If you then enter another command, such as more < fifo1, the output from ps will be passed to more using the pipe fifo1.

  1. About.com
  2. Technology
  3. Linux
  4. Linux HowTos
  5. Bash How-To's
  6. Bash Process Substitution and Named Pipes

©2014 About.com. All rights reserved.