Standard file streams

6 April 2021

On Unix-like systems, file streams are used to redirect input or output. If you have spent some time on the command line then you have probably used input and/or output redirection, perhaps without understanding the concept. This article explains the basics and includes various examples. It will teach you why a command like mysql example_db < example_db.sql imports a database dump and why you see stuff like > /dev/null 2>&1 in cron jobs.

STDIN, STDOUT and STDERR

There are three standard file streams, or descriptors, that are always open for use when you execute a command: standard input (stdin), standard output (stdout) and standard error (stderr).

Usually, stdin is your keyboard. For instance, when you type the command ls your keyboard provides the input. However, the input can also come from another command. Similarly, stdout is usually the terminal window: after you enter ls the command’s output is printed to the screen. As with stdin it is possible to redirect the output elsewhere (for stance, to a file).

Each descriptor has a numeric representation: stdin is 0; stdout is 1 and stderr is 2.

I/O redirection

You can redirect file streams so that your command gets input from either a file or another command (instead of your keyboard). For example, when you import a MySQL/MariaDB database you use input redirection to read an SQL dump file file into your mysql command:

$ mysql example_db < example_db.sql

The input redirection happens at the less than sign (<). The command reads: redirect the file example_db.sql to the command mysql example_db.

Similarly, when you export a database you use output redirection:

$ mysqldump example_db > example_db.sql

This command redirects the output of the command mysqldump example_db to a file named example_db.sql.

Redirecting errors

By default, any errors thrown by a command are printed to the screen. If you want to redirect stderr to a separate file you can use its file descriptor (2) followed by a greater than symbol (>) and the name of the file errors should be written to:

$ command 2> error.log

If you want both stdout and stderr to go the same place then you can use a special shorthand:

# command > output.log 2>&1

In the Bash shell you can shorten this to >&:

$ command >& output.log

Redirection examples

To illustrate how input and output redirection works I will use a command that both triggers an error and produces some output. The below find command uses the -maxdepth option, which limits how deep the find command dives into a directory tree. However, the -maxdepth option should come before any other options – if you don’t do so the command prints a warning:

$ find ~/git/ -type f -name README.md -maxdepth 2
find: warning: you have specified the -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it).  Please specify options before other arguments.
 
/home/c2/git/blockbot/README.md
/home/c2/git/dotfiles/README.md
/home/c2/git/errorbot/README.md

The find command still produces output: it found various files named README.md in the ~/git directory. So, we can now play with redirected stdout and stderr.

To redirect error messages to a separate file you can add output redirection for stderr:

$ find ~/git/ -type f -name README.md -maxdepth 2 2> error.log
/home/c2/git/blockbot/README.md
/home/c2/git/dotfiles/README.md
/home/c2/git/errorbot/README.md

The warning is no longer printed to the screen. Instead, it was redirected to the error.log file:

$ cat error.log
find: warning: you have specified the -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it).  Please specify options before other arguments.

You can of course also write just stdout to a file:

$ find ~/git/ -type f -name README.md -maxdepth 2 1> files
find: warning: you have specified the -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it).  Please specify options before other arguments.

The above command prints the warning (stderr) and doesn’t list any files found by the command. The output (stdout) was instead written to a file named files:

$ cat files
/home/c2/git/blockbot/README.md
/home/c2/git/dotfiles/README.md
/home/c2/git/errorbot/README.md

This type of output redirection is very common. So common, in fact, that you don’t have to use 1> to redirect stdout to a file. You can instead simply use >.

You can also write stdout to one file and stderr to another file:

$ find ~/git/ -type f -name README.md -maxdepth 2 > files 2> error.log
 
$ cat files 
/home/c2/git/blockbot/README.md
/home/c2/git/dotfiles/README.md
/home/c2/git/errorbot/README.md
 
$ cat error.log 
find: warning: you have specified the -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it).  Please specify options before other arguments.

And finally, you can of course also write stdout and stderr to the same file. The easiest way to do so is by using the 2>&1 shorthand:

$ find ~/git/ -type f -name README.md -maxdepth 2 > find.log 2>&1

It’s useful to understand the syntax. First, you tell the shell that you want to redirect the output to a file named find.log and you then specify that you want to redirect both stderr and stdout. Understanding this makes it easier to get your head round cron jobs, where the output is typically redirected to the /dev/null file (which I will get to shortly).

Redirection vs appending

When you redirect output to a file using > any existing content in the file is overwritten. If you want to instead append the output then you can use two greater than signs. The below command overwrites (or creates) the file named files and appends any errors to the error.log file:

$ find ~/git/ -type f -name README.md -maxdepth 2 > files 2>> error.log

Redirecting output to /dev/null

So far all the examples wrote stdout and/or stderr to a file. If you want to discard any output then you can instead redirect it to /dev/null. This is a special file whose sole purpose is to delete any data written to it.

You most commonly see this type of redirection in cron jobs. For instance, here is a cron job that redirects both stdout and stderr to /dev/null:

# crontab -l | grep blockbot
*/30 * * * * /usr/bin/bash /root/blockbot --checkload > /dev/null 2>&1