Input and output

Everything is a file

File descriptors

Unix follows a simple rule: everything is a file. A file on the disk, of course, but also the terminal, a device (/dev/null, /dev/urandom), a pipe between two programs, a network connection. All of them are read and written with the same two functions, read and write.

For the program, such an open resource is designated by a simple integer, the file descriptor (often abbreviated fd). It is the index of a cell in the table of open resources that the kernel keeps for each process.

Three descriptors are already open when any program starts:

The functions of this chapter are not part of C, but of the POSIX standard, that of Unix systems. With -std=c11, gcc only declares standard C; to also get POSIX, compile with -D_POSIX_C_SOURCE=200809L. All the programs of the chapter are compiled like this:

$ gcc -std=c11 -Wall -Wextra -Werror -D_POSIX_C_SOURCE=200809L prog.c -o prog

read and write

#include <unistd.h>

ssize_t read(int fd, void *buf, size_t n);
ssize_t write(int fd, const void *buf, size_t n);

• read(fd, buf, n) reads at most n bytes from fd, and stores them at address buf. It returns the number of bytes read.
• write(fd, buf, n) writes at most n bytes taken from address buf. It returns the number of bytes written.

These are system calls: kernel functions, which the program calls directly. Their manual page is in section 2 (man 2 read), those of the C library in section 3 (man 3 printf). People often write read(2) or printf(3) to say which section is meant.

#include <unistd.h>

int main(void)
{
    write(STDOUT_FILENO, "bonjour\n", 8);
    write(STDERR_FILENO, "erreur\n", 7);
    return 0;
}

$ ./bonjour
bonjour
erreur
$ ./bonjour > sortie.txt
erreur
$ cat sortie.txt
bonjour

The shell redirected standard output into the file; standard error stayed on the terminal. That is the whole point of two separate outputs: error messages do not get mixed up with results.

ssize_t is a signed integer the size of size_t: it must be able to return -1.

Errors, and errno

On failure, a system call returns -1, and stores in the global variable errno (declared by <errno.h>) a code that says why. perror("msg") prints msg: followed by the explanation in plain words, and strerror(errno) returns that explanation as a string. The most common codes for read and write:

• EBADF: fd is not an open descriptor (or not in the right direction);
• EFAULT: buf points outside the program's memory;
• EIO: hardware input/output error.

Two codes are not real errors, you just have to try again:

• EINTR: the call was interrupted by the arrival of a signal (chapter 6) before it could read anything;
• EAGAIN: the descriptor is in non-blocking mode, and there is nothing to read for now. You only meet it if you asked for that mode.

Partial reads, end of file

Three rules govern read:

1. It is blocking: if there is nothing to read (a terminal where nothing has been typed, an empty pipe), the program is put to sleep until data arrives.
2. It returns what is available, even if that is less than what was asked for. On a disk file, you generally get everything; on a terminal, you get a line; on a pipe or the network, what has arrived.
3. It returns 0 at the end of the file. That is the only reliable way to detect it.

write too may write less than asked (on a full pipe, a network connection, a full disk). So the result of read and write must always be looked at. Reading exactly n bytes takes a loop:

size_t pos = 0;
while (pos < n)
{
    ssize_t r = read(fd, buf + pos, n - pos);
    if (r == 0)
        break;                  /* end of file before n bytes */
    if (r == -1)
    {
        if (errno == EINTR)
            continue;
        /* real error: handle it */
    }
    pos += r;
}

A cat

The cat command copies to standard output the files it is given, or its standard input if it is given none, or if one of them is called -. Let us write it.

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define TAILLE 4096

/* Writes the n bytes of buf, even if write only takes part of them. */
static int ecrire_tout(int fd, const char *buf, size_t n)
{
    while (n > 0)
    {
        ssize_t w = write(fd, buf, n);
        if (w == -1)
        {
            if (errno == EINTR)
                continue;
            return -1;
        }
        buf += w;
        n -= w;
    }
    return 0;
}

/* Copies fd to standard output. Returns 0, or -1 on error. */
static int copier(int fd)
{
    char    buf[TAILLE];
    ssize_t r;

    while ((r = read(fd, buf, TAILLE)) != 0)
    {
        if (r == -1)
        {
            if (errno == EINTR)
                continue;
            return -1;
        }
        if (ecrire_tout(STDOUT_FILENO, buf, r) == -1)
            return -1;
    }
    return 0;
}

int main(int argc, char *argv[])
{
    int statut = 0;

    if (argc < 2)
        return copier(STDIN_FILENO) == -1 ? 1 : 0;

    for (int i = 1; i < argc; i++)
    {
        int fd = STDIN_FILENO;
        if (strcmp(argv[i], "-") != 0)
        {
            fd = open(argv[i], O_RDONLY);
            if (fd == -1)
            {
                fprintf(stderr, "moncat: %s: %s\n", argv[i], strerror(errno));
                statut = 1;
                continue;
            }
        }
        if (copier(fd) == -1)
        {
            fprintf(stderr, "moncat: %s: %s\n", argv[i], strerror(errno));
            statut = 1;
        }
        if (fd != STDIN_FILENO)
            close(fd);
    }
    return statut;
}

$ ./moncat a.txt b.txt
ligne 1
ligne 2
fin
$ echo "du clavier" | ./moncat a.txt - b.txt
ligne 1
ligne 2
du clavier
fin
$ ./moncat a.txt absent.txt b.txt
ligne 1
ligne 2
moncat: absent.txt: No such file or directory
fin
$ echo $?
1
$ ./moncat /tmp
moncat: /tmp: Is a directory

A few choices, which correct the original version:

• We read in blocks of 4096 bytes, not byte by byte: each system call is expensive (the program hands over to the kernel), so as few as possible must be made.
• Each block read is written in full, thanks to ecrire_tout.
• A file that cannot be opened does not stop the program: it is reported, we move on to the next one, and the exit status will say there was a problem. This is how the real cat behaves.
• The error message is built right after the failure, with strerror(errno). The original built the message in advance with malloc, strncpy and strncat (so that no call would modify errno between the failure and perror): it was fragile, and the message was never freed in case of error.
• Each file opened is closed again: a process is only entitled to a limited number of descriptors (often 1024), and a cat of two thousand files would fail.

Opening, closing, seeking

open and close

#include <fcntl.h>

int open(const char *chemin, int drapeaux);
int open(const char *chemin, int drapeaux, mode_t mode);

#include <unistd.h>

int close(int fd);

open returns a new descriptor, always the lowest free number, or -1 on failure. close releases it. The flags say how to open. Exactly one of these is required:

• O_RDONLY: read-only;
• O_WRONLY: write-only;
• O_RDWR: read and write.

You can add to it, with the “bitwise or” operator |:

• O_CREAT: creates the file if it does not exist. It is the only case where the third parameter, mode, is used;
• O_TRUNC: if the file exists, empties it;
• O_APPEND: each write goes to the end of the file;
• O_EXCL: with O_CREAT, fails if the file already exists.

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    int fd = open("essai.txt", O_WRONLY | O_CREAT | O_TRUNC, 0666);
    if (fd == -1)
    {
        perror("essai.txt");
        return 1;
    }
    write(fd, "abc\n", 4);
    close(fd);
    return 0;
}

This is what the shell does for > essai.txt.

Permissions, and the umask

The mode gives the permissions of the created file, in octal (the leading 0 tells the compiler so): 0666 is rw-rw-rw-, read and write for everyone. But the kernel removes the permissions forbidden by the process's umask, a mask inherited from the shell: the actual permissions are mode & ~umask.

$ umask
0002
$ ./creer; ls -l essai.txt
-rw-rw-r-- … essai.txt
$ umask 077; rm essai.txt; ./creer; ls -l essai.txt
-rw------- … essai.txt

With a umask of 002, the write permission of “others” is removed; with 077, all the permissions of the group and of others. So always give 0666 for a data file (0777 for an executable or a directory), and let the user decide with their umask.

lseek

For a disk file, the kernel keeps a current position: each read or write of k bytes moves it forward by k. It is common to reading and writing, and shared by all the descriptors that designate the same opening of the file (after a dup, or a fork). lseek moves it:

off_t lseek(int fd, off_t decalage, int depart);

depart is:

• SEEK_SET: decalage counts from the start of the file;
• SEEK_CUR: from the current position;
• SEEK_END: from the end of the file.

lseek returns the new position, counted from the start, or -1. So lseek(fd, 0, SEEK_END) returns the size of the file, and lseek(fd, 0, SEEK_CUR) the current position. off_t is a signed integer, 64 bits wide on Linux. On a pipe or a terminal, where “moving” makes no sense, lseek fails with ESPIPE.

You can position yourself after the end of the file. If you write there, the hole between the old end and the new write reads as null bytes, but takes up no space on the disk:

int fd = open("troue.bin", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, "debut", 5);
off_t pos = lseek(fd, 1000000, SEEK_CUR);
write(fd, "fin", 3);
printf("position après le saut : %lld\n", (long long)pos);
printf("taille : %lld\n", (long long)lseek(fd, 0, SEEK_END));
close(fd);

position après le saut : 1000005
taille : 1000008
$ du -h troue.bin
8,0K    troue.bin

The file is a million bytes long, but only takes up 8 KiB on the disk: it is a sparse file. On Linux, lseek also accepts SEEK_DATA and SEEK_HOLE, to jump from one data area to the next; not all file systems support them.

Redirecting

dup and dup2

int dup(int ancien);
int dup2(int ancien, int nouveau);

• dup(ancien) returns a new descriptor (the lowest free one) that designates the same resource as ancien.
• dup2(ancien, nouveau) makes nouveau a copy of ancien. If nouveau was open, it is closed first.

After a dup, the two descriptors share everything, current position included. The resource is only really closed when all the descriptors that designate it have been closed.

This is how the shell does its redirections. For prog > journal.txt, it opens journal.txt, makes it descriptor 1 with dup2, then starts prog, which writes to its standard output without knowing anything. You can do the same in a program, and even go back, by keeping a copy of the old output:

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    int sauve = dup(STDOUT_FILENO);
    int fd = open("journal.txt", O_WRONLY | O_CREAT | O_TRUNC, 0666);
    if (fd == -1)
    {
        perror("journal.txt");
        return 1;
    }

    dup2(fd, STDOUT_FILENO);
    close(fd);
    write(STDOUT_FILENO, "dans le fichier\n", 16);

    dup2(sauve, STDOUT_FILENO);
    close(sauve);
    write(STDOUT_FILENO, "à l'écran\n", 12);
    return 0;
}

$ ./redir
à l'écran
$ cat journal.txt
dans le fichier

Once dup2 is done, fd is no longer needed: the file stays open through descriptor 1. We close it straight away. (The string "à l'écran\n" is 12 bytes long and not 10: à and é each take two bytes in UTF-8.)

Pipes

A queue between two processes

A pipe is a one-way communication channel: you write at one end, you read at the other, in the order of writing. This is what the shell builds for ls | wc -l: the output of ls becomes the input of wc.

int pipe(int t[2]);

pipe creates the pipe and stores two descriptors in the array: t[0] is the output of the pipe (where you read), t[1] its input (where you write). To remember it: 0 is standard input, where you read; 1 standard output, where you write.

• Reading is destructive: a byte that is read is removed from the pipe.
• The pipe has no “author”: a process that reads from a pipe it has just written to reads what it wrote itself. So each participant must clearly be a reader or a writer, and two-way communication requires two pipes.
• A pipe only makes sense between two processes: you create it, then call fork (chapter 6), which duplicates the process with its descriptors. Parent and child then have access to both ends of the pipe.

Closing the unused ends

Two rules make closing indispensable:

• read on an empty pipe only returns 0 (end of file) when all the inputs of the pipe are closed, in every process. As long as one remains open, someone could still write: read waits.
• write on a pipe whose outputs are all closed kills the process with the SIGPIPE signal: no one will ever read.

So each process closes, right after the fork, the end it does not use. Forgetting to close its t[1] in the reading process means putting it to sleep forever: it waits for an end of file that it itself prevents from arriving.

#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    int  t[2];
    char c;
    pipe(t);
    if (fork() != 0)
    {
        close(t[0]);
        write(t[1], "x", 1);
        close(t[1]);
        wait(NULL);
    }
    else
    {   /* forgotten: close(t[1]); */
        while (read(t[0], &c, 1) > 0)
            printf("lu : %c\n", c);
        printf("fin du fils\n");
    }
    return 0;
}

$ ./tube_bloque
lu : x
^C

The child reads x, then waits indefinitely; the parent waits for the child in wait. You have to interrupt with Ctrl-C. (If the output is redirected into a file, you do not even see lu : x: the line was still in printf's buffer when the program was killed, see below.)

And the other way round:

int t[2];
pipe(t);
close(t[0]);
printf("avant\n");
fflush(stdout);
write(t[1], "x", 1);
printf("après\n");

$ ./sigpipe
avant
$ echo $?
141

après is never printed: write killed the process. The shell returns 128 + the number of the signal that killed the program, and SIGPIPE is signal 13.

A complete example

The parent sends a message to the child through a pipe, eight bytes at a time; the child reads it four bytes at a time and copies it to standard output. Nothing forces both sides to read and write in blocks of the same size: the pipe is a stream of bytes, with no division.

#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>

static void lecteur(int fd)
{
    char    buf[4];
    ssize_t r;
    while ((r = read(fd, buf, sizeof buf)) != 0)
    {
        if (r == -1)
        {
            if (errno == EINTR)
                continue;
            perror("lecteur");
            return;
        }
        write(STDOUT_FILENO, buf, r);
    }
}

static void redacteur(int fd, const char *texte)
{
    size_t reste = strlen(texte);
    while (reste > 0)
    {
        ssize_t w = write(fd, texte, reste < 8 ? reste : 8);
        if (w == -1)
        {
            if (errno == EINTR)
                continue;
            perror("redacteur");
            return;
        }
        texte += w;
        reste -= w;
    }
}

int main(void)
{
    int t[2];
    if (pipe(t) == -1)
    {
        perror("pipe");
        return 1;
    }
    if (fork() != 0)
    {   /* the parent writes */
        close(t[0]);
        redacteur(t[1], "Un message qui traverse le tube, huit octets par huit.\n");
        close(t[1]);
        wait(NULL);
    }
    else
    {   /* the child reads */
        close(t[1]);
        lecteur(t[0]);
        close(t[0]);
    }
    return 0;
}

$ ./tube
Un message qui traverse le tube, huit octets par huit.

The parent closes its input of the pipe before waiting for the child: that is what allows the child to see the end of file and terminate. In the other order, each would wait for the other.

The input and output of stdio.h

Buffers

A system call is expensive: control has to be handed over to the kernel, and come back. Writing a file character by character with write would be very slow. Managing blocks yourself, as in moncat, quickly becomes tedious. The C library does it for us: that is <stdio.h>.

A FILE * hides a descriptor and a buffer, an area of memory. printf, fputs, fputc write into the buffer; the contents are only sent to the kernel, by a single write, when the buffer is full, or in certain specific cases. Likewise, fgets or scanf read a big block in one go, and then draw from it. stdin, stdout and stderr are the FILE * of descriptors 0, 1 and 2; fopen and fclose open and close the others.

When is the buffer flushed?

• stdout to a terminal: at each end of line (line buffering);
• stdout to a file or a pipe: when the buffer is full, a few kilobytes (full buffering);
• stderr: never buffered, each write goes out immediately;
• in every case: when fflush is called, when the file is closed, and at the normal end of the program (return from main or exit).

The traps

The buffer makes writing deferred: a printf does not write at the moment it is called. As long as you only use stdio.h, you do not notice. But if you mix printf and write:

printf("un ");
write(STDOUT_FILENO, "deux ", 5);
printf("trois\n");

$ ./tampon
deux un trois

un stayed in the buffer (no end of line yet), while deux went out directly. Do not mix the two on the same descriptor, or call fflush(stdout) before each write.

More surprising, with fork, which duplicates all the memory of the process, buffers included:

#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    printf("avant le fork\n");
    if (fork() == 0)
        return 0;
    wait(NULL);
    return 0;
}

$ ./tampon_fork
avant le fork
$ ./tampon_fork > f.txt
$ cat f.txt
avant le fork
avant le fork

On the terminal, the line went out as soon as the \n came, before the fork. Redirected into a file, it was waiting in the buffer; the fork copied the buffer, and each of the two processes flushed it when terminating. The same program does not write the same thing depending on where its output goes. Hence the rule: fflush(stdout) before any fork.

A file in memory: mmap

Rather than reading a file into an array, you can ask the kernel to map it into the process's memory: the file appears as an array of bytes, which you walk through with pointers. The kernel loads the pages of the file as you touch them (virtual memory, in chapter 3).

#include <sys/mman.h>

void *mmap(void *adresse, size_t longueur, int protection, int drapeaux,
           int fd, off_t decalage);
int   munmap(void *adresse, size_t longueur);

Counting the lines of a file:

#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    if (argc != 2)
    {
        fprintf(stderr, "usage : %s fichier\n", argv[0]);
        return 1;
    }
    int fd = open(argv[1], O_RDONLY);
    struct stat s;
    if (fd == -1 || fstat(fd, &s) == -1)
    {
        perror(argv[1]);
        return 1;
    }
    if (s.st_size == 0)
    {
        printf("0 ligne\n");
        return 0;
    }
    char *p = mmap(NULL, s.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
    close(fd);
    if (p == MAP_FAILED)
    {
        perror("mmap");
        return 1;
    }
    size_t lignes = 0;
    for (off_t i = 0; i < s.st_size; i++)
        if (p[i] == '\n')
            lignes++;
    printf("%zu lignes\n", lignes);
    munmap(p, s.st_size);
    return 0;
}

$ ./compter moncat.c
77 lignes
$ wc -l moncat.c
77 moncat.c

• fstat gives information about an open file, including its size, st_size.
• PROT_READ: we will only read. MAP_PRIVATE: if we wrote, the changes would remain private to the process; with MAP_SHARED, they would go into the file.
• On failure, mmap does not return NULL but MAP_FAILED.
• Once the mapping is made, the descriptor can be closed: the mapping remains valid until munmap.
• An empty file (length 0) cannot be mapped, hence the test.

mmap is also used to allocate memory without a file (chapter 3), and to share an area of memory between processes.