System Calls

The kernel implements a programming interface called the system call interface. User-space programs cannot directly access hardware or kernel data structures — they must request privileged operations by crossing the user/kernel boundary via a system call.

How System Calls Work

User mode vs. kernel mode

The CPU operates in two privilege levels (rings on x86):

Normal application code runs in user mode. System calls are the controlled gateway into kernel mode.

The trap mechanism

A system call is initiated by a trap (software interrupt). On x86-32 this is int 0x80; on x86-64 it is the syscall instruction. The sequence:

1. User space: Program loads the syscall number into rax (x86-64) or eax (x86-32) and arguments into the designated registers, then executes syscall / int 0x80.
2. Hardware: Saves the current instruction pointer, stack pointer, and CPU flags onto the kernel stack; switches privilege level to ring 0; jumps to the kernel’s trap handler (whose address was registered at boot).
3. Kernel: Looks up the syscall number in the syscall table, calls the corresponding kernel function, places the return value in rax/eax.
4. Hardware: On sysret / iret, restores user-mode registers and privilege level, resumes the program at the instruction after the trap.

Trap table (boot-time setup)

When the machine boots, the OS runs in kernel mode and registers the addresses of its trap handlers with the hardware — one handler per exception type (syscall, page fault, divide-by-zero, hardware interrupt, etc.). This is stored in the Interrupt Descriptor Table (IDT). Once set, the hardware uses this table for the lifetime of the boot until reboot. This is why user code cannot redirect syscalls — only the kernel can modify the IDT.

Syscall numbers

Every syscall has an integer ID. The kernel maintains a syscall table that maps each number to a kernel function pointer.

x86-64 calling convention for syscalls

On error, rax holds a negative errno value (e.g., -2 for ENOENT).

C library wrappers

Most programs never invoke syscalls directly. The C standard library (glibc on Linux) provides a wrapper function for every syscall. The wrapper:

• Marshals arguments into the correct registers
• Executes the trap instruction
• Translates a negative return value into errno and returns -1

ssize_t write(int fd, const void *buf, size_t count);
// ↑ glibc wrapper — internally executes: mov rax, 1; syscall

Direct syscall invocation is done with syscall(2):

#include <sys/syscall.h>
#include <unistd.h>

long ret = syscall(SYS_write, 1, "hello\n", 6);

Process States

A process moves through these states during its lifetime:

           fork()
             │
         ┌───▼───┐
         │ Ready │ ◄────── waiting for CPU
         └───┬───┘
  scheduled  │  preempted
         ┌───▼───┐
         │Running│ ── I/O or event wait ──► Blocked
         └───┬───┘                          │
             │                              │ I/O complete
             │ exit()           ┌───────────┘
         ┌───▼───┐
         │Zombie │ ◄── exit called; waiting for parent to wait()
         └───┬───┘
             │ parent calls wait()
          (reaped)

Process Management Syscalls

fork

Creates a new child process that is an almost-exact copy of the parent.

• The child inherits the parent’s address space (copy-on-write), CPU register state, open file descriptors, signal handlers, and environment.
• The OS creates a new PCB (Process Control Block / task_struct) and assigns the child a new PID.
• Both parent and child resume execution at the return of fork().

Return values:

#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

int main(void)
{
    pid_t pid = fork();

    if (pid < 0) {
        perror("fork failed");
        return 1;
    } else if (pid == 0) {
        /* child */
        printf("Child: my PID is %d, parent PID is %d\n", getpid(), getppid());
    } else {
        /* parent */
        printf("Parent: my PID is %d, child PID is %d\n", getpid(), pid);
    }

    return 0;
}

exec

The exec family replaces the calling process’s image with a new program loaded from a binary file. The PID is preserved but the address space, code, data, heap, and stack are completely replaced. Open file descriptors survive unless marked O_CLOEXEC.

exec variants:

execve is the actual syscall; all others are C library wrappers built on top of it.

#include <unistd.h>
#include <stdio.h>

int main(void)
{
    char *args[] = { "ls", "-lh", "/home", NULL };

    execvp("ls", args);

    /* exec only returns on failure */
    perror("execvp failed");
    return 1;
}

fork + exec is the standard pattern for spawning a new program:

pid_t pid = fork();
if (pid == 0) {
    /* child: replace image with new program */
    execvp("ls", (char *[]){ "ls", "-la", NULL });
    perror("exec failed");
    _exit(1);
}
/* parent continues here */

exit

Terminates the calling process.

#include <stdlib.h>
void exit(int status);     // flushes stdio buffers, runs atexit() handlers, then calls _exit()

#include <unistd.h>
void _exit(int status);    // raw syscall — immediate termination, no cleanup

On exit:

1. The OS closes all file descriptors and releases most resources.
2. A SIGCHLD signal is delivered to the parent.
3. The process moves to the zombie state — the PCB remains in the process table with exit status info until the parent calls wait.

wait and waitpid

A parent reaps a zombie child by calling wait or waitpid. This removes the child’s entry from the process table and retrieves its exit status.

#include <sys/wait.h>

pid_t wait(int *status);
pid_t waitpid(pid_t pid, int *status, int options);

wait — blocks until any child exits; returns the child’s PID.

waitpid — more flexible:

options:

• WNOHANG — return immediately if no child has exited (non-blocking)
• WUNTRACED — also return if a child has stopped (not just exited)

Status macros:

int status;
pid_t child = waitpid(-1, &status, 0);

if (WIFEXITED(status))
    printf("exited normally, status %d\n", WEXITSTATUS(status));
else if (WIFSIGNALED(status))
    printf("killed by signal %d\n", WTERMSIG(status));

Full fork/exec/wait example:

#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    pid_t pid = fork();

    if (pid < 0) {
        perror("fork");
        return 1;
    } else if (pid == 0) {
        execvp("ls", (char *[]){ "ls", "-la", NULL });
        perror("exec");
        _exit(1);
    }

    int status;
    waitpid(pid, &status, 0);

    if (WIFEXITED(status))
        printf("child exited with status %d\n", WEXITSTATUS(status));

    return 0;
}

If a parent exits before reaping its children, the children are reparented to PID 1 (init / systemd), which reaps them automatically.

getpid / getppid

#include <unistd.h>

pid_t getpid(void);    // PID of the calling process
pid_t getppid(void);   // PID of the calling process's parent

File I/O Syscalls

Everything in Unix is a file — regular files, devices, sockets, and pipes are all accessed through file descriptors using the same four calls.

File descriptors

Every process starts with three open file descriptors:

open

#include <fcntl.h>
int open(const char *path, int flags, mode_t mode);

Returns a file descriptor (non-negative integer) on success, -1 on error.

Common flags:

mode sets file permissions when O_CREAT is used (e.g., 0644):

int fd = open("output.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0) {
    perror("open");
    exit(1);
}

read

#include <unistd.h>
ssize_t read(int fd, void *buf, size_t count);

Reads up to count bytes from fd into buf. Returns the number of bytes actually read, 0 on EOF, -1 on error. May return fewer bytes than requested (short read) — always loop:

char buf[4096];
ssize_t n;
while ((n = read(fd, buf, sizeof(buf))) > 0) {
    /* process n bytes */
}
if (n < 0)
    perror("read");

write

#include <unistd.h>
ssize_t write(int fd, const void *buf, size_t count);

Writes up to count bytes from buf to fd. Returns bytes written, or -1 on error. Like read, may write fewer than count bytes — loop on short writes:

const char *msg = "Hello, World!\n";
size_t total = strlen(msg);
size_t written = 0;
while (written < total) {
    ssize_t n = write(STDOUT_FILENO, msg + written, total - written);
    if (n < 0) { perror("write"); break; }
    written += n;
}

close

#include <unistd.h>
int close(int fd);

Releases the file descriptor. Always check the return value — close can fail (e.g., flushing buffered writes to disk on NFS). Always close file descriptors when done to avoid FD leaks.

if (close(fd) < 0)
    perror("close");

Signal Syscalls

Signals are asynchronous notifications delivered to a process by the kernel or another process.

Common signals

kill

Despite the name, kill sends any signal to any process (not just SIGKILL):

#include <signal.h>
int kill(pid_t pid, int sig);

kill(child_pid, SIGTERM);   // ask child to terminate gracefully
kill(child_pid, SIGKILL);   // force kill (cannot be blocked)

sigaction

The preferred way to install a signal handler (more portable and controllable than signal()):

#include <signal.h>

void handler(int sig) {
    /* sig is the signal number */
    write(STDOUT_FILENO, "caught signal\n", 14);  /* async-signal-safe only */
}

int main(void)
{
    struct sigaction sa = {
        .sa_handler = handler,
        .sa_flags   = SA_RESTART,   /* restart interrupted syscalls */
    };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGINT, &sa, NULL);

    /* ... */
}

SA_RESTART causes syscalls interrupted by the signal to restart automatically rather than returning EINTR.

Only async-signal-safe functions may be called inside a signal handler (e.g., write, _exit, kill). printf, malloc, and most library functions are not safe inside handlers.

Memory Syscalls

brk / sbrk

brk sets the program break — the top of the heap. sbrk moves it by a relative increment. malloc uses these internally.

#include <unistd.h>

void *sbrk(intptr_t increment);  // returns old break address
int   brk(void *addr);           // sets break to addr

Direct use is rare — always prefer malloc/free.

mmap

Maps a file or anonymous memory region into the process’s address space:

#include <sys/mman.h>

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
int   munmap(void *addr, size_t length);

Common uses:

• Anonymous memory (alternative to malloc for large allocations): fd = -1, flags = MAP_PRIVATE | MAP_ANONYMOUS
• File mapping (load a file directly into memory): flags = MAP_SHARED or MAP_PRIVATE, fd = open file descriptor

/* allocate 4096 bytes of anonymous memory */
void *mem = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (mem == MAP_FAILED) {
    perror("mmap");
    exit(1);
}

/* map a file read-only */
int fd = open("data.bin", O_RDONLY);
void *data = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);  /* fd can be closed after mmap */

prot flags: