Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

🏠 Back to Blog

Linux Observability Sources

These interfaces provide the data for observability tools on Linux:

/proc - per-process counters /proc, ‘sys’ - system-wide counters /sys - device configuration and counters /sys/fs/cgroup - cgroup statistics ptrace - per-process tracing perf_event - Hardware counters (PMCs) netlink - network statistics libpcap - network packet capture

Various files are provided in /proc for per-process statistics. Here is an example of what may be available for a given PID:

All examples using /proc/18

[root@docker01 ~]# ll /proc/18
dr-xr-xr-x. 2 root root 0 Jan  9 09:24 attr
-rw-r--r--. 1 root root 0 Jan  9 09:24 autogroup
-r--------. 1 root root 0 Jan  9 09:24 auxv
-r--r--r--. 1 root root 0 Jan  9 09:24 cgroup
--w-------. 1 root root 0 Jan  9 09:24 clear_refs
-r--r--r--. 1 root root 0 Jan  7 14:05 cmdline
-rw-r--r--. 1 root root 0 Jan  9 09:24 comm
-rw-r--r--. 1 root root 0 Jan  9 09:24 coredump_filter
-r--r--r--. 1 root root 0 Jan  9 09:24 cpu_resctrl_groups
-r--r--r--. 1 root root 0 Jan  9 09:24 cpuset
lrwxrwxrwx. 1 root root 0 Jan  9 09:24 cwd -> /
-r--------. 1 root root 0 Jan  9 09:24 environ
lrwxrwxrwx. 1 root root 0 Jan  9 09:24 exe
dr-x------. 2 root root 0 Jan  7 14:06 fd
dr-x------. 2 root root 0 Jan  9 09:24 fdinfo
-rw-r--r--. 1 root root 0 Jan  9 09:24 gid_map
-r--------. 1 root root 0 Jan  9 09:24 io
-r--r--r--. 1 root root 0 Jan  9 09:24 limits
-rw-r--r--. 1 root root 0 Jan  9 09:24 loginuid
dr-x------. 2 root root 0 Jan  9 09:24 map_files
-r--r--r--. 1 root root 0 Jan  9 09:24 maps
-rw-------. 1 root root 0 Jan  9 09:24 mem
-r--r--r--. 1 root root 0 Jan  9 09:24 mountinfo
-r--r--r--. 1 root root 0 Jan  9 09:24 mounts
-r--------. 1 root root 0 Jan  9 09:24 mountstats
dr-xr-xr-x. 7 root root 0 Jan  9 09:24 net
dr-x--x--x. 2 root root 0 Jan  7 15:26 ns
-r--r--r--. 1 root root 0 Jan  9 09:24 numa_maps
-rw-r--r--. 1 root root 0 Jan  9 09:24 oom_adj
-r--r--r--. 1 root root 0 Jan  9 09:24 oom_score
-rw-r--r--. 1 root root 0 Jan  9 09:24 oom_score_adj
-r--------. 1 root root 0 Jan  9 09:24 pagemap
-r--------. 1 root root 0 Jan  9 09:24 patch_state
-r--------. 1 root root 0 Jan  9 09:24 personality
-rw-r--r--. 1 root root 0 Jan  9 09:24 projid_map
lrwxrwxrwx. 1 root root 0 Jan  9 09:24 root -> /
-rw-r--r--. 1 root root 0 Jan  9 09:24 sched
-r--r--r--. 1 root root 0 Jan  9 09:24 schedstat
-r--r--r--. 1 root root 0 Jan  9 09:24 sessionid
-rw-r--r--. 1 root root 0 Jan  9 09:24 setgroups
-r--r--r--. 1 root root 0 Jan  9 09:24 smaps
-r--r--r--. 1 root root 0 Jan  9 09:24 smaps_rollup
-r--------. 1 root root 0 Jan  9 09:24 stack
-r--r--r--. 1 root root 0 Jan  7 14:05 stat
-r--r--r--. 1 root root 0 Jan  9 09:24 statm
-r--r--r--. 1 root root 0 Jan  7 14:05 status
-r--------. 1 root root 0 Jan  9 09:24 syscall
dr-xr-xr-x. 3 root root 0 Jan  9 09:24 task
-rw-r--r--. 1 root root 0 Jan  9 09:24 timens_offsets
-r--r--r--. 1 root root 0 Jan  9 09:24 timers
-rw-rw-rw-. 1 root root 0 Jan  9 09:24 timerslack_ns
-rw-r--r--. 1 root root 0 Jan  9 09:24 uid_map
-r--r--r--. 1 root root 0 Jan  9 09:24 wchan

The exact list of files depends on the kernel version and CONFIG options. Those related to per-process performance observability include: limits - in-effect resource limits maps - mapped memory regions sched - various CPU scheduler statistics schedstat - CPU runtime, latency, and time slices smaps - mapped memory regions with usage statistics stat - Process status and statistics, including total CPU and memory usage statm - memory usage summary in units of pages status - stat and statm information, labeled fd - directory of file descriptor symlinks cgroup - cgroup memborship information task - directory of per-task statistics

/proc also contains system-wide statistics in these directories:

[root@docker01 proc]# ls -Fd /proc [a-z]*
acpi/      bus/     consoles  devices    driver/      filesystems  iomem    kallsyms  key-users    kpagecount  locks    misc     mtrr          partitions   schedstat  slabinfo  swaps          sysvipc/      tty/     vmallocinfo
asound/    cgroups  cpuinfo   diskstats  execdomains  fs/          ioports  kcore     kmsg         kpageflags  mdstat   modules  net@          /proc/       scsi/      softirqs  sys/           thread-self@  uptime   vmstat
buddyinfo  cmdline  crypto    dma        fb           interrupts   irq/     keys      kpagecgroup  loadavg     meminfo  mounts@  pagetypeinfo  sched_debug  self@      stat      sysrq-trigger  timer_list    version  zoneinfo

  • /proc/cpuinfo

    • Description: Contains information about the CPU such as its type, make, model, number of cores, and processing power.

  • /proc/meminfo

    • Description: Provides details on the system’s memory usage including total and available physical memory, swap space, and various other memory parameters.

  • /proc/loadavg

    • Description: Shows the load average of the system, indicating how busy the system is. Displays averages over 1, 5, and 15 minutes.

  • /proc/uptime

    • Description: Indicates how long the system has been running since its last restart.

  • /proc/mounts

    • Description: Lists all the mounts currently in use by the system, similar to the mount command.

  • /proc/net

    • Description: Contains various network-related information including network configuration, statistics, connections, and more.

  • /proc/partitions

    • Description: Shows the partition table of all the storage devices in the system.

  • /proc/cmdline

    • Description: Displays the parameters passed to the kernel at the time it was started.

  • /proc/version

    • Description: Contains information about the version of the Linux kernel, GCC version used for the kernel build, and the build time.

  • /proc/filesystems

    • Description: Lists all the file systems currently supported by the kernel.

  • /proc/sys

    • Description: Contains a collection of interfaces to query and modify kernel parameters at runtime.

/sys

  • Linux provides a sysfs file system, mounted on /sys, which was introduced with the 2.6 kernel to provide a directory based structure for kernel statistics.
  • Netlink is a special socket address family (AF_NETLINK) for fetching kernel information.
  • To use Netlink, open a socket with the AF_NETLINK address family and then use a series of send(2) and recv(2) calls to pass requests and receiving information in binary structs.
  • The libnetlink library helps with usage.

Tracepoints

  • Tracepoints are a Linux Kernel event source based on static instrumentation.
  • Tracepoints are hard-coded instrumentation points placed at logical locations in kernel code.
  • Available tracepoints can be listed using the perf list tracepoint command
  • Apart from showing when an event happened, tracepoints can also show contextual data about an event.