Firefox https://access.redhat.

com/solutions/5441

Subscriptions Downloads Containers Support Cases

Products & Services Knowledgebase What is SystemTap and how to use it?

What is SystemTap and how to use it?

$ SOLUTION VERIFIED - Updated April 13 2021 at 2:15 AM - English

Environment
• Red Hat Enterprise Linux (all versions)

Issue
• What is SystemTap and how to use it?

Resolution
SystemTap is an innovative tool which allows for simplified information gathering on the running Linux kernel.
The purpose of using SystemTap is to obtain information on either performance issues or functional problems
(bugs). By using SystemTap, developers and system administrators can debug problems and gather profiling and
performance data without having to create and install instrumented kernels or other packages.

In essence, SystemTap provides the infrastructure (a command line interface and scripting language) needed to
gather information. The actual job that SystemTap performs relies on user-developed scripts tailored to a
specific purpose. Currently, there are a number of example SystemTap scripts pre-made for general use.

The operation of SystemTap is quite simple. The stap command takes as an argument the name of a
SystemTap file (called a probe). There may be additional command line arguments passed depending on the
probe. SystemTap translates the probe into C and compiles the C source as a kernel module. It then inserts the
resulting kernel module into the running kernel to perform the probe functions defined in the script. The output
is printed to the console or can be redirected to file.

Requirements
Because SystemTap compiles scripts from C code and launches probes for kernel instrumentation, it requires
several packages in order to function. See the Installing SystemTap chapter of the SystemTap Beginners Guide
for detailed installation instructions.

The easiest way to satisfy the requirements is to simply subscribe the system to the relevant debuginfo channels
in RHSM, then run the following commands which should set up the environment for SystemTap:

# yum install systemtap # stap-prep

To set the environment up manually, in addition to the systemtap package, the following packages must also
be installed:

• kernel-devel for the running kernel

• kernel-debuginfo for the running kernel
• kernel-debuginfo-common for the running kernel
• gcc
• systemtap

For example, for a 2.6.32-71.18.2.el6.x86_64 kernel you'll need the following kernel packages along with

1 sur 4 10/02/2022, 09:42

Firefox https://access.redhat.com/solutions/5441

the latest gcc and systemtap packages.

kernel-debuginfo-2.6.32-71.18.2.el6.x86_64 kernel-debuginfo-common-
x86_64-2.6.32-71.18.2.el6.x86_64 kernel-devel-2.6.32-71.18.2.el6.x86_64

The gcc and kernel-devel packages are available on Red Hat Network and can be installed using yum
(RHEL 5 onwards).

Once packages are installed, confirm everything is in place once by running the following command :

# rpm -qa | egrep -e kernel-'(devel|debug)' -e systemtap

Legacy

On RHEL 4, use up2date . For Red Hat Enterprise Linux (RHEL) 4, the kernel-debuginfo package is not
available via up2date and must be installed from the ISO image available on Red Hat Network or from
https://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4AS/en/os/Debuginfo/ (substitute 4AS in the
URL with 4WS , 4ES , or 4Desktop , depending on the variant of RHEL you have installed. Note: Red Hat
Enterprise Linux 4 does not have kernel-debuginfo-common package.

For RHEL 5, you can download the kernel-debuginfo and kernel-debuginfo-common packages from
https://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/.

Using Systemtap
You can verify the SystemTap environment with the stap-report command.

An example script is shown below:

#! /usr/bin/env stap # Using statistics and maps to examine kernel memory allocations global
kmalloc probe kernel.function("__kmalloc") { kmalloc[execname()] <<< $size } # Exit after 10
seconds probe timer.ms(10000) { exit () } probe end { foreach ([name] in kmalloc) {
printf("Allocations for %s\\n", name) printf("Count: %d allocations\\n", @count(kmalloc[name]))
printf("Sum: %d Kbytes\\n", @sum(kmalloc[name])/1000) printf("Average: %d bytes\\n",
@avg(kmalloc[name])) printf("Min: %d bytes\\n", @min(kmalloc[name])) printf("Max: %d bytes\\n",
@max(kmalloc[name])) print("\\nAllocations by size in bytes\\n") print(@hist_log(kmalloc[name]))
printf("-------------------------------------------------------\\n\\n"); } }

This script, drawn from the SystemTap project wiki, can be used to print information kernel memory allocations
of the system. The script can be invoked as follows:

stap kmalloc2.stp

For issues during the compile or loading of the module within the stap command, append the parameter -vv
to show more verbose output.

SystemTap will then translate the probe into C, compile the C program, and insert the probe into the running
kernel. Truncated output is below:

------------------------------------------------------- Allocations for httpd Count: 10

allocations Sum: 0 Kbytes Average: 0 bytes Min: 0 bytes Max: 0 bytes Allocations by size in bytes
value |-------------------------------------------------- count 0 |@@@@@@@@@@ 10 1 | 0 2 | 0
------------------------------------------------------- Allocations for sendmail Count: 2
allocations Sum: 0 Kbytes Average: 24 bytes Min: 24 bytes Max: 24 bytes Allocations by size in
bytes value |-------------------------------------------------- count 4 | 0 8 | 0 16 |@@ 2 32 | 0
64 | 0 -------------------------------------------------------

This is a simple example that just touches the surface of the capabilities offered by SystemTap. System
Administrators could use this information to better understand kernel memory allocation on the running system
and adjust kernel tuning parameters accordingly. Application developers can use this information as an overview
of which applications are receiving more kernel memory allocations, which can be used as a starting point for

2 sur 4 10/02/2022, 09:42

Firefox https://access.redhat.com/solutions/5441

deeper application profiling.

Additional Information about SystemTap

• Complement to this guide
• SystemTap Beginners Guide
• The SystemTap project Wiki: http://sourceware.org/systemtap/wiki/HomePage
• The SystemTap project Homepage: http://sourceware.org/systemtap/
• Red Hat Magazine overview of SystemTap: http://www.redhat.com/magazine/011sep05/features
/systemtap/

Product(s) Red Hat Enterprise Linux Component systemtap Category Troubleshoot

Tags performance rhel_4 rhel_5 rhel_6 rhel_7

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red
Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it
becomes available, these articles may be presented in a raw and unedited form.

People who viewed this solution also viewed

How to make a systemtap kernel module load persistently across reboots?

Solution - 28 janv. 2021

Can SystemTap debuginfo be placed somewhere else?

Solution - 9 nov. 2021

How to build a Systemtap kernel module ?

Solution - 28 janv. 2021

4 Comments

Bryan Totty 31 March 2014 4:36 PM

I have created a command for RHEL 6 server that you can use exactly if you want to temporarily enable the debug
RED HAT repository on a system and not have to permanently enable it.
GURU
This should run an install of the needed packages with yum and select the current kernel version of the required
3376
Points debuginfo packages.

3 sur 4 10/02/2022, 09:42

Firefox https://access.redhat.com/solutions/5441

# yum install yum-utils systemtap kernel-devel kernel-debuginfo-`uname -r` kernel-debuginfo-common-

x86_64-`uname -r` --enablerepo=rhel-6-server-debug-rpms

≤ Reply

Frank Eigler 26 February 2016 3:55 PM

FE It may be worth noting that debuginfo requirements vary by the script. Many systemtap scripts do not require
RED HAT kernel-debuginfo at all. The #stap-prep script probably still should install it (just in case); if it doesn't work for you,
ACTIVE please file a bug.
CONTRIBUTOR

112 Points ≤ Reply

S Heijmans 14 October 2021 11:33 AM

SH How do I use this on OCP4/RHCOS node?

COMMUNITY
MEMBER ≤ Reply

65 Points

Frank Eigler 14 October 2021 6:49 PM

FE To perform traditional systemwide probing, you'd need a privileged container (so it can install the
RED HAT custom-built kernel modules) that contains the relevant systemtap RPMs. That container also needs
ACTIVE to have the kernel-devel* RPM that matches the host. It may or may not need debuginfo installed,
CONTRIBUTOR
depending on script. If you are using RHEL8 era systemtap, and you have a debuginfod server on your
112 Points
network, this last part can be fully automated.

≤ Reply

Copyright © 2022 Red Hat, Inc.

4 sur 4 10/02/2022, 09:42