What is o-profile?

See the description below extracted from http://oprofile.sourceforge.net/, o-profile official website

OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead. OProfile is released under the GNU GPL.

It consists of a kernel driver and a daemon for collecting sample data, and several post-profiling tools for turning data into information.

OProfile leverages the hardware performance counters of the CPU to enable profiling of a wide variety of interesting statistics, which can also be used for basic time-spent profiling. All code is profiled: hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.

OProfile is currently in alpha status; however it has proven stable over a large number of differing configurations; it is being used on machines ranging from laptops to 16-way NUMA-Q boxes. As always, there is no warranty.

Why don’t we use gprof?

For embedded system , we do not use gprof (the GNU profiler) because it is not supported by uClibc. The main reason for this is explained below:
uClibc no longer supports ‘gcc -fprofile-arcs -pg’ style profiling, which causes your application to generate a ‘gmon.out’ file that can then be analyzed by ‘gprof’. Not only does this require explicit extra support in uClibc, it requires that you rebuild everything with profiling support. There is both a size and performance penalty to profiling your applications this way, as well as Heisenberg effects, where the act of measuring changes what is measured.

Preparing your system for o-profile

Your kernel must support o-profile, for this you have to reconfigure it with o-profile enabled as a module.
We’ll use EP9307 linux 2.6 for example in this blog entry.

For EP9307 target, we use make linuxconfig in edb9307 to enable O-PROFILE.

O-profile uses a script to control the profiling called opcontrol.
This script requires bash 2 or superior, so we must cross-compile bash for the platform under test, if it is not already present. It can be downloaded at http://ftp.gnu.org/gnu/bash/.
This script also use a fair amount a shell command (dirname, awk, which, cut etc..) that may have been disabled in busybox, if you see error when running opcontrol, you have to check the command missing and enable them in busybox. For Ep9307 , make busyboxconfig in edb9307 directory can be used to change the settings.

Cross-Compiling o-profile

O-profile must be cross-compiled for the platform under test.
It requires the following libraries:
libintl, libbdf, libliberty (part of binutils)
libpopt
First you need to compile the binutils (preferrably the same used by your cross compiler) as follows:
./configure –target=arm-linux –host-linux –build=i686 –without-gettext –without-intl –prefix=$SRCBLB/libs
make
make install
You also need to compile gettext (It can be download from one of GNU servers) to be able to compile binutils (even though this was disabled)

To compile libpopt: (Download popt)
./configure –target=arm-linux –host=arm-linux –prefix-$SRCBLD/libs/
make
make install

To compile oprofile:
./configure –target=arm-linux –host=arm-linux –without-x –with-kernel-support –with-extra-libs=$SRCBLD/libs/lib –with-extra-includes=$SRCBLD/libs/include –prefix=$SRCBLD/tools/oprofile-0.9.1
make
make install
Finally you have to copy the following files to their relevant directories within the ramdisk.
$SRCBLD/lib/bash-2.0.5/bash
$SRCBLD/libs/lib/libpopt.so*
$SRCBLD/libs/lib/libintl.so*
$SRCBLD/tools/oprofile-0.9.1-$BLD_FLAVOR/op*

Enabling profiling in your applications

The good thing here is that contrary to gprof, there is nothing to do to enable o-profile in your application. Although you’d better compile it using debug information (CFLAGS += -g).

Using o-profile

First type bash to start the bash shell which is necessary to use opcontrol script.

Getting the profiling data

Initialize oprofile: opcontrol –no-vmlinux
start your application : e.g. /usr/app/app_dbg
Start oprofile: opcontrol –start
wait a few seconds to get profiling data
Stop oprofile: opcontrol –shutdown

Analyzing the results

Opreport is used to display the profiling result, a list of common commands can be found below:

Quick report:
opreport
See example of simple report

Longer report:
opreport -l
See example of “long” report

Report with callgraph, you can see what functions are calling other functions in the output:
opreport –callgraph
See example of callgraph report

Report showing source file and line for each symbol:
opreport /usr/app/app_dbg –debug-info –symbols
See example of report showing source code mixed with profiling information

Detailled report:
opreport –details
See example of detailed report

The profiling was performed while playing a MPEG-4 video with mplayer on EP9307.

For further information please see: http://oprofile.sourceforge.net/doc/index.html

How-to use o-profile

Table of Contents

What is o-profile? 1

Overview 1

Features 2

Why don’t we use gprof? 3

Preparing your system for o-profile 4

Cross-Compiling o-profile 4

Enabling profiling in your applications 6

Using o-profile 6

Getting the profiling data 6

Analysing the results 6

NB: Most users only need to read the last 2 sections: Enabling profiling in your applications & Using O-profile. The other sections explain in details how the tools have been compiled.

What is o-profile?

See the description below extracted from http://oprofile.sourceforge.net/, o-profile official website:

Overview

OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead. OProfile is released under the GNU GPL.

It consists of a kernel driver and a daemon for collecting sample data, and several post-profiling tools for turning data into information.

OProfile leverages the hardware performance counters of the CPU to enable profiling of a wide variety of interesting statistics, which can also be used for basic time-spent profiling. All code is profiled: hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.

OProfile is currently in alpha status; however it has proven stable over a large number of differing configurations; it is being used on machines ranging from laptops to 16-way NUMA-Q boxes. As always, there is no warranty.

Features

Unobtrusive: No special recompilations, wrapper libraries or the like are necessary. Even debug symbols (-g option to gcc) are not necessary unless you want to produce annotated source.
No kernel patch is needed – just insert the module.
System-wide profiling: All code running on the system is profiled, enabling analysis of system performance.
Performance counter support: Enables collection of various low-level data, and assocation with particular sections of code.
Call-graph support: With an x86 2.6 kernel, OProfile can provide gprof-style call-graph profiling data.
Low overhead: OProfile has a typical overhead of 1-8%, dependent on sampling frequency and workload.
Post-profile analysis: Profile data can be produced on the function-level or instruction-level detail. Source trees annotated with profile information can be created. A hit list of applications and functions that take the most time across the whole system can be produced.
System support: OProfile works across a range of CPUs, include the Intel range, AMD’s Athlon and AMD64 processors range, the Alpha, and more. OProfile will work against almost any 2.2, 2.4 and 2.6 kernels, and works on both UP and SMP systems from desktops to the scariest NUMAQ boxes.

Why don’t we use gprof?

For embedded system , we do not use gprof (the GNU profiler) because it is not supported by uClibc. The main reason for this is explained below:

uClibc no longer supports ‘gcc -fprofile-arcs -pg’ style profiling, which causes your application to generate a ‘gmon.out’ file that can then be analyzed by ‘gprof’. Not only does this require explicit extra support in uClibc, it requires that you rebuild everything with profiling support. There is both a
size and performance penalty to profiling your applications this way, as well as Heisenberg effects, where the act of measuring changes what is measured.

There exist a number of less invasive alternatives that do not require you to specially instrument your application, and recompile and relink everything.

Preparing your system for o-profile

Your kernel must support o-profile, for this you have to reconfigure it with o-profile enabled as a module. For kokstb project, the make linuxconfig in $OSROOTDIR/edb9307 as been used to change the settings.

This script also use a fair amount a shell command (dirname, awk, which, cut etc..) that may have been disabled in busybox, if you see error when running opcontrol, you have to check the command missing and enable them in busybox. For kokstb project, the make busyboxconfig in $OSROOTDIR/edb9307 as been used to change the settings.

Cross-Compiling o-profile

O-profile must be cross-compiled for the platform under test.
It requires the following libraries:

libintl, libbdf, libliberty (part of binutils)
libpopt

First you need to compile the binutils (preferrably the same used by your cross compiler) as follows:
./configure –target=arm-linux –host-linux –build=i686 –without-gettext –without-intl –prefix=$SRCBLB/libs
make
make install

You also need to compile gettext (Can be obtained from cvs co lib/gettext)to be able to compile binutils (even though this was disabled)

To compile libpopt: (Can be obtained from cvs: cvs co lib/popt-1.10.4)

./configure –target=arm-linux –host=arm-linux –prefix-$SRCBLD/libs/
make
make install

Finally you have to copy the following files to their relevant directories within the ramdisk. This should be done automatically in the Makefile when BLD_PROFILE = Y.

$SRCBLD/lib/bash-2.0.5/bash
$SRCBLD/libs/lib/libpopt.so*
$SRCBLD/libs/lib/libintl.so*
$SRCBLD/tools/oprofile-0.9.1-$BLD_FLAVOR/op*

For kokstb and EP9307, the bash, libpopt, libintl, and oprofile tools are located in tools/oprofile-.0.9.1-EP9307.

Enabling profiling in your applications

In order to enable profiling support in your application, you need to set BLD_PROFILE=Y on your development PC as follows: export BLD_PROFILE=Y.
If BLD_FLAVOR is set to x86, then gprof will be used, if it is set to another value (e.g. EP9307), then oprofile will be used.
Once BLD_PROFILE is set, you do make clean then make.
If you need to have more debug information, also set BLD_DEBUG=Y, do make cleanall, then make all.
Then load the kernel and ramdisk to your platform as usual.

Using o-profile

First type bash to start the bash shell which is necessary to use opcontrol script.

Getting the profiling data

Initialize oprofile: opcontrol –no-vmlinux
start your application : e.g. /usr/kokstb/kokstb_dbg
Start oprofile: opcontrol –start
wait a few seconds to get profiling data
Stop oprofile: opcontrol –shutdown

Analysing the results

Opreport is used to display the profiling result, a list of common commands can be found below:

Quick report:

opreport
See \\10.10.10.251\doc\Profiling\simplereport.txt

Longer report:

opreport -l

See \\10.10.10.251\doc\Profiling\report.txt

Report with callgraph, you can see what functions are calling other functions in the output:
opreport –callgraph
See \\10.10.10.251\doc\Profiling\report_callgraph.txt

Report showing source file and line for each symbol:
opreport /usr/koksb/kokstb_dbg –debug-info –symbols
See \\10.10.10.251\doc\Profiling\report_kokstb_debug.txt

Detailled report:
opreport –details
See \\10.10.10.251\doc\Profiling\report_details.txt

For further information please see: http://oprofile.sourceforge.net/doc/index.html

Jean-Luc Aufranc (CNXSoft)

Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.