Getting mutrace to work on zLinux

Recently I got the question on how to trace mutex contention in libpthread on zLinux. There are several solutions to this

1. Use SystemTap with the futexes.stp sample script
2. Use Valgrind with the drd tool (see also this article) 
3. Use mutrace as a lightweight tool

The last tool I've discovered in searching for solutions. However it hasn't been clear if it runs on zLinux or not. Usually it's just a ./configure and make to get a tool running but this one turned out to be a little bit more difficult.

I started of on a standard SLES11 SP2 with some of the development tools installed. So I downloaded the source from the mutrace-git and installed it in a directory. Then I called the ./bootstrap.sh script. Sure enough it was failing:
...
+ aclocal -I m4
configure.ac:21: error: Autoconf version 2.68 or higher is required
configure.ac:21: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63

SLES11 SP2 hast autoconf 2.63, which isn't that ancient and SUSE had been patching and fixing it now in the second service pack. So I gave it a try and modified configure.ac to accept a minimum level of 2.63. Next run:

...

checking for library containing bfd_init... no
configure: error: *** libbfd not found

This means that the system is missing the devel package of the binutils. After installing the binutil-devel package with
zypper install binutils-devel
the bootstrap script finished successfully. At the end I noted that it used a -O0 in the gcc options, which from a performance perspective is really bad on zLinux. So I changed that in the Makefile to a -O2.

So now only the compile had to work and sure enough it ended with a 

mutrace.c: In function setup:
mutrace.c:441: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:442: error: #pragma GCC diagnostic not allowed inside functions
mutrace.c:444: error: #pragma GCC diagnostic not allowed inside functions
make[1]: *** [libmutrace_la-mutrace.lo] Error 1

So this tool was using an advanced gcc feature that the gcc-4.3 from SUSE didn't have. Fortunately SUSE includes an updated version gcc-4.6 that can be installed along with the standard system compiler. The package name is gcc46 and instead of gcc you call gcc-4.6. After changing the Makefile once more the compile went smoothly.
Finally tried it on a small test program and it seems to work fine.
mutrace: Showing statistics for process a.out (PID: 12050).
mutrace: 1 mutexes used.

Mutex #0 (0x0x80003088) first referenced by:
        /root/mutrace-e23dc42/.libs/libmutrace.so(pthread_mutex_lock+0x9e) [0x3fffd07c28e]
        ./a.out(functionCount1+0x20) [0x80000e34]
        /lib64/libpthread.so.0(+0x836e) [0x3fffd05436e]
        /lib64/libc.so.6(+0xef17e) [0x3fffcfb417e]

mutrace: Showing 1 mutexes in order of (write) contention count:

 Mutex #   Locked  Changed    Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] Flags
       0       83       13        7         0.155        0.089        0.001 M-.--.
     ...      ...      ...      ...           ...          ...          ... ||||||
                                                                            /|||||
          Object:                                      M = Mutex, W = RWLock /||||
           State:                                  x = dead, ! = inconsistent /|||
             Use:                                  R = used in realtime thread /||
      Mutex Type:                   r = RECURSIVE, e = ERRORCHECK, a = ADAPTIVE /|
  Mutex Protocol:                                       i = INHERIT, p = PROTECT /
     RWLock Kind:  r = PREFER_READER, w = PREFER_WRITER, W = PREFER_WRITER_NONREC

mutrace: Note that rwlocks are shown as two lines: write locks then read locks.

mutrace: Note that the flags column R is only valid in --track-rt mode!

mutrace: 1 condition variables used.

Condvar #0 (0x0x800030b0) first referenced by:
        /root/mutrace-e23dc42/.libs/libmutrace.so(pthread_cond_wait+0x7a) [0x3fffd07caea]
        ./a.out(functionCount1+0x32) [0x80000e46]
        /lib64/libpthread.so.0(+0x836e) [0x3fffd05436e]
        /lib64/libc.so.6(+0xef17e) [0x3fffcfb417e]

mutrace: Showing 1 condition variables in order of wait contention count:

  Cond #    Waits  Signals    Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] Flags
       0        6       67        0        0.106         0.000        0.000     -.
     ...      ...      ...      ...          ...           ...          ...     ||
                                                                                /|
           State:                                     x = dead, ! = inconsistent /
             Use:                                     R = used in realtime thread

mutrace: Note that the flags column R is only valid in --track-rt mode!

mutrace: Total runtime is 0.319 ms.

mutrace: Results for SMP with 16 processors.