Tracing System and Library Calls w/ strace and ltrace

For whatever reason, sometimes debugging a program on a unix system requires more than just gdb. I came across two tools recently that can be used for more advanced debugging. strace and it's brother ltrace are applications that will trace system calls during runtime. This means that they will trace the interaction between the process and the system. ltrace is the more featured of the two because it has the ability to trace dynamic library calls. Common usage can be seen in this example, below:

longhorn$ vglrun ltrace -f -S testFont 2>&1 | grep -i font
SYS_open("/usr/lib64/libfontconfig.so.1", 0, 04342330000) = 3
SYS_read(3, "Name:\ttestFont\nState:\tR (running"..., 4096) = 833
SYS_read(3, "Name:\ttestFont\nState:\tR (running"..., 4096) = 833
SYS_read(3, "Name:\ttestFont\nState:\tR (running"..., 4096) = 833
SYS_open("testFont", 0, 00)                      = 3
SYS_open("/usr/lib64/libfontconfig.so.1", 0, 00) = 3
SYS_read(10, "Name:\ttestFont\nState:\tR (running"..., 4096) = 833
SYS_read(5, "Name:\ttestFont\nState:\tR (running"..., 4096) = 833

You can see both system calls and library calls in this example. You'll notice I piped the output to grep to search for a specific call. The "-S" option allow you to trace system calls, making ltrace's functionality a superset of strace's. In addition, the flag "-f" causes ltrace to also follow any child processes the main process may create.

One big difference between the two is that ltrace does not currently support multi-threaded applications because the kernel will send a SIGTRAP command to a traced process, causing premature termination. strace does work with multi-threaded applications, however. See this link for more info.

These trace applications can be very useful for finding out exactly what the code is doing at the system level, and can you you better understand how the code really works. By searching the net it seems that it is very useful for hackers and crackers and reverse engineering as well!