Tips for debugging Mbed TLS

This is a collection of tips for debugging TF-PSA-Crypto or Mbed TLS. It may also be useful for debugging applications using these projects, but that is not this document’s main purpose.

This document assumes some familiarity with the project, e.g. that you already know how to build and test it.

This document is written primarily with Linux in mind. Similar platforms such as macOS will require few adaptations. Windows (except WSL) is out of scope.

Reproducing CI builds with debugging

Getting the build products from `all.sh`

Normally, all.sh cleans up after itself. However, it will leave build products around if a compilation or runtime step fails. If you want to see build products from a passing component, add the command false after the build steps.

If you have a wrapper around all.sh, note that passing --keep-going (-k) makes it clean up on errors as well.

Cancelling all.sh with Ctrl+C (SIGINT) makes it clean up. But using Ctrl+\\ (SIGQUIT) bypassing the cleanup. Also, you can use Ctrl+Z to inspect an intermediate step.

Editing `all.sh` for debugging

To reproduce an all.sh component locally, but with debugging enabled:

For most builds using make (without CMake), in particular including all driver builds: add ASAN_CFLAGS='-Og -g3' or ASAN_CFLAGS='-O0 -g3' before the build step.
For builds using CMake: add or change the build type to Debug or ASanDbg, e.g. cmake -DCMAKE_BUILD_TYPE=Debug.

After changing the source, you’ll need to re-run all.sh, including its initial cleanup state which is not trivial to bypass. To speed this up, enable ccache. In most all.sh components, you can enable ccache by setting

CC="ccache ${CC:cc}" ASAN_CC="ccache clang"

Sanitizers

Sanitizers used in test scripts

ASan: AddressSanitizer

Documentation: https://github.com/google/sanitizers/wiki/addresssanitizer
Detects: buffer overflows, use after free, memory leaks
Compilers: GCC, Clang
Compiler flags: -fsanitize=address -fno-sanitize-recover=all (in both CFLAGS and LDFLAGS)
CMake build types: ASan, ASanDbg
Used in: most builds in all.sh

MSan: MemorySanitizer

Documentation: https://github.com/google/sanitizers/wiki/memorysanitizer
Detects: uninitialized memory
Compilers: GCC, Clang
Compiler flags: -fsanitize=memory (in both CFLAGS and LDFLAGS)
CMake build types: MemSan, MemSanDbg
Used in: component_test_memsan*

TSan: ThreadSanitizer

Documentation: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
Detects: race conditions
Compilers: GCC, Clang
Compiler flags: -fsanitize=thread (in both CFLAGS and LDFLAGS)
CMake build types: TSan, TSanDbg
Used in: component_test_tsan*

UBSan: UndefinedBehaviorSanitizer

Documentation: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
Detects: null pointer misuse, bitwise shift amount out of range, signed integer overflow, …
Compilers: GCC, Clang
Compiler flags: -fsanitize=undefined (in both CFLAGS and LDFLAGS)
CMake build types: ASan, ASanDbg
Used in: most builds in all.sh

Valgrind

Valgrind mostly duplicates Asan+Msan, but very occasionally finds something that they don’t.

Documentation: https://valgrind.org/docs/manual/manual.html
Detects: buffer overflows, use after free, memory leaks, uninitialized memory
- We don’t currently use it for race conditions.
Compilers: any
Compiler flags: N/A — runtime instrumentation only
CMake target: make memcheck

Run with:

valgrind -q --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=50 --log-file=myprogram.MemoryChecker.log myprogram
grep . myprogram.MemoryChecker.log myprogram

Used in: component_release_test_valgrind*

Getting symbolic backtraces from symbolizers

By default, ASan/MSan/TSan/UBSan display traces without symbolic information. For traces with symbol names, you need to set environment variables:

export ASAN_OPTIONS=symbolize=1
export MSAN_OPTIONS=symbolize=1
export TSAN_OPTIONS=symbolize=1
export UBSAN_OPTIONS=print_stacktrace=1

With Clang, depending on how it’s installed, you may need to specify the path to the correct version of llvm-symbolizer in ASAN_SYMBOLIZER_PATH, MSAN_SYMBOLIZER_PATH and TSAN_SYMBOLIZER_PATH. For example:

if ASAN_SYMBOLIZER_PATH=$(readlink -f "$(command -v clang)") &&
   ASAN_SYMBOLIZER_PATH="${ASAN_SYMBOLIZER_PATH%/*}/llvm-symbolizer"
then
  export ASAN_SYMBOLIZER_PATH
  export MSAN_SYMBOLIZER_PATH="$ASAN_SYMBOLIZER_PATH"
  export TSAN_SYMBOLIZER_PATH="$ASAN_SYMBOLIZER_PATH"
fi

See SanitizerCommonFlags for more flags you can use in $xxSAN_OPTIONS.

Sanitizers for constant-time testing

See “Mbed TLS test guidelines — Constant-flow testing”.

Reverse debugging

What is reverse debugging?

Also known as back-in-time debugging or time travel debugging.

Reverse debugging allows you to go backward in time when stepping through a program. For example, a reverse single step after returning from a function goes back to the function’s return statement.

Tools for reverse debugging

Gdb supports reverse debugging, but not out of the box, it requires some complex setup.
LLDB does not support reverse debugging as of 2025.
Visual Studio (under Windows) supports reverse debugging since 2017.

Reverse debugging works by taking snapshots of a program and recording its inputs and outputs. It may or may not work when the program interacts with its environment in complex ways, since the environment does not roll back when the program does.

Replay debuggers

A replay debugger records one execution of the program. It then replays this same execution, simulating all inputs and outputs.

Replay debugging on Linux with rr

Install the Mozilla Record and Replay framework (rr) from https://rr-project.org/ or e.g. apt install rr.

If needed, give yourself debugging permission:

# The Ubuntu default is 4 which is too paranoid.
sudo sysctl kernel.perf_event_paranoid=1.​
# Make this persistent across reboots.
echo 'kernel.perf_event_paranoid = 1' >>/etc/sysctl.d/zz-local.conf​

To debug a program, build it with debugging symbols as usual (-O0 –g3 or –Og -g3). Then run it once to save a full trace of the execution:

rr record tests/test_suite_ssl

Then rr replay gives you a gdb interface where reverse execution actually works. You can use reverse-xxx commands such as:

rs (reverse-step) steps into functions.
rn (reverse-next) steps over function calls.
reverse-finish goes back to where the current function was called.
set exec-direction reverse changes step, next, etc. to go backwards. Switch this off with set exec-direction forward.

If you use a frontend, configure it to run rr replay instead of gdb myprogram. If the frontend uses gdb’s machine interface, use rr replay -i=mi … instead of gdb -i=mi ….

Replay debugging on macOS with warpspeed

Try warpspeed.