Debugging C++ Coroutines

Introduction

For performance and other architectural reasons, the C++ Coroutines feature in the Clang compiler is implemented in two parts of the compiler. Semantic analysis is performed in Clang, and Coroutine construction and optimization takes place in the LLVM middle-end.

However, this design forces us to generate insufficient debugging information. Typically, the compiler generates debug information in the Clang frontend, as debug information is highly language specific. However, this is not possible for Coroutine frames because the frames are constructed in the LLVM middle-end.

To mitigate this problem, the LLVM middle end attempts to generate some debug information, which is unfortunately incomplete, since much of the language specific information is missing in the middle end.

This document describes how to use this debug information to better debug coroutines.

Terminology

Due to the recent nature of C++20 Coroutines, the terminology used to describe the concepts of Coroutines is not settled. This section defines a common, understandable terminology to be used consistently throughout this document.

coroutine type

A coroutine function is any function that contains any of the Coroutine Keywords co_await, co_yield, or co_return. A coroutine type is a possible return type of one of these coroutine functions. Task and Generator are commonly referred to coroutine types.

coroutine

By technical definition, a coroutine is a suspendable function. However, programmers typically use coroutine to refer to an individual instance. For example:

std::vector<Task> Coros; // Task is a coroutine type.
for (int i = 0; i < 3; i++)
  Coros.push_back(CoroTask()); // CoroTask is a coroutine function, which
                               // would return a coroutine type 'Task'.

In practice, we typically say “Coros contains 3 coroutines” in the above example, though this is not strictly correct. More technically, this should say “Coros contains 3 coroutine instances” or “Coros contains 3 coroutine objects.”

In this document, we follow the common practice of using coroutine to refer to an individual coroutine instance, since the terms coroutine instance and coroutine object aren’t sufficiently defined in this case.

coroutine frame

The C++ Standard uses coroutine state to describe the allocated storage. In the compiler, we use coroutine frame to describe the generated data structure that contains the necessary information.

The structure of coroutine frames

The structure of coroutine frames is defined as:

struct {
  void (*__r)(); // function pointer to the `resume` function
  void (*__d)(); // function pointer to the `destroy` function
  promise_type; // the corresponding `promise_type`
  ... // Any other needed information
}

In the debugger, the function’s name is obtainable from the address of the function. And the name of resume function is equal to the name of the coroutine function. So the name of the coroutine is obtainable once the address of the coroutine is known.

Get the suspended points

An important requirement for debugging coroutines is to understand suspended points, which are where the coroutine is currently suspended and awaiting.

For simple cases like the above, inspecting the value of the __coro_index variable in the coroutine frame works well.

However, it is not quite so simple in really complex situations. In these cases, it is necessary to use the coroutine libraries to insert the line-number.

For example:

// For all the promise_type we want:
class promise_type {
  ...
+  unsigned line_number = 0xffffffff;
};

#include <source_location>

// For all the awaiter types we need:
class awaiter {
  ...
  template <typename Promise>
  void await_suspend(std::coroutine_handle<Promise> handle,
                     std::source_location sl = std::source_location::current()) {
        ...
        handle.promise().line_number = sl.line();
  }
};

In this case, we use std::source_location to store the line number of the await inside the promise_type. Since we can locate the coroutine function from the address of the coroutine, we can identify suspended points this way as well.

The downside here is that this comes at the price of additional runtime cost. This is consistent with the C++ philosophy of “Pay for what you use”.

Get the asynchronous stack

Another important requirement to debug a coroutine is to print the asynchronous stack to identify the asynchronous caller of the coroutine. As many implementations of coroutine types store std::coroutine_handle<> continuation in the promise type, identifying the caller should be trivial. The continuation is typically the awaiting coroutine for the current coroutine. That is, the asynchronous parent.

Since the promise_type is obtainable from the address of a coroutine and contains the corresponding continuation (which itself is a coroutine with a promise_type), it should be trivial to print the entire asynchronous stack.

This logic should be quite easily captured in a debugger script.

Get the living coroutines

Another useful task when debugging coroutines is to enumerate the list of living coroutines, which is often done with threads. While technically possible, this task is not recommended in production code as it is costly at runtime. One such solution is to store the list of currently running coroutines in a collection:

inline std::unordered_set<void*> lived_coroutines;
// For all promise_type we want to record
class promise_type {
public:
    promise_type() {
        // Note to avoid data races
        lived_coroutines.insert(std::coroutine_handle<promise_type>::from_promise(*this).address());
    }
    ~promise_type() {
        // Note to avoid data races
        lived_coroutines.erase(std::coroutine_handle<promise_type>::from_promise(*this).address());
    }
};

In the above code snippet, we save the address of every lived coroutine in the lived_coroutines unordered_set. As before, once we know the address of the coroutine we can derive the function, promise_type, and other members of the frame. Thus, we could print the list of lived coroutines from that collection.

Please note that the above is expensive from a storage perspective, and requires some level of locking (not pictured) on the collection to prevent data races.