This is the first post from the zymtrace engineering team. We are building a continuous whole-system profiler for GPU workloads, and one of the foundational design decisions we had to make was how to represent CUDA kernel launches in a flamegraph format. That decision is not straightforward, and getting it wrong produces output that misleads rather than informs.
This post explains the problem, how we approached it, and what the resulting flamegraph representation tells you versus what it deliberately does not try to tell you.
What a CUDA Kernel Launch Actually Is
A CUDA kernel launch is not an instantaneous event. It involves two distinct phases that happen on two different processors.
The first phase is the CPU-side launch: your code calls cudaLaunchKernel (or its driver equivalent cuLaunchKernel), which packages the kernel arguments, validates them against the kernel's parameter signature, and places a launch command into a hardware command buffer or software CUDA stream queue. This is the phase that the CPU executes and that a CPU profiler can observe. Its duration is typically 2 to 20 microseconds depending on the number of kernel parameters, whether tensor shapes have changed since the last launch (affecting JIT compilation state), and system load on the CUDA driver.
The second phase is the device-side execution: the CUDA hardware scheduler picks up the kernel from the command buffer and issues it to available Streaming Multiprocessors. The kernel runs asynchronously with respect to the CPU. Its duration could be 50 microseconds or 50 milliseconds depending on the computation. The CPU does not wait for this to complete unless there is an explicit synchronization call.
These two phases are physically separated in time and space. A naive profiler records either one or the other. The interesting information for performance analysis requires correlating both.
The Concurrency Problem in Flamegraphs
Traditional CPU flamegraphs represent a single thread of execution on a timeline. Each frame is a function call that starts and ends, and child frames are nested within parent frames. The horizontal axis is wall clock time, and frames do not overlap.
GPU kernels break this model in two ways. First, kernels execute asynchronously, so the CPU-side call ends long before the GPU-side execution completes. If you draw the kernel frame from CPU launch to CPU return, you misrepresent the actual duration. If you draw it from GPU start to GPU end, it is no longer nested under the CPU call that launched it, and the flamegraph loses the call-path context.
Second, multiple kernels can run concurrently on the same GPU. Two kernels issued on different streams, if they have available SM resources, may execute simultaneously. A timeline view that draws them as sequential is inaccurate. A view that tries to show true concurrency becomes a Gantt chart, not a flamegraph, and loses the call-path hierarchy that makes flamegraphs useful for root-cause attribution.
Our Representation: Attribution Over Timeline
We made a specific design choice: zymtrace's flamegraph represents attribution, not timeline execution. The width of a frame represents total GPU execution time attributed to that call path, not the wall clock span between when the CPU started and finished the call.
What this means concretely: if your Python code calls torch.nn.functional.linear() and that dispatches a matrix multiply kernel that runs for 4 milliseconds on the GPU, the linear frame in the flamegraph is 4 milliseconds wide, even though the Python-side call took 5 microseconds and returned to the caller immediately. The 4 milliseconds is the GPU execution cost attributed to that call path.
This means the flamegraph width is a measure of GPU compute cost, not CPU wall time. Two kernels that ran concurrently are both shown as full-width frames under their respective call paths. The total of all frame widths can exceed the actual wall clock duration of the profiling window. This is intentional and documented in the UI.
Why Attribution Is the Right Metric for Cost Analysis
The question we are trying to answer with the flamegraph is: which code path is responsible for the GPU compute cost in this workload? Attribution answers that question directly. A call path that dispatches kernels accounting for 30% of total GPU execution time should be 30% wide in the flamegraph, regardless of how those kernels are scheduled relative to each other or to kernels from other call paths.
If we showed timeline execution instead, concurrent kernels would appear narrower than their actual cost, and the flamegraph widths would depend on scheduling decisions rather than the actual compute work being done. Two identical workloads with different stream concurrency settings would produce different-looking flamegraphs even though the total work is the same. Attribution is stable across scheduling variations, which makes it more useful for optimization decisions.
What We Record at Each Launch
When zymtrace's eBPF agent observes a cudaLaunchKernel call, it records the following at the CPU side: the full user-space call stack (from the Python frame through the C extension and ATen dispatch layers), the kernel's entry point address (which we resolve to a symbol name), the grid and block dimensions, the timestamp, and the CUDA stream identifier.
The device-side execution time is correlated using CUDA events. We bracket each kernel launch with cudaEventRecord calls on the launch stream, which give us precise GPU-side timestamps for when the kernel actually started and ended on the device. The event timestamps and the CPU-side stack record are joined in the profiler backend to produce the attributed flamegraph entry.
The kernel symbol name is resolved from the entry point address using CUDA's symbol lookup. For kernels in shared libraries (like PyTorch's CUDA extension), we use the shared library's symbol table. For JIT-compiled kernels (like those from torch.compile or Triton), we use the compiled module's symbol table which is available through CUDA's driver API. Names like ampere_fp16_s16816gemm_fp16_128x128_ldg8_f2f_stages_32x1_nn are verbose, but they encode the kernel variant and are uniquely attributable to specific operation and hardware configurations.
Practical Interpretation: What You Should and Should Not Conclude
The flamegraph is reliable for identifying which call paths account for the most GPU compute time. If a call path is wide in the flamegraph, the kernels dispatched from that path are genuinely expensive in terms of GPU execution time.
The flamegraph does not tell you whether those kernels are inefficient given the hardware. A kernel that is wide because it is doing a large, well-optimized matrix multiply is fundamentally different from a kernel that is wide because of poor memory access patterns or low SM occupancy. Both appear as wide frames, and the flamegraph does not distinguish them. For hardware efficiency analysis, you need CUDA hardware counters, which Nsight or CUPTI-based tools provide. The flamegraph is the starting point that tells you where to look; hardware counters tell you why a specific kernel is slow.
We are working on adding occupancy and memory bandwidth utilization annotations to the flamegraph frames for kernels where CUPTI can provide those metrics without per-kernel overhead. That would bring the "where is it expensive" and "why is it slow" views closer together. For now they are separate workflows, and understanding the boundary between them is the first step to using both correctly.