Differential Flame Graphs

Learn how to use and interpret differential flame graphs.

There are multiple ways to visualize and use profiling data, including differential flame graphs, which can help easily identify function regressions or changes in the execution context.

Differential flame graphs are graph visualizations that highlight how the call stack frequencies of a program have changed, a task which can be tedious without them. They're often used in analysis for finding function regressions and understanding which functions got slower, faster, or were newly added. Keep in mind that differential flame graphs aren't really a new concept, but just aggregate flame graphs drawn using the colors generated by the diff of the two.

In Sentry, differential flame graphs are shown whenever a function regression is detected. When a regression happens, Sentry generates two aggregate flame graphs. One from the period leading up to the regression, and another from the time right after it. A differential flame graph is then used to visualize the difference between the two.

In a differential flame graph, the color red means that a function's execution time has increased or that there's a newly present function. The color blue means that the execution time has decreased.

Be on the lookout for red frames when interpreting differential flame graphs. They're the reason why your application code has been performing worse.

Differential flame graph lock contention

The differential flame graph above is showing that the function call to ModuleLock.acquire has increased in duration, hinting at a lock contention problem which is slowing down the code.

Let's look at another example:

Differential flame graph new functions

The above differential flame graph shows that new function calls to OrganizationTeamsEndpoint.dispatch were introduced, which made the code run slower, while calls to ChunkUploadEndpoint.get decreased.

If you look closely, you'll notice that the top right corner has a Before -> After and After -> Before toggle. This is because by default, differential flame graphs draw the aggregate flame graph from after a regression has occurred as the source of truth. This means that any code that may have been removed, will no longer be drawn (after all, it's not there anymore). This is why negating the view is useful, as it allows us to compare the data from before and peek into the future of how our code will change.

`` Here's an example of what our module lock problem looks like when using the profiling data from after the regression has occurred as the source of truth:

Differential flame graph new functions

And here's what it looks like if we're using the data from before the regression occurred as our source of truth:

Differential flame graph new functions

By using the Before -> After toggle we can see that ModuleLock.acquire was faster before the regression, but we can also see some of the function calls inside our ChunkUploadEndpoint, which are no longer being called.

While differential flame graphs in Sentry use a different color scheme and pull from a different source of data than regular flame graphs, you can pan and zoom around the chart and search for frames the same way you do with regular flame graph.

The table under the differential flame graph shows both the largest increases and the largest decreases in function duration.

Differential flame graph table

Differential flame graphs are a good way to quickly visualize potential sources of performance regressions or improvements. Currently, they're only available as part of the automated function regression workflow, but we're working on adding them to other parts of the profiling workflow.

Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").