Celery for Python

Learn how to set up Sentry in your Celery app, capture your first errors and traces, and view them in Sentry.

You need:

  • A Sentry account and project
  • Your application up and running
  • Celery 4.4.7+
  • Python 3.6+

Run the command for your preferred package manager to add the Sentry SDK to your application:

Copied
pip install "sentry-sdk"

Choose the features you want to configure, and this guide will show you how:

Want to learn more about these features?
  • Issues (always enabled): Sentry's core error monitoring product that automatically reports errors, uncaught exceptions, and unhandled rejections. If you have something that looks like an exception, Sentry can capture it.
  • Tracing: Track software performance while seeing the impact of errors across multiple systems. For example, distributed tracing allows you to follow a request from the frontend to the backend and back.
  • Profiling: Gain deeper insight than traditional tracing without custom instrumentation, letting you discover slow-to-execute or resource-intensive functions in your app.
  • Logs: Centralize and analyze your application logs to correlate them with errors and performance issues. Search, filter, and visualize log data to understand what's happening in your applications.
  • Application Metrics (always enabled): Track and analyze custom application metrics, such as response times and database query durations, to understand trends and patterns in your application's performance and behavior over time.

Configuration should happen as early as possible in your application's lifecycle.

If you have the celery package in your dependencies, the Celery integration will be enabled automatically when you initialize the Sentry SDK.

When using Celery without Django, you'll need to initialize the Sentry SDK in both your application and the Celery worker processes spawned by the Celery daemon.

tasks.py
Copied
from celery import Celery, signals
import sentry_sdk
# ___PRODUCT_OPTION_START___ metrics
from sentry_sdk import metrics
# ___PRODUCT_OPTION_END___ metrics

# Initializing Celery
app = Celery("tasks", broker="...")

# Initialize Sentry SDK on Celery startup
@signals.celeryd_init.connect
def init_sentry(**_kwargs):
    sentry_sdk.init(
        dsn="___PUBLIC_DSN___",
        # Add request headers and IP for users,
        # see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info
        send_default_pii=True,
        # ___PRODUCT_OPTION_START___ performance
        # Set traces_sample_rate to 1.0 to capture 100%
        # of transactions for tracing.
        traces_sample_rate=1.0,
        # ___PRODUCT_OPTION_END___ performance
        # ___PRODUCT_OPTION_START___ profiling
        # To collect profiles for all profile sessions,
        # set `profile_session_sample_rate` to 1.0.
        profile_session_sample_rate=1.0,
        # Profiles will be automatically collected while
        # there is an active span.
        profile_lifecycle="trace",
        # ___PRODUCT_OPTION_END___ profiling
    )

# Task definitions go here
@app.task
def add(x, y):
    return x + y

The celeryd_init signal is triggered when the Celery daemon starts, before the worker processes are spawned. If you need to initialize Sentry for each individual worker process, use the worker_init signal instead.

main.py
Copied
from tasks import add
import sentry_sdk
# ___PRODUCT_OPTION_START___ metrics
from sentry_sdk import metrics
# ___PRODUCT_OPTION_END___ metrics

def main():
    # Initializing Sentry SDK in our process
    sentry_sdk.init(
        dsn="___PUBLIC_DSN___",
        # Add data like request headers and IP for users, if applicable;
        # see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info
        send_default_pii=True,
        # ___PRODUCT_OPTION_START___ performance
        # Set traces_sample_rate to 1.0 to capture 100%
        # of transactions for tracing.
        traces_sample_rate=1.0,
        # ___PRODUCT_OPTION_END___ performance
        # ___PRODUCT_OPTION_START___ profiling
        # To collect profiles for all profile sessions,
        # set `profile_session_sample_rate` to 1.0.
        profile_session_sample_rate=1.0,
        # Profiles will be automatically collected while
        # there is an active span.
        profile_lifecycle="trace",
        # ___PRODUCT_OPTION_END___ profiling
    )

    # Enqueueing a task to be processed by Celery
    with sentry_sdk.start_transaction(name="calling-a-celery-task"):
    # or sentry_sdk.traces.start_span(name="calling-a-celery-task", parent_span=None) in stream mode
        result = add.delay(4, 4)

if __name__ == "__main__":
    main()

If you're using Celery with Django in a typical setup, have initialized the SDK in your settings.py file (as described in the Django integration documentation), and have your Celery configured to use the same settings as config_from_object, there's no need to initialize the Celery SDK separately.

To further customize your setup, review the Options section below.

Sentry automatically captures errors and exceptions raised in your Celery tasks and reports them as issues.

To learn how to manually report issues, see Capturing Errors.

The Sentry SDK automatically creates spans for your Celery tasks, and propagates the trace from the code that enqueues a task to the worker that runs it.

You can also manually capture performance data – see Custom Instrumentation to learn more.

Distributed tracing extends the trace from the code running your Celery task to include the code that initiated the task.

You can disable this globally with the propagate_traces option, documented in the options below. If you set propagate_traces to False, all Celery tasks will start their own trace.

If you want to have more fine-grained control over trace distribution, you can override the propagate_traces option by passing the sentry-propagate-traces header when starting the Celery task:

Copied
import sentry_sdk

# Enable global distributed traces (this is the default, just to be explicit)
sentry_sdk.init(
    # same as above
    integrations=[
        CeleryIntegration(
            propagate_traces=True
        ),
    ],
)

# This will propagate the trace:
my_task_a.delay("some parameter")

# This will propagate the trace:
my_task_b.apply_async(
    args=("some_parameter", )
)

# This will NOT propagate the trace. The task will start its own trace:
my_task_b.apply_async(
    args=("some_parameter", ),
    headers={"sentry-propagate-traces": False},
)

# Note: overriding the tracing behaviour using `task_x.delay()` is not possible.

Let's test your setup and confirm that data reaches your Sentry project.

To verify that Sentry captures errors and creates issues in your Sentry project, add this intentional error to your application:

tasks.py
Copied
from celery import Celery, signals
import sentry_sdk

app = Celery("tasks", broker="...")

@signals.celeryd_init.connect
def init_sentry(**_kwargs):
    sentry_sdk.init(...)  # same as above

@app.task
def debug_sentry():
    1/0

Trigger the error by calling debug_sentry.delay().

To confirm that your SDK is initialized on worker start, pass debug=True to sentry_sdk.init(). This will add extra output to your Celery logs when the SDK is initialized. If you see the output during worker startup, and not just after a task has started, then it's working correctly.

To test your tracing configuration, create a custom transaction and span:

Copied
import sentry_sdk

with sentry_sdk.start_transaction(op="task", name="Transaction Name"):
    span = sentry_sdk.start_span(name="Custom Span Name")
    span.finish()

To verify that Sentry catches your logs (which are enabled by default), add some log statements to your application:

Copied
import sentry_sdk

sentry_sdk.logger.info("This is an info log message")
sentry_sdk.logger.warning("This is a warning message")
sentry_sdk.logger.error("This is an error message")

Send test metrics from your app to verify that metrics are arriving in Sentry:

Copied
from sentry_sdk import metrics

metrics.count("checkout.failed", 1)
metrics.gauge("queue.depth", 42)
metrics.distribution("cart.amount_usd", 187.5)

Now, head over to your project on Sentry.io to view the collected data (it takes a couple of moments for the data to appear).

Need help locating the captured errors in your Sentry project?
  • Open the Issues page and select an error from the issues list to view the full details and context of this error. For more details, see the Issue Details documentation.
  • Open the Traces page and select a trace to reveal more information about each span, its duration, and any errors. For an interactive UI walkthrough, click here.
  • Open the Profiles page, select a transaction, and then a profile ID to view its flame graph. For more information, click here.
  • Open the Logs page and filter by service, environment, or search keywords to view log entries from your application. For an interactive UI walkthrough, click here.
  • Open the Application Metrics page to view and analyze your metrics. For more details, see this interactive walkthrough.

To set options on CeleryIntegration to change its behavior, add it explicitly to your sentry_sdk.init():

Copied
import sentry_sdk
from sentry_sdk.integrations.celery import CeleryIntegration

sentry_sdk.init(
    # same as above
    integrations=[
        CeleryIntegration(
            monitor_beat_tasks=True,
            exclude_beat_tasks=[
                "unimportant-task",
                "payment-check-.*"
            ],
        ),
    ],
)

You can pass the following keyword arguments to CeleryIntegration():

propagate_traces

Typebool
DefaultTrue

Propagate Sentry tracing information to the Celery task. This makes it possible to link Celery task errors to the function that triggered the task.

If this is set to False:

  • errors in Celery tasks won't be matched to the triggering function.
  • your Celery tasks will start a new trace and won't be connected to the trace in the calling function.

See Distributed Traces to learn how to get more fine-grained control over distributed tracing in Celery tasks.

monitor_beat_tasks

Typebool
DefaultFalse

Turn auto-instrumentation on or off for Celery Beat tasks using Sentry Crons.

See Celery Beat Auto Discovery to learn more.

exclude_beat_tasks

Typelist[str]
DefaultNone

A list of Celery Beat tasks that should be excluded from auto-instrumentation using Sentry Crons. Only applied if monitor_beat_tasks is set to True.

The list can contain strings with the names of tasks in the Celery Beat schedule to be excluded. It can also include regular expressions to match multiple tasks. For example, if you include "payment-check-.*" every task starting with payment-check- will be excluded from auto-instrumentation.

See Celery Beat Auto Discovery to learn more.

At this point, you should have integrated Sentry into your application and should already be sending data to your Sentry project.

Now's a good time to customize your setup and look into more advanced topics. Our next recommended steps for you are:

Are you having problems setting up the SDK?
Previous
Bottle
Next
Crons
Was this helpful?
Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").