Celery Beat Infrastructure Testing Guide¶

Overview¶

This guide provides tools to test your Celery Beat infrastructure and verify that scheduled tasks work correctly.

Sentry Error Tracking Integration¶

Celery Beat and Worker services are configured with Sentry error tracking for comprehensive monitoring of scheduled tasks.

How It Works¶

The PiqueTickets application uses signal-based Sentry initialization for Celery services:

Celery Beat: Initializes Sentry on startup via the beat_init signal with monitor_beat_tasks=True enabled
Celery Worker: Initializes Sentry on startup via the celeryd_init signal
Configuration: Reuses PII scrubbing and environment settings from Django settings
Environment-based Sampling:
Development: 100% of errors captured
Production: 10% sampling to manage costs

Verification¶

Check that Sentry is active:

# Check Celery Beat logs for Sentry initialization
docker compose logs celery_beat | grep "Sentry initialized"
# Expected: "Sentry initialized for Celery - Environment: development, Beat Monitoring: Enabled"

# Verify task monitoring is active
# Look for sentry-task-enqueued-time in task messages
docker compose logs celery_worker | grep "sentry-task"

Monitoring in Sentry Dashboard¶

Navigate to your Sentry dashboard
Filter by environment:development or environment:production
Look for Celery-related events tagged with:
celery_task_name: The task that failed
celery_task_id: Unique task identifier
Integration type: celery

Implementation Details¶

See apps/api/brktickets/celery.py for signal-based initialization: - init_sentry_for_celery(): Main initialization function - @signals.celeryd_init.connect: Worker initialization - @signals.beat_init.connect: Beat scheduler initialization

Testing Tools¶

Option 1: Django Management Command (Recommended)¶

# Test with default 3-minute delay
python manage.py test_celery_beat

# Test with custom delay
python manage.py test_celery_beat --minutes 5

# Clean up all test tasks
python manage.py test_celery_beat --cleanup

Option 2: Standalone Python Script¶

# Test with default 3-minute delay
python test_beat_infrastructure.py

# Test with custom delay
python test_beat_infrastructure.py --minutes 5

# Clean up all test tasks
python test_beat_infrastructure.py --cleanup

Option 3: Docker Exec (For containerized environments)¶

# Using docker compose
docker compose exec api python manage.py test_celery_beat

# Using docker directly
docker exec -it <api-container-id> python manage.py test_celery_beat

What the Test Does¶

The test script performs the following checks:

1. Database Connection Test ✓¶

Verifies PostgreSQL connectivity
Checks for django_celery_beat tables
Confirms migrations are up to date

2. Redis Connection Test ✓¶

Tests Redis cache connectivity
Verifies read/write operations
Confirms cache backend configuration

3. Celery Configuration Test ✓¶

Checks Celery app initialization
Verifies broker URL
Confirms timezone settings
Validates DatabaseScheduler is configured

4. Task Creation Test ✓¶

Creates a one-off scheduled task
Schedules it to run in N minutes (default: 3)
Uses existing send_show_reminder task
Provides monitoring instructions

Expected Output¶

======================================================================
  CELERY BEAT INFRASTRUCTURE TEST
======================================================================

[1/4] Testing Database Connection...
  ✓ Database connected
  ✓ Found 4 django_celery_beat tables

[2/4] Testing Redis Connection...
  ✓ Redis connected
  ✓ Cache backend: RedisCache

[3/4] Testing Celery Configuration...
  ✓ Celery app name: brktickets
  ✓ Broker: redis://localhost:6379/0...
  ✓ Timezone: America/Phoenix
  ✓ Beat scheduler: django_celery_beat.schedulers:DatabaseScheduler
  ✓ Using DatabaseScheduler (correct)

[4/4] Creating Test Task (scheduled in 3 minutes)...
  ✓ Test task created: test_celery_beat_20251014_153000
  ✓ Scheduled for: 2025-10-14 15:30:00 MST
  ✓ Task ID: 42
  ℹ This will fail gracefully (order 999999 does not exist)

======================================================================
  CURRENTLY SCHEDULED TASKS
======================================================================

🟢 [TEST] test_celery_beat_20251014_153000
    Scheduled: 2025-10-14 15:30:00 MST
    Task: tickets.task.send_show_reminder
    Status: Pending (3 minutes)

🟢 [PROD] show_reminder_order_123
    Scheduled: 2025-10-15 15:00:00 MST
    Task: tickets.task.send_show_reminder
    Status: Pending (1440 minutes)

======================================================================
  MONITORING INSTRUCTIONS
======================================================================

1. Check Celery Beat logs:
   docker compose logs -f celery_beat
   # Look for: "Scheduler: Sending due task test_celery_beat_..."

2. Check Celery Worker logs:
   docker compose logs -f celery_worker
   # Look for task execution

3. Check in Django Admin:
   http://localhost:8000/admin/django_celery_beat/periodictask/

4. Watch for execution:
   # Task should execute in approximately 3 minutes
   # Expected: Task will fail gracefully (order 999999 not found)
   # Success = Task was picked up and executed on schedule

5. Clean up test tasks:
   python manage.py test_celery_beat --cleanup

======================================================================
  ✓ Infrastructure test setup complete!
======================================================================

Monitoring the Test¶

1. Watch Celery Beat Logs¶

In a separate terminal, watch the beat scheduler:

docker compose logs -f celery_beat

Look for this line when the task executes:

celery_beat | Scheduler: Sending due task test_celery_beat_20251014_153000

2. Watch Celery Worker Logs¶

In another terminal, watch the worker:

docker compose logs -f celery_worker

Look for task execution logs:

celery_worker | [2025-10-14 15:30:00,123: INFO/MainProcess] Received task: tickets.task.send_show_reminder
celery_worker | [2025-10-14 15:30:00,456: WARNING/ForkPoolWorker-1] Order with id 999999 does not exist

3. Check Django Admin¶

Navigate to:

http://localhost:8000/admin/django_celery_beat/periodictask/

You should see the test task with: - Name: test_celery_beat_YYYYMMDD_HHMMSS - Task: tickets.task.send_show_reminder - One-off: ✓ - Enabled: ✓ - Clocked: (your scheduled time)

Understanding Results¶

✓ Success Indicators¶

Task is scheduled: Test task appears in admin and logs
Beat picks it up: Beat logs show task being sent at scheduled time
Worker executes: Worker logs show task execution
Expected error: Task fails with "Order does not exist" (this is correct!)

✗ Failure Indicators¶

Database errors: Run python manage.py migrate
Redis errors: Check Redis is running: docker compose ps redis
Task not executed: Check both beat and worker are running
Wrong scheduler: Verify CELERY_BEAT_SCHEDULER in settings.py

Troubleshooting¶

Task doesn't execute¶

Check beat is running:

docker compose ps celery_beat
# Should show "Up"

Check worker is running:

docker compose ps celery_worker
# Should show "Up"

Restart services:

docker compose restart celery_beat celery_worker

Database tables missing¶

Run migrations:

docker compose exec api python manage.py migrate

Redis connection failed¶

Check Redis:

docker compose ps redis
# Should show "healthy"

Restart Redis:

docker compose restart redis

Wrong timezone¶

Check settings:

# brktickets/settings.py
TIME_ZONE = "America/Phoenix"  # Should match your location
CELERY_TIMEZONE = TIME_ZONE    # Should match TIME_ZONE

Cleanup¶

After testing, clean up test tasks:

python manage.py test_celery_beat --cleanup

This removes all tasks starting with test_celery_beat_.

Production Testing¶

To test the actual show reminder system in production:

Create a test show 2-3 days in the future
Create a test order for that show
Check that a reminder task is scheduled
Monitor logs on the day of the reminder
Verify email is sent

Files Created¶

/tickets/management/commands/test_celery_beat.py - Django management command
/test_beat_infrastructure.py - Standalone script
/CELERY_BEAT_TESTING.md - This guide