Skip to main content

Celery Background Task Processing - README

Production-ready Celery integration for asynchronous task processing in CODITECT License Platform.

Quick Start​

Prerequisites​

  • Redis server running
  • Django application configured
  • Dependencies installed (pip install -r requirements.txt)

Run Locally (3 Commands)​

# Terminal 1: Start Redis
docker run -d -p 6379:6379 redis:7-alpine

# Terminal 2: Start Celery Worker
celery -A license_platform worker --loglevel=info

# Terminal 3: Start Celery Beat (scheduler)
celery -A license_platform beat --loglevel=info

Deploy to Kubernetes (2 Commands)​

kubectl apply -f k8s/celery-worker-deployment.yaml
kubectl apply -f k8s/celery-beat-deployment.yaml

What's Included​

Background Tasks​

  1. Zombie Session Cleanup (Hourly)

    • Automatically cleans up sessions that expired in Redis
    • Runs every hour on the hour
    • Processes in batches of 100 for performance

  2. License Expiration Checks (Daily)

    • Checks for licenses expiring within 30 days
    • Sends notifications at 30, 7, 1 days before expiration
    • Email integration pending

  3. Usage Metrics Aggregation (TBD)

    • Placeholder for future analytics processing

Architecture Components​

  • Django Application - Enqueues tasks
  • Redis - Message broker and result backend
  • Celery Workers - Process tasks asynchronously (2-10 pods)
  • Celery Beat - Schedules periodic tasks (1 pod)

Testing Tasks​

Django Shell​

python manage.py shell

from licenses.tasks import cleanup_zombie_sessions

# Execute task
task = cleanup_zombie_sessions.delay()

# Check status
print(task.id)
print(task.status)

# Get result
result = task.get(timeout=10)
print(result)
# {'sessions_found': 5, 'sessions_cleaned': 5, 'errors': 0, 'duration_seconds': 0.45}

Run Tests​

# Run Celery task tests
pytest tests/test_celery_tasks.py -v

# Run with coverage
pytest tests/test_celery_tasks.py --cov=licenses.tasks --cov-report=html

Monitoring​

View Logs​

# Local development
celery -A license_platform events

# Kubernetes
kubectl logs -f deployment/celery-worker -n coditect-license-platform
kubectl logs -f deployment/celery-beat -n coditect-license-platform

Check Worker Status​

celery -A license_platform inspect active
celery -A license_platform inspect stats

Configuration​

Environment Variables​

VariableDescriptionDefault
REDIS_HOSTRedis hostnamelocalhost
REDIS_PORTRedis port6379
REDIS_DBRedis database0
REDIS_PASSWORDRedis passwordNone

Celery Settings​

Configured in license_platform/celery.py:

  • Broker: Redis
  • Result Backend: Redis
  • Serialization: JSON
  • Timezone: UTC
  • Task Time Limit: 5 minutes (hard), 4 minutes (soft)
  • Worker Concurrency: 4 tasks per worker
  • Max Tasks Per Child: 1000 (restart after 1000 tasks)

Documentation​

File Structure​

coditect-cloud-backend/
├── license_platform/
│   ├── celery.py              # Celery app configuration
│   └── __init__.py            # Auto-load Celery
├── licenses/
│   └── tasks.py               # Task definitions
├── k8s/
│   ├── celery-worker-deployment.yaml  # Worker pods + HPA
│   └── celery-beat-deployment.yaml    # Beat scheduler
├── tests/
│   └── test_celery_tasks.py   # Task tests
├── docs/
│   ├── celery-integration.md  # Comprehensive docs
│   └── celery-quick-start.md  # Quick reference
└── requirements.txt            # Dependencies (celery, django-redis)

Troubleshooting​

Workers Not Processing Tasks​

# Check Redis connection
redis-cli ping  # Should return PONG

# Restart workers
kubectl rollout restart deployment/celery-worker -n coditect-license-platform

Beat Not Scheduling Tasks​

# Check beat logs
kubectl logs deployment/celery-beat -n coditect-license-platform

# Verify schedule
celery -A license_platform inspect scheduled

Task Failures​

# View task traceback
from celery.result import AsyncResult

task_id = 'your-task-id'
result = AsyncResult(task_id)

if result.failed():
    print(result.traceback)

Production Checklist​

  • Dependencies installed (celery, django-redis)
  • Celery app configured
  • Tasks defined and tested
  • Redis connection verified
  • Kubernetes manifests created
  • Health checks implemented
  • Autoscaling configured (HPA)
  • Documentation complete
  • Email integration (license expirations)
  • Monitoring/alerting setup
  • Production deployment

Support​

For issues or questions:

  1. Check documentation in docs/celery-integration.md
  2. Review implementation summary in CELERY-implementation-summary.md
  3. Run tests: pytest tests/test_celery_tasks.py
  4. Check logs: kubectl logs -f deployment/celery-worker

Status: Production-Ready ✅

Last Updated: November 30, 2025

Maintainer: CODITECT Platform Team