5. Integrate Celery & Django¶

5.1. Create default folders/files:¶

Create a folder in time_tasks called celery and add __init__.py & conf.py files.

mkdir time_tasks/celery
echo "" > time_tasks/celery/__init__.py
echo "" > time_tasks/celery/conf.py

5.2. Create `time_tasks/celery/init.py`¶

# time_tasks/celery/__init__.py
from __future__ import absolute_import, unicode_literals
import os

from celery import Celery
from celery.schedules import crontab

# You can use rabbitmq instead here.
BASE_REDIS_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379')

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'time_tasks.settings')

app = Celery('time_tasks')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

app.conf.broker_url = BASE_REDIS_URL

# this allows you to schedule items in the Django admin.
app.conf.beat_scheduler = 'django_celery_beat.schedulers:DatabaseScheduler'

The time_tasks.settings in os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'time_tasks.settings') is also set in manage.py so, when in doubt, use the value in manage.py for your project. This is for our celery worker process. My Django project name is time_tasks so I use it for celery as well with app = Celery('time_tasks').

5.3. Create `time_tasks/celery/conf.py`¶

# time_tasks/celery/conf.py

# This sets the django-celery-results backend
CELERY_RESULT_BACKEND = 'django-db'

5.4. Update `time_tasks/init.py`¶

This file is next to settings.py and the time_tasks/celery folder created above.

# time_tasks/__init__.py
from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

5.5. Update `time_tasks/settings.py`¶

# time_tasks/settings.py
INSTALLED_APPS += [
    'django_celery_beat,
    'django_celery_results,
]

from .celery.conf import *

Now run:

python manage.py migrate

4. Setup Project 6. Web Scraping Client