django-queued-storage
=====================
This storage backend enables having a local and a remote storage
backend. It will save any file locally and queue a task to transfer it
somewhere else.
If the file is accessed before it's transferred, the local copy is
returned.
The default tasks use `Celery <http://celeryproject.org/>`_ for queing
transfer tasks but is agnostic about your choice.
Installation
------------
::
pip install django-queued-storage
Configuration
-------------
- Follow the configuration instructions for
`django-celery <https://github.com/ask/django-celery>`_
- Set up a `caching
backend <https://docs.djangoproject.com/en/1.3/topics/cache/#setting-up-the-cache>`_
- Add ``queued_storage`` to your ``INSTALLED_APPS`` tuple
Usage
-----
The ``QueuedRemoteStorage`` can be used as a drop-in replacement
wherever ``django.core.file.storage.Storage`` might otherwise be
required.
This example is using
`django-storages <http://code.welldev.org/django-storages/>`_ for the
remote backend:
::
from django.db import models
from queued_storage.storage import QueuedRemoteStorage
class MyModel(models.Model):
image = ImageField(storage = QueuedRemoteStorage(
local = 'django.core.files.storage.FileSystemStorage',
remote = 'storages.backends.s3boto.S3BotoStorage'))
Backends
--------
- ``queued_storage.backend.QueuedRemoteStorage``:
Base class for queued storages. You can use this to specify your own
backends.
- ``queued_storage.backend.DoubleFilesystemStorage``:
Used for testing, but can be handy if you want uploaded files to be
stored in two places. Example:
::
image = ImageField(
storage = DoubleFilesystemStorage(
local_kwargs = {'location': '/backup'},
remote_kwargs = {'location': settings.MEDIA_ROOT}))
- ``queued_storage.backend.S3Storage``:
Shortcut for the above example.
::
image = ImageField(storage = S3Storage())
- ``queued_storage.backend.DelayedStorage``:
This backend does *not* transfer files to the remote location
automatically.
::
image = ImageField(storage = DelayedStorage(
'django.core.files.storage.FileSystemStorage',
'storages.backends.s3boto.S3BotoStorage'))
>>> m = MyModel(image = File(open('image.png')))
>>> # Save locally:
>>> m.save()
>>> # Transfer to remote location:
>>> m.file.storage.transfer(m.file.name)
Useful if you want to do preprocessing
Fields
------
- ``queued_storage.backend.RemoteFileField``:
Tiny wrapper around any ``QueuedRemoteStorage``, provides a
convenient method to transfer files. The above ``DelayedStorage``
example would look like this:
::
image = RemoteFileField(storage = DelayedStorage(
'django.core.files.storage.FileSystemStorage',
'storages.backends.s3boto.S3BotoStorage'))
>>> m = MyModel(image = File(open('image.png')))
>>> # Save locally:
>>> m.save()
>>> # Transfer to remote location:
>>> m.file.transfer()
Tasks
-----
- ``queued_storage.backend.Transfer``:
The default task. Transfers to a remote location. The actual
transfer is implemented in the remote backend.
- ``queued_storage.backend.TransferAndDelete``:
Once the file was transferred, the local copy is deleted.
To create new tasks, do something like this:
::
from celery.registry import tasks
from queued_storage.backend import Transfer
class TransferAndDelete(Transfer):
def transfer(self, name, local, remote, **kwargs):
result = super(TransferAndDelete, self).transfer(name, local, remote, **kwargs)
if result:
local.delete(name)
return result
tasks.register(TransferAndDelete)
The result is ``True`` if the transfer was successful, else ``False``
and the task is retried.
In case you don't want to use Celery, have a look
`here <https://github.com/flashingpumpkin/django-queued-storage/blob/master/queued_storage/tests/__init__.py#L80>`_.
To use a different task, pass it into the backend:
::
image = models.ImageField(storage = S3Storage(task = TransferAndDelete))
Settings
--------
- ``QUEUED_STORAGE_CACHE_KEY``:
Use a different key for caching.
- ``QUEUED_STORAGE_RETRIES``:
How many retries should be attempted before aborting.
- ``QUEUED_STORAGE_RETRY_DELAY``:
The delay between retries.
RELEASE NOTES
=============
v0.3 - *BACKWARDS INCOMPATIBLE*
- Added tests
- Added ``S3Storage`` and ``DelayedStorage``
- Added ``TransferAndDelete`` task
- Classes renamed to be consistent