.. You should enable this project on travis-ci.org and coveralls.io to make
these badges work. The necessary Travis and Coverage config files have been
generated for you.
.. image:: https://travis-ci.org/davidread/ckanext-downloadall.svg?branch=master
:target: https://travis-ci.org/davidread/ckanext-downloadall
.. image:: https://img.shields.io/pypi/v/ckanext-downloadall.svg
:target: https://pypi.python.org/project/ckanext-downloadall/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/pyversions/ckanext-downloadall.svg
:target: https://pypi.python.org/project/ckanext-downloadall/
:alt: Supported Python versions
.. image:: https://img.shields.io/pypi/status/ckanext-downloadall.svg
:target: https://pypi.python.org/project/ckanext-downloadall/
:alt: Development Status
.. image:: https://img.shields.io/pypi/l/ckanext-downloadall.svg
:target: https://pypi.python.org/project/ckanext-downloadall/
:alt: License
===================
ckanext-downloadall
===================
This CKAN extension adds a "Download all" button to datasets. This downloads
a zip file containing all the resource files and a datapackage.json.
.. image:: demo.png
This zip file is a good way to package data for storing or sending, because:
* you keep all the data files together
* you include the documentation (metadata) - avoids the common problem of being
handed some data files and not know anything about it or where to find info
* the metadata is machine-readable, so can be used by tools, software and in
automated workflows. For example:
* validating a series of data releases all meet a standard schema
* loading it into a database, using the column types and foreign key
relations specified in the metadata
The `datapackage.json <https://frictionlessdata.io/specs/data-package/>`_ is a
`Frictionless Data <https://frictionlessdata.io/specs/data-package/>`_
standard, also known as a Data Package.
Technical notes
~~~~~~~~~~~~~~~
If the resource is pushed/xloaded to DataStore then the schema (column types)
is also included in the datapackage.json file.
This extension uses a CKAN background job to create the zip every time a
dataset is created or updated (or its data dictionary is changed). This suits
CKANs where all files are uploaded - if the underlying data file changes
without the CKAN URL changing, then the zip will not include the update (until
something else triggers the zip to update).
(This extension is inspired by `ckanext-packagezip
<https://github.com/datagovuk/ckanext-packagezip>`_, but that is old and relied
on ckanext-archiver and IPipe.)
------------
Requirements
------------
Designed to work with CKAN 2.7+
Ideally it is used in conjunction with DataStore and
`xloader <https://github.com/ckan/ckanext-xloader>`_ (or datapusher), so that the
Data Dictionary is included as a schema in the datapackage.json, to describe
the column types.
------------
Installation
------------
To install ckanext-downloadall:
1. Activate your CKAN virtual environment, for example::
. /usr/lib/ckan/default/bin/activate
2. Install the ckanext-downloadall Python package into your virtual environment::
pip install ckanext-downloadall
3. Add ``downloadall`` to the ``ckan.plugins`` setting in your CKAN
config file (by default the config file is located at
``/etc/ckan/default/production.ini``). e.g. ::
ckan.plugins = downloadall
4. Restart the CKAN worker. For example if you've deployed it with supervisord::
sudo supervisorctl restart ckan-worker:ckan-worker-00
5. Restart CKAN server. For example if you've deployed CKAN with Apache on
Ubuntu::
sudo service apache2 reload
6. Ensure the background job 'worker' process is running - see
https://docs.ckan.org/en/2.8/maintaining/background-tasks.html#running-background-jobs
---------------
Config Settings
---------------
::
# Include additional fields from the dataset in the datapackage.json (e.g.
# those defined in a ckanext-scheming schema)
# (optional, space separated list).
ckanext.downloadall.dataset_fields_to_add_to_datapackage = district county
----------------------
Command-line interface
----------------------
There is a command-line interface::
downloadall --help
Examples of use::
downloadall update-zip gold-prices
downloadall update-all-zips
---------------
Troubleshooting
---------------
**"All resource data" appears as a normal resource, instead of seeing a
"Download All" button**
You need to enable this extension in the CKAN config and restart the server.
See the Installation section above.
**ImportError: No module named datapackage**
This means you have an older version of ckanapi, which is a dependency of
ckanext-downloadall. Install a newer version.
**OSError: [Errno 13] Permission denied: '/data/ckan/resources/c89'**
You are trying to update zips from the command-line but running the tasks
synchronously, rather than with the normal worker process. In this case you
need to run it as the `www-data` user e.g.::
sudo -u www-data /usr/lib/ckan/default/bin/downloadall -c /etc/ckan/default/production.ini update-all-zips --synchronous
------------------------
Development Installation
------------------------
To install ckanext-downloadall for development, activate your CKAN virtualenv
and do::
git clone https://github.com/davidread/ckanext-downloadall.git
cd ckanext-downloadall
python setup.py develop
pip install -r dev-requirements.txt
Remember to run the worker (in a separate terminal)::
paster --plugin=ckan jobs worker --config=/etc/ckan/default/development.ini
-----------------
Running the Tests
-----------------
To run the tests, do::
nosetests --nologcapture --with-pylons=test.ini
To run the tests and produce a coverage report, first make sure you have
coverage installed in your virtualenv (``pip install coverage``) then run::
nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.downloadall --cover-inclusive --cover-erase --cover-tests
----------------------------------------------
Releasing a New Version of ckanext-downloadall
----------------------------------------------
ckanext-downloadall is availabe on PyPI as https://pypi.org/project/ckanext-downloadall/.
To publish a new version to PyPI follow these steps:
1. Update the version number in the ``setup.py`` file.
See `PEP 440 <http://legacy.python.org/dev/peps/pep-0440/#public-version-identifiers>`_
for how to choose version numbers.
2. Update the CHANGELOG.md with details of this release.
3. Make sure you have the latest version of necessary packages::
pip install --upgrade setuptools wheel twine
4. Create a source and binary distributions of the new version::
python setup.py sdist bdist_wheel && twine check dist/*
Fix any errors you get.
5. Upload the source distribution to PyPI::
twine upload dist/*
6. Commit any outstanding changes::
git commit -a
git push
7. Tag the new release of the project on GitHub with the version number from
the ``setup.py`` file. For example if the version number in ``setup.py`` is
0.0.1 then do::
git tag 0.0.1
git push --tags