معرفی شرکت ها
django-elastic-migrations-0.8.2
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Manage Elasticsearch Indexes in Django
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | django-elastic-migrations-0.8.2 |
| نام | django-elastic-migrations |
| نسخه کتابخانه | 0.8.2 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Harvard Business School, HBX Department |
| ایمیل نویسنده | pnore@hbs.edu |
| آدرس صفحه اصلی | https://github.com/HBS-HBX/django-elastic-migrations |
| آدرس اینترنتی | https://pypi.org/project/django-elastic-migrations/ |
| مجوز | MIT |
Django Elastic Migrations
=========================
`django-elastic-migrations`_ is a Django app for creating, indexing and changing schemas of Elasticsearch indexes.
.. image:: https://travis-ci.com/HBS-HBX/django-elastic-migrations.svg?branch=master
:target: https://travis-ci.com/HBS-HBX/django-elastic-migrations
:alt: Build Status
.. image:: https://codecov.io/gh/HBS-HBX/django-elastic-migrations/branch/master/graph/badge.svg
:target: https://codecov.io/gh/HBS-HBX/django-elastic-migrations
:alt: codecov
.. _django-elastic-migrations: https://pypi.org/project/django-elastic-migrations/
Overview
--------
Elastic has given us basic python tools for working with its search indexes:
* `elasticsearch-py`_, a python interface to elasticsearch's REST API
* `elasticsearch-dsl-py`_, a Django-esque way of declaring Elasticsearch schemas,
built upon `elasticsearch-py`_
Django Elastic Migrations adapts these tools into a Django app which also:
* Provides Django management commands for ``list``\ ing indexes, as well as performing
``create``, ``update``, ``activate`` and ``drop`` actions on them
* Implements concurrent bulk indexing powered by python ``multiprocessing``
* Gives Django test hooks for Elasticsearch
* Records a history of all actions that change Elasticsearch indexes
* Supports AWS Elasticsearch 6.0, 6.1 (6.2 TBD; see `#3 support elasticsearch-dsl 6.2`_)
* Enables having two or more servers share the same Elasticsearch cluster
.. _elasticsearch-py: https://github.com/elastic/elasticsearch-py
.. _elasticsearch-dsl-py: https://github.com/elastic/elasticsearch-dsl-py
.. _#3 support elasticsearch-dsl 6.2: https://github.com/HBS-HBX/django-elastic-migrations/issues/3
Models
^^^^^^
Django Elastic Migrations provides comes with three Django models:
**Index**, **IndexVersion**, and **IndexAction**:
*
**Index** - a logical reference to an Elasticsearch index.
Each ``Index`` points to multiple ``IndexVersions``, each of which contains
a snapshot of that ``Index`` schema at a particular time. Each ``Index`` has an
*active* ``IndexVersion`` to which all actions are directed.
*
**IndexVersion** - a snapshot of an Elasticsearch ``Index`` schema at a particular
point in time. The Elasticsearch index name is the name of the *Index* plus the
primary key id of the ``IndexVersion`` model, e.g. ``movies-1``. When the schema is
changed, a new ``IndexVersion`` is added with name ``movies-2``, etc.
*
**IndexAction** - a record of a change that impacts an ``Index``, such as updating
the index or changing which ``IndexVersion`` is active in an ``Index``.
Management Commands
^^^^^^^^^^^^^^^^^^^
Use ``./manage.py es --help`` to see the list of all of these commands.
Read Only Commands
~~~~~~~~~~~~~~~~~~
* ``./manage.py es_list``
* help: For each *Index*\ , list activation status and doc
count for each of its *IndexVersions*
* usage: ``./manage.py es_list``
Action Commands
~~~~~~~~~~~~~~~
These management commands add an Action record in the database,
so that the history of each *Index* is recorded.
* ``./manage.py es_create`` - create a new index.
* ``./manage.py es_activate`` - *activate* a new ``IndexVersion``. all
updates and reads for that ``Index`` by will then go to that version.
* ``./manage.py es_update`` - update the documents in the index.
* ``./manage.py es_clear`` - remove the documents from an index.
* ``./manage.py es_drop`` - drop an index.
* ``./manage.py es_dangerous_reset`` - erase elasticsearch and reset the
Django Elastic Migrations models.
For each of these, use ``--help`` to see the details.
Usage
^^^^^
Installation
~~~~~~~~~~~~
#. ``pip install django-elastic-migrations``; see `django-elastic-migrations`_ on PyPI
#. Put a reference to this package in your ``requirements.txt``
#. Ensure that a valid ``elasticsearch-dsl-py`` version is accessible, and configure
the path to your configured Elasticsearch singleton client in your django settings:
``DJANGO_ELASTIC_MIGRATIONS_ES_CLIENT = "tests.es_config.ES_CLIENT"``.
There should only be one ``ES_CLIENT`` instantiated in your application.
#. Add ``django_elastic_migrations`` to ``INSTALLED_APPS`` in your Django
settings file
#. Add the following information to your Django settings file:
::
DJANGO_ELASTIC_MIGRATIONS_ES_CLIENT = "path.to.your.singleton.ES_CLIENT"
# optional, any unique number for your releases to associate with indexes
DJANGO_ELASTIC_MIGRATIONS_GET_CODEBASE_ID = subprocess.check_output(['git', 'describe', "--tags"]).strip()
# optional, can be used to have multiple servers share the same
# elasticsearch instance without conflicting
DJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIX = "qa1_"
#. Create the ``django_elastic_migrations`` tables by running ``./manage.py migrate``
#. Create an ``DEMIndex``:
::
from django_elastic_migrations.indexes import DEMIndex, DEMDocType
from .models import Movie
from elasticsearch_dsl import Text
MoviesIndex = DEMIndex('movies')
@MoviesIndex.doc_type
class MovieSearchDoc(DEMDocType):
text = TEXT_COMPLEX_ENGLISH_NGRAM_METAPHONE
@classmethod
def get_queryset(self, last_updated_datetime=None):
"""
return a queryset or a sliceable list of items to pass to
get_reindex_iterator
"""
qs = Movie.objects.all()
if last_updated_datetime:
qs.filter(last_modified__gt=last_updated_datetime)
return qs
@classmethod
def get_reindex_iterator(self, queryset):
return [
MovieSearchDoc(
text="a little sample text").to_dict(
include_meta=True) for g in queryset]
#. Add your new index to DJANGO_ELASTIC_MIGRATIONS_INDEXES in settings/common.py
#. Run ``./manage.py es_list`` to see the index as available:
::
./manage.py es_list
Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
| Index Base Name | Index Version Name | Created | Active | Docs | Tag |
+======================+=====================================+=========+========+=======+===========+
| movies | | 0 | 0 | 0 | Current |
| | | | | | (not |
| | | | | | created) |
+----------------------+-------------------------------------+---------+--------+-------+-----------+
Reminder: an index version name looks like 'my_index-4', and its base index name
looks like 'my_index'. Most Django Elastic Migrations management commands
take the base name (in which case the activated version is used)
or the specific index version name.
#. Create the ``movies`` index in elasticsearch with ``./manage.py es_create movies``:
::
$> ./manage.py es_create movies
The doc type for index 'movies' changed; created a new index version
'movies-1' in elasticsearch.
$> ./manage.py es_list
Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
| Index Base Name | Index Version Name | Created | Active | Docs | Tag |
+======================+=====================================+=========+========+=======+===========+
| movies | movies-1 | 1 | 0 | 0 | 07.11.005 |
| | | | | | -93-gd101 |
| | | | | | a1f |
+----------------------+-------------------------------------+---------+--------+-------+-----------+
Reminder: an index version name looks like 'my_index-4', and its base index name
looks like 'my_index'. Most Django Elastic Migrations management commands
take the base name (in which case the activated version is used)
or the specific index version name.
#. Activate the ``movies-1`` index version, so all updates and reads go to it.
::
./manage.py es_activate movies
For index 'movies', activating 'movies-1' because you said so.
#. Assuming you have implemented ``get_reindex_iterator``, you can call
``./manage.py es_update`` to update the index.
::
$> ./manage.py es_update movies
Handling update of index 'movies' using its active index version 'movies-1'
Checking the last time update was called:
- index version: movies-1
- update date: never
Getting Reindex Iterator...
Completed with indexing movies-1
$> ./manage.py es_list
Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
| Index Base Name | Index Version Name | Created | Active | Docs | Tag |
+======================+=====================================+=========+========+=======+===========+
| movies | movies-1 | 1 | 1 | 3 | 07.11.005 |
| | | | | | -93-gd101 |
| | | | | | a1f |
+----------------------+-------------------------------------+---------+--------+-------+-----------+
Deployment
^^^^^^^^^^
* Creating and updating a new index schema can happen before you deploy.
For example, if your app servers are running with the ``movies-1`` index activated, and you
have a new version of the schema you'd like to pre-index, then log into another
server and run ``./manage.py es_create movies`` followed by
``./manage.py es_update movies --newer``. This will update documents in all ``movies``
indexes that are newer than the active one.
* After deploying, you can run
``./manage.py es_activate movies`` to activate the latest version. Be sure to cycle your
gunicorn workers to ensure the change is caught by your app servers.
* During deployment, if ``get_reindex_iterator`` is implemented in such a way as to respond
to the datetime of the last reindex date, then you can call
``./manage.py es_update movies --resume``, and it will index *only those documents that have
changed since the last reindexing*. This way you can do most of the indexing ahead of time,
and only reindex a portion at the time of the deployment.
Django Testing
^^^^^^^^^^^^^^
#. (optional) update ``DJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIX`` in
your Django settings. The default test prefix is ``test_``. Every
test will create its own indexes.
::
if 'test' in sys.argv:
DJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIX = 'test_'
#. Override TestCase - ``test_utilities.py``
.. code-block::
from django_elastic_migrations import DEMIndexManager
class MyTestCase(TestCase):
def _pre_setup(self):
DEMIndexManager.test_pre_setup()
super(MyTestCase, self)._pre_setup()
def _post_teardown(self):
DEMIndexManager.test_post_teardown()
super(MyTestCase, self)._post_teardown()
Excluding from Django's ``dumpdata`` command
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When calling `django's dumpdata command <https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata>`_\,
you likely will want to exclude the database tables used in this app:
::
from django.core.management import call_command
params = {
'database': 'default',
'exclude': [
# we don't want to include django_elastic_migrations in dumpdata,
# because it's environment specific
'django_elastic_migrations.index',
'django_elastic_migrations.indexversion',
'django_elastic_migrations.indexaction'
],
'indent': 3,
'output': 'path/to/my/file.json'
}
call_command('dumpdata', **params)
An example of this is included with the
`moviegen management command`_.
.. _moviegen management command: https://github.com/HBS-HBX/django-elastic-migrations/blob/master/tests/management/commands/moviegen.py
Tuning Bulk Indexing Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, ``/.manage.py es_update`` will divide the result of
``DEMDocType.get_queryset()`` into batches of size ``DocType.BATCH_SIZE``.
Override this number to change the batch size.
There are many configurable paramters to Elasticsearch's `bulk updater <https://elasticsearch-py.readthedocs.io/en/master/helpers.html?highlight=bulk#elasticsearch.helpers.streaming_bulk>`_.
To provide a custom value, override ``DEMDocType.get_bulk_indexing_kwargs()``
and return the kwargs you would like to customize.
Development
-----------
This project uses ``make`` to manage the build process. Type ``make help``
to see the available ``make`` targets.
Elasticsearch Docker Compose
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``docker-compose -f local.yml up``
`See docs/docker_setup for more info <./docs/docker_setup.rst>`_
Requirements
^^^^^^^^^^^^
This project uses `pip-tools`_. The ``requirements.txt`` files are generated
and pinned to latest versions with ``make upgrade``:
* run ``make requirements`` to run the pip install.
* run ``make upgrade`` to upgrade the dependencies of the requirements to the latest
versions. This process also excludes ``django`` and ``elasticsearch-dsl``
from the ``requirements/test.txt`` so they can be injected with different
versions by tox during matrix testing.
.. _pip-tools: https://github.com/jazzband/pip-tools
Populating Local ``tests_movies`` Database Table With Data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It may be helpful for you to populate a local database with Movies test
data to experiment with using ``django-elastic-migrations``. First,
migrate the database:
``./manage.py migrate --run-syncdb --settings=test_settings``
Next, load the basic fixtures:
``./manage.py loaddata tests/100films.json``
You may wish to add more movies to the database. A management command
has been created for this purpose. Get a `Free OMDB API key here <https://www.omdbapi.com/apikey.aspx>`_\ ,
then run a query like this (replace ``MYAPIKEY`` with yours):
.. code-block::
$> ./manage.py moviegen --title="Inception" --api-key="MYAPIKEY"
{'actors': 'Leonardo DiCaprio, Joseph Gordon-Levitt, Ellen Page, Tom Hardy',
'awards': 'Won 4 Oscars. Another 152 wins & 204 nominations.',
'boxoffice': '$292,568,851',
'country': 'USA, UK',
'director': 'Christopher Nolan',
'dvd': '07 Dec 2010',
'genre': 'Action, Adventure, Sci-Fi',
'imdbid': 'tt1375666',
'imdbrating': '8.8',
'imdbvotes': '1,721,888',
'language': 'English, Japanese, French',
'metascore': '74',
'plot': 'A thief, who steals corporate secrets through the use of '
'dream-sharing technology, is given the inverse task of planting an '
'idea into the mind of a CEO.',
'poster': 'https://m.media-amazon.com/images/M/MV5BMjAxMzY3NjcxNF5BMl5BanBnXkFtZTcwNTI5OTM0Mw@@._V1_SX300.jpg',
'production': 'Warner Bros. Pictures',
'rated': 'PG-13',
'ratings': [{'Source': 'Internet Movie Database', 'Value': '8.8/10'},
{'Source': 'Rotten Tomatoes', 'Value': '86%'},
{'Source': 'Metacritic', 'Value': '74/100'}],
'released': '16 Jul 2010',
'response': 'True',
'runtime': 148,
'title': 'Inception',
'type': 'movie',
'website': 'http://inceptionmovie.warnerbros.com/',
'writer': 'Christopher Nolan',
'year': '2010'}
To save the movie to the database, use the ``--save`` flag. Also useful is
the ``--noprint`` option, to suppress json. Also, if you add
``OMDB_API_KEY=MYAPIKEY`` to your environment variables, you don't have
to specify it each time:
.. code-block::
$ ./manage.py moviegen --title "Closer" --noprint --save
Saved 1 new movie(s) to the database: Closer
Now that it's been saved to the database, you may want to create a fixture,
so you can get back to this state in the future.
.. code-block::
$ ./manage.py moviegen --makefixture=tests/myfixture.json
dumping fixture data to tests/myfixture.json ...
[...........................................................................]
Later, you can restore this database with the regular ``loaddata`` command:
.. code-block::
$ ./manage.py loaddata tests/myfixture.json
Installed 101 object(s) from 1 fixture(s)
There are already 100 films available using ``loaddata`` as follows:
.. code-block::
$ ./manage.py loaddata tests/100films.json
Running Tests Locally
^^^^^^^^^^^^^^^^^^^^^
Run ``make test``. To run all tests and quality checks locally,
run ``make test-all``.
To just run linting, ``make quality``. Please note that if any of the
linters return a nonzero code, it will give an ``InvocationError`` error
at the end. See `tox's documentation for InvocationError`_ for more information.
We use ``edx_lint`` to compile ``pylintrc``. To update the rules,
change ``pylintrc_tweaks`` and run ``make pylintrc``.
.. _tox's documentation for InvocationError: https://tox.readthedocs.io/en/latest/example/general.html#understanding-invocationerror-exit-codes
Cutting a New Version
^^^^^^^^^^^^^^^^^^^^^
* optional: run ``make update`` to update dependencies
* bump version in `django_elastic_migrations/__init__.py <https://github.com/HBS-HBX/django-elastic-migrations/blob/master/django_elastic_migrations/__init__.py#L13>`_.
* update `CHANGELOG.rst <https://github.com/HBS-HBX/django-elastic-migrations/blob/master/CHANGELOG.rst>`_.
* ``make clean``
* ``python3 setup.py sdist bdist_wheel``
* ``twine check dist/django-elastic-migrations-*.tar.gz`` to see if there are any syntax mistakes before tagging
* submit PR bumping the version
* ensure test matrix is passing on travis and merge PR
* pull changes to master
* ``make clean``
* ``python3 setup.py sdist bdist_wheel``
* ``twine check dist/django-elastic-migrations-*.tar.gz`` to see if there are any syntax mistakes before tagging
* ``twine upload -r testpypi dist/django-elastic-migrations-*.tar.gz``
* `Check it at https://test.pypi.org/project/django-elastic-migrations/ <https://test.pypi.org/project/django-elastic-migrations/>`_
* ``python3 setup.py tag`` to tag the new version
* ``twine upload -r pypi dist/django-elastic-migrations-*.tar.gz``
* `Update new release at https://github.com/HBS-HBX/django-elastic-migrations/releases <https://github.com/HBS-HBX/django-elastic-migrations/releases/>`_
Changelog
---------
0.8.2 (2018-11-20)
^^^^^^^^^^^^^^^^^^
* fix `#59 twine check error in 0.8.1 <https://github.com/HBS-HBX/django-elastic-migrations/issues/59>`_
0.8.1 (2018-11-19)
^^^^^^^^^^^^^^^^^^
* fix `#50 add test coverage for es_list <https://github.com/HBS-HBX/django-elastic-migrations/issues/50>`_
* fix `#58 ignore indexes with a dot in name in es_list --es-only and es_dangerous_reset <https://github.com/HBS-HBX/django-elastic-migrations/issues/58>`_
0.8.0 (2018-11-13)
^^^^^^^^^^^^^^^^^^
* fix `#6 support Django 2 <https://github.com/HBS-HBX/django-elastic-migrations/issues/6>`_
* fix `#43 remove es_deactivate <https://github.com/HBS-HBX/django-elastic-migrations/issues/43>`_
* fix `#44 add django 1.10 and 1.11 to test matrix <https://github.com/HBS-HBX/django-elastic-migrations/issues/44>`_
* fix `#45 remove support for python 2 <https://github.com/HBS-HBX/django-elastic-migrations/issues/45>`_
* In practice, Python 2 may work, but it is removed from the test matrix and won't be updated
0.7.8 (2018-11-13)
^^^^^^^^^^^^^^^^^^
* fix `#7 Convert Readme to rst for pypi <https://github.com/HBS-HBX/django-elastic-migrations/issues/7>`_
* first release on PyPI
* update project dependencies
0.7.7 (2018-09-17)
^^^^^^^^^^^^^^^^^^
* fix `#41 stack trace when indexing in py3 <https://github.com/HBS-HBX/django-elastic-migrations/issues/41>`_
0.7.6 (2018-09-11)
^^^^^^^^^^^^^^^^^^
* fix `#36 es_update --start flag broken <https://github.com/HBS-HBX/django-elastic-migrations/issues/39>`_
0.7.5 (2018-08-20)
^^^^^^^^^^^^^^^^^^
* fix `#35 open multiprocessing log in context handler <https://github.com/HBS-HBX/django-elastic-migrations/issues/35>`_
0.7.4 (2018-08-15)
^^^^^^^^^^^^^^^^^^
* fix `#33 error when nothing to resume using --resume <https://github.com/HBS-HBX/django-elastic-migrations/issues/33>`_
0.7.3 (2018-08-14)
^^^^^^^^^^^^^^^^^^
* fix #31 es_update movies --newer --workers does not store worker information
0.7.2 (2018-08-13)
^^^^^^^^^^^^^^^^^^
* fix #21 wrong batch update total using multiprocessing in 0.7.1
* fix #23 KeyError _index_version_name in es_update --newer
* address #25 use pks for queryset inside workers #29
0.7.1 (2018-08-07)
^^^^^^^^^^^^^^^^^^
* fixed gh #8 es_dangerous_reset --es-only to sync database to ES
* fixed gh #17 make es_dangerous_reset remove dem models
* improved test coverage
* added tests for ``es_create --es-only``
* added ``IndexVersion.hard_delete()`` (not called by default)
* added ``hard_delete`` flag to ``DropIndexAction``
* added ``hard_delete`` flag to ``DEMIndexManager.test_post_teardown()``
* updated ``__str__()`` of ``IndexAction`` to be more descriptive
0.7.0 (2018-08-06)
^^^^^^^^^^^^^^^^^^
* fixed gh #5: "add python 3 support and tests"
0.6.1 (2018-08-03)
^^^^^^^^^^^^^^^^^^
* fixed gh #9: "using elasticsearch-dsl 6.1, TypeError in DEMIndex.save"
0.6.0 (2018-08-01)
^^^^^^^^^^^^^^^^^^
* Added test structure for py2 - GH #2
* Renamed default log handler from ``django-elastic-migrations`` to ``django_elastic_migrations``
0.5.3 (2018-07-23)
^^^^^^^^^^^^^^^^^^
* First basic release
نحوه نصب
نصب پکیج whl django-elastic-migrations-0.8.2:
pip install django-elastic-migrations-0.8.2.whl
نصب پکیج tar.gz django-elastic-migrations-0.8.2:
pip install django-elastic-migrations-0.8.2.tar.gz