elasticsearch-fabric
====================
This package provides a unified command line interface to Elasticsearch
in Fabric.
Installation
------------
The current release, published on PyPI, can be installed using the
following command:
.. code:: sh
$ pip install elasticsearch-fabric
Configuration
-------------
Tasks
~~~~~
If you plan to use the built-in tasks, include the module in your
fabfile module (e.g. fabfile.py). Most likely you might want to assign
an alias for the task namespace:
.. code:: python
from esfabric import tasks as es
Environment
~~~~~~~~~~~
- ``elasticsearch_clients``: Customize elasticsearch client
configurations.
- ``elasticsearch_alias``: Default Elasticsearch client alias in
elasticsearch\_clients. default "default"
- ``elasticsearch_dest_alias``: Reindex dest Elasticsearch client alias
in elasticsearch\_clients. default elasticsearch\_alias
Examples
^^^^^^^^
.. code:: python
# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from esfabric import tasks as es
env.elasticsearch_clients = {
"default": Elasticsearch(**{
"host": "localhost",
"port": 9200,
"send_get_body_as": "POST"
}),
"example": Elasticsearch(**{
"host": "search.example.org",
"port": 443,
"send_get_body_as": "POST",
"use_ssl": True,
"verify_certs": True
})
}
Elasticsearch with Shield
~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure the client to use basic authentication:
.. code:: python
# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from esfabric import tasks as es
env.elasticsearch_clients = {
"default": Elasticsearch(**{
"host": "localhost",
"port": 9200,
"send_get_body_as": "POST",
"http_auth": ('user', 'secret')
})
}
Running on AWS with IAM
~~~~~~~~~~~~~~~~~~~~~~~
.. code:: python
# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from elasticsearch import RequestsHttpConnection
from requests_aws4auth import AWS4Auth
from esfabric import tasks as es
awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')
env.elasticsearch_clients = {
"default": Elasticsearch(**{
"host": "YOURHOST.us-east-1.es.amazonaws.com",
"port": 443,
"send_get_body_as": "POST",
"http_auth": awsauth,
"use_ssl": True,
"verify_certs": True,
"connection_class": RequestsHttpConnection
})
}
Checking the setup
------------------
For checking if everything is set up properly, you can run the included
task *info*. For example, running
.. code:: sh
$ fab es.info
you can show a similar result:
.. code:: sh
{
"cluster_name": "elasticsearch",
"tagline": "You Know, for Search",
"version": {
"lucene_version": "5.5.0",
"build_hash": "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"number": "2.3.3",
"build_timestamp": "2016-05-17T15:40:04Z",
"build_snapshot": false
},
"name": "Ares"
}
Done.
Example use
-----------
You can do this for example with the following command:
.. code:: sh
# -------------------------------------------------------------
# by Default connect to localhost:9200
$ fab es.info
{
"cluster_name": "elasticsearch",
"tagline": "You Know, for Search",
"version": {
"lucene_version": "5.5.0",
"build_hash": "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"number": "2.3.3",
"build_timestamp": "2016-05-17T15:40:04Z",
"build_snapshot": false
},
"name": "Ares"
}
Done.
# -------------------------------------------------------------
# index a document
#
# $ cat doc.json
# {
# "title": "Hello Elasticsearch",
# "description": "elasticsearch fabric test"
# }
$ cat doc.json | fab es.index:index=blog,doc_type=posts,id=1
{
"_type": "posts",
"created": true,
"_shards": {
"successful": 1,
"failed": 0,
"total": 2
},
"_version": 1,
"_index": "blog",
"_id": "1"
}
Done.
# -------------------------------------------------------------
# get the document.
$ fab es.get:index=blog,doc_type=posts,id=1
{
"_type": "posts",
"_source": {
"description": "elasticsearch fabric test",
"title": "Hello Elasticsearch"
},
"_index": "blog",
"_version": 1,
"found": true,
"_id": "1"
}
Done.
# -------------------------------------------------------------
# simple query search.
$ fab es.search:q="title:hello"
{
"hits": {
"hits": [
{
"_score": 0.19178301,
"_type": "posts",
"_id": "1",
"_source": {
"description": "elasticsearch fabric test",
"title": "Hello Elasticsearch"
},
"_index": "blog"
}
],
"total": 1,
"max_score": 0.19178301
},
"_shards": {
"successful": 26,
"failed": 0,
"total": 26
},
"took": 4,
"timed_out": false
}
Done.
# -------------------------------------------------------------
# request body search.
#
# $ cat query.json
# {
# "query": {
# "match": {
# "title": "hello"
# }
# }
# }
$ cat query.json | fab es.search
{
"hits": {
"hits": [
{
"_score": 0.19178301,
"_type": "posts",
"_id": "1",
"_source": {
"description": "elasticsearch fabric test",
"title": "Hello Elasticsearch"
},
"_index": "blog"
}
],
"total": 1,
"max_score": 0.19178301
},
"_shards": {
"successful": 26,
"failed": 0,
"total": 26
},
"took": 8,
"timed_out": false
}
Done.
# -------------------------------------------------------------
# change number of replicas
#
# cat indices
$ fab es.cat.indices
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open blog 5 1 1 0 3.9kb 3.9kb
# change number of replicas
$ fab es.helpers.change_replicas:index=blog,number_of_replicas=0
{
"acknowledged": true
}
# cat indices
$ fab es.cat.indices:v=1
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open blog 5 0 1 0 3.9kb 3.9kb
# -------------------------------------------------------------
# reindex blog to blog2
$ fab es.helpers.reindex:source_index=blog,dest_index=blog2
{
"dest": {
"index": "blog2",
"host": "http://localhost:9200"
},
"source": {
"index": "blog",
"host": "http://localhost:9200"
},
"errors": 0,
"success": 1
}
Done.
# cat indices
$ fab es.cat.indices:v=1
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open blog2 5 1 1 0 3.7kb 3.7kb
green open blog 5 0 1 0 3.9kb 3.9kb
Client selection
----------------
.. code:: python
# fabfile.py
from esfabric import tasks as es
from esfabric.tasks import client_selection
env.elasticsearch_clients = {
"client1": Elasticsearch(**{
...
}),
"client2": Elasticsearch(**{
...
})
}
.. code:: sh
$ fab c:client2 es.info
``c`` is client\_selection alias
Logging
-------
you can enable the elasticsearch.trace logger and have it log a shell
transcript of your session using curl:
.. code:: python
# fabfile.py
import logging
tracer = logging.getLogger('elasticsearch.trace')
tracer.setLevel(logging.DEBUG)
tracer.addHandler(logging.FileHandler('/tmp/elasticsearch-py.sh'))
Custom Task
-----------
.. code:: python
from esfabric import tasks as es
from fabric.api import execute, task
@task
def change_replicas(number_of_replicas=1):
execute(es.cat.indices, v=1)
execute(es.helpers.change_replicas, number_of_replicas=number_of_replicas)
execute(es.cat.indices, v=1)
run task:
.. code:: sh
$ fab change_replicas:10
Available commands
------------------
The following command will show a list of avaliable commands.
.. code:: sh
$ fab -l
- Available commands
- es.bulk
- es.c
- es.clear\_scroll
- es.client\_selection
- es.count
- es.count\_percolate
- es.create
- es.delete
- es.delete\_by\_query
- es.delete\_script
- es.delete\_template
- es.exists
- es.explain
- es.field\_stats
- es.get
- es.get\_script
- es.get\_source
- es.get\_template
- es.index
- es.info
- es.mget
- es.mpercolate
- es.msearch
- es.msearch\_template
- es.mtermvectors
- es.percolate
- es.ping
- es.put\_script
- es.put\_template
- es.reindex
- es.reindex\_rethrottle
- es.render\_search\_template
- es.scroll
- es.search
- es.search\_shards
- es.search\_template
- es.suggest
- es.termvectors
- es.update
- es.update\_by\_query
- es.cat.aliases
- es.cat.allocation
- es.cat.count
- es.cat.fielddata
- es.cat.health
- es.cat.indices
- es.cat.master
- es.cat.nodeattrs
- es.cat.nodes
- es.cat.pending\_tasks
- es.cat.plugins
- es.cat.recovery
- es.cat.repositories
- es.cat.segments
- es.cat.shards
- es.cat.snapshots
- es.cat.thread\_pool
- es.cluster.allocation\_explain
- es.cluster.get\_settings
- es.cluster.health
- es.cluster.pending\_tasks
- es.cluster.put\_settings
- es.cluster.reroute
- es.cluster.state
- es.cluster.stats
- es.helpers.bulk
- es.helpers.change\_replicas
- es.helpers.reindex
- es.helpers.scan
- es.indices.analyze
- es.indices.clear\_cache
- es.indices.close
- es.indices.create
- es.indices.delete
- es.indices.delete\_alias
- es.indices.delete\_template
- es.indices.exists
- es.indices.exists\_alias
- es.indices.exists\_template
- es.indices.exists\_type
- es.indices.flush
- es.indices.flush\_synced
- es.indices.forcemerge
- es.indices.get
- es.indices.get\_alias
- es.indices.get\_field\_mapping
- es.indices.get\_mapping
- es.indices.get\_settings
- es.indices.get\_template
- es.indices.get\_upgrade
- es.indices.open
- es.indices.put\_alias
- es.indices.put\_mapping
- es.indices.put\_settings
- es.indices.put\_template
- es.indices.recovery
- es.indices.refresh
- es.indices.rollover
- es.indices.segments
- es.indices.shard\_stores
- es.indices.shrink
- es.indices.stats
- es.indices.update\_aliases
- es.indices.upgrade
- es.indices.validate\_query
- es.nodes.hot\_threads
- es.nodes.info
- es.nodes.stats
- es.snapshot.create
- es.snapshot.create\_repository
- es.snapshot.delete
- es.snapshot.delete\_repository
- es.snapshot.get
- es.snapshot.get\_repository
- es.snapshot.restore
- es.snapshot.status
- es.snapshot.verify\_repository