.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================
===============================
Acumos Python Client User Guide
===============================
|Build Status|
``acumos`` is a client library that allows modelers to push their Python models
to the `Acumos platform <https://www.acumos.org/>`__.
Installation
============
You will need a Python 3.6 or 3.7 environment in order to install ``acumos``.
Python 3.8 and later can also be used starting with version 0.9.5, some AI
framework like Tensor Flow was not supported in Python 3.8 and later.
You can use `Anaconda <https://www.anaconda.com/download/>`__
(preferred) or `pyenv <https://github.com/pyenv/pyenv>`__ to install and
manage Python environments.
If you’re new to Python and need an IDE to start developing, we
recommend using `Spyder <https://github.com/spyder-ide/spyder>`__ which
can easily be installed with Anaconda.
The ``acumos`` package can be installed with pip:
.. code:: bash
pip install acumos
Protocol Buffers
----------------
The ``acumos`` package uses protocol buffers and **assumes you have
the protobuf compiler** ``protoc`` **installed**. Please visit the `protobuf
repository <https://github.com/google/protobuf/releases/tag/v3.4.0>`__
and install the appropriate ``protoc`` for your operating system.
Installation is as easy as downloading a binary release and adding it to
your system ``$PATH``. This is a temporary requirement that will be
removed in a future version of ``acumos``.
**Anaconda Users**: You can easily install ``protoc`` from `an Anaconda
package <https://anaconda.org/anaconda/libprotobuf>`__ via:
.. code:: bash
conda install -c anaconda libprotobuf
.. |Build Status| image:: https://jenkins.acumos.org/buildStatus/icon?job=acumos-python-client-tox-verify-master
:target: https://jenkins.acumos.org/job/acumos-python-client-tox-verify-master/
.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================
=============================
Acumos Python Client Tutorial
=============================
This tutorial provides a brief overview of ``acumos`` for creating
Acumos models. The tutorial is meant to be followed linearly, and some
code snippets depend on earlier imports and objects. Full examples are
available in the ``examples/`` directory of the `Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>`__.
#. `Importing Acumos`_
#. `Creating A Session`_
#. `A Simple Model`_
#. `Exporting Models`_
#. `Defining Types`_
#. `Using DataFrames with scikit-learn`_
#. `Declaring Requirements`_
#. `Declaring Options`_
#. `Keras and TensorFlow`_
#. `Testing Models`_
#. `More Examples`_
Importing Acumos
================
First import the modeling and session packages:
.. code:: python
from acumos.modeling import Model, List, Dict, create_namedtuple, create_dataframe
from acumos.session import AcumosSession
Creating A Session
==================
An ``AcumosSession`` allows you to export your models to Acumos. You can
either dump a model to disk locally, so that you can upload it via the
Acumos website, or push the model to Acumos directly.
If you’d like to push directly to Acumos, create a session with the ``push_api`` argument:
.. code:: python
session = AcumosSession(push_api="https://my.acumos.instance.com/push")
See the onboarding page of your Acumos instance website to find the correct
``push_api`` URL to use.
If you’re only interested in dumping a model to disk, arguments aren’t needed:
.. code:: python
session = AcumosSession()
A Simple Model
==============
Any Python function can be used to define an Acumos model using `Python
type hints <https://docs.python.org/3/library/typing.html>`__.
Let’s first create a simple model that adds two integers together.
Acumos needs to know what the inputs and outputs of your functions are.
We can use the Python type annotation syntax to specify the function
signature.
Below we define a function ``add_numbers`` with ``int`` type parameters
``x`` and ``y``, and an ``int`` return type. We then build an Acumos
model with an ``add`` method.
**Note:** Function
`docstrings <https://www.python.org/dev/peps/pep-0257/>`__ are included
with your model and used for documentation, so be sure to include one!
.. code:: python
def add_numbers(x: int, y: int) -> int:
'''Returns the sum of x and y'''
return x + y
model = Model(add=add_numbers)
Exporting Models
================
We can now export our model using the ``AcumosSession`` object created
earlier. The ``push`` and ``dump_zip`` APIs are shown below. The ``dump_zip`` method will
save the model to disk so that it can be onboarded via the Acumos website. The
``push`` method pushes the model directly to Acumos.
.. code:: python
session.push(model, 'my-model')
session.dump_zip(model, 'my-model', '~/my-model.zip') # creates ~/my-model.zip
For more information on how to onboard a dumped model via the Acumos website,
see the `web onboarding guide <https://docs.acumos.org/en/latest/submodules/portal-marketplace/docs/user-guides/portal-user/portal/portal-onboarding-intro.html#on-boarding-by-web>`__.
**Note:** Pushing a model to Acumos will prompt you for an onboarding token if
you have not previously provided one. The interactive prompt can be avoided by
exporting the ``ACUMOS_TOKEN`` environment variable, which corresponds to an
authentication token that can be found in your account settings on the Acumos
website.
Defining Types
==============
In this example, we make a model that can read binary images and output
some metadata about them. This model makes use of a custom type
``ImageShape``.
We first create a ``NamedTuple`` type called ``ImageShape``, which is
like an ordinary ``tuple`` but with field accessors. We can then use
``ImageShape`` as the return type of ``get_shape``. Note how
``ImageShape`` can be instantiated as a new object.
.. code:: python
import io
import PIL
ImageShape = create_namedtuple('ImageShape', [('width', int), ('height', int)])
def get_format(data: bytes) -> str:
'''Returns the format of an image'''
buffer = io.BytesIO(data)
img = PIL.Image.open(buffer)
return img.format
def get_shape(data: bytes) -> ImageShape:
'''Returns the width and height of an image'''
buffer = io.BytesIO(data)
img = PIL.Image.open(buffer)
shape = ImageShape(width=img.width, height=img.height)
return shape
model = Model(get_format=get_format, get_shape=get_shape)
**Note:** Starting in Python 3.6, you can alternatively use this simpler
syntax:
.. code:: python
from acumos.modeling import NamedTuple
class ImageShape(NamedTuple):
'''Type representing the shape of an image'''
width: int
height: int
Defining Unstructured Types
===========================
The `create_namedtuple` function allows us to create types with structure,
however sometimes it's useful to work with unstructured data, such as plain
text, dictionaries or byte strings. The `new_type` function allows for just
that.
For example, here's a model that takes in unstructured text, and returns the
number of words in the text:
.. code:: python
from acumos.modeling import new_type
Text = new_type(str, 'Text')
def count(text: Text) -> int:
'''Counts the number of words in the text'''
return len(text.split(' '))
def create_text(x: int, y: int) -> Text:
'''Returns a string containing ints from x to y'''
return " ".join(map(str, range(x, y+1)))
def reverse_text(text: Text) -> Text:
'''Returns an empty image buffer from dimensions'''
return text[::-1]
By using the `new_type` function, you inform `acumos` that `Text` is
unstructured, and therefore `acumos` will not create any structured types or
messages for the `count` function.
You can use the `new_type` function to create dictionaries or byte string
type unstructured data as shown below.
.. code:: python
from acumos.modeling import new_type
Dict = new_type(dict, 'Dict')
Image = new_type(byte, 'Image')
Using DataFrames with scikit-learn
==================================
In this example, we train a ``RandomForestClassifier`` using
``scikit-learn`` and use it to create an Acumos model.
When making machine learning models, it’s common to use a dataframe data
structure to represent data. To make things easier, ``acumos`` can
create ``NamedTuple`` types directly from ``pandas.DataFrame`` objects.
``NamedTuple`` types created from ``pandas.DataFrame`` objects store
columns as named attributes and preserve column order. Because
``NamedTuple`` types are like ordinary ``tuple`` types, the resulting
object can be iterated over. Thus, iterating over a ``NamedTuple``
dataframe object is the same as iterating over the columns of a
``pandas.DataFrame``. As a consequence, note how ``np.column_stack`` can
be used to create a ``numpy.ndarray`` from the input ``df``.
Finally, the model returns a ``numpy.ndarray`` of ``int`` corresponding
to predicted iris classes. The ``classify_iris`` function represents
this as ``List[int]`` in the signature return.
.. code:: python
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
iris = load_iris()
X = iris.data
y = iris.target
clf = RandomForestClassifier(random_state=0)
clf.fit(X, y)
# here, an appropriate NamedTuple type is inferred from a pandas DataFrame
X_df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
IrisDataFrame = create_dataframe('IrisDataFrame', X_df)
# ==================================================================================
# # or equivalently:
#
# IrisDataFrame = create_namedtuple('IrisDataFrame', [('sepal_length', List[float]),
# ('sepal_width', List[float]),
# ('petal_length', List[float]),
# ('petal_width', List[float])])
# ==================================================================================
def classify_iris(df: IrisDataFrame) -> List[int]:
'''Returns an array of iris classifications'''
X = np.column_stack(df)
return clf.predict(X)
model = Model(classify=classify_iris)
Check out the ``sklearn`` examples in the examples directory for full
runnable scripts.
Declaring Requirements
======================
If your model depends on another Python script or package that you wrote, you can
declare the dependency via the ``acumos.metadata.Requirements`` class:
.. code:: python
from acumos.metadata import Requirements
Note that only pure Python is supported at this time.
Custom Scripts
--------------
Custom scripts can be included by giving ``Requirements`` a sequence of paths
to Python scripts, or directories containing Python scripts. For example, if the
model defined in ``model.py`` depended on ``helper1.py``:
::
model_workspace/
├── model.py
├── helper1.py
└── helper2.py
this dependency could be declared like so:
.. code:: python
from helper1 import do_thing
def transform(x: int) -> int:
'''Does the thing'''
return do_thing(x)
model = Model(transform=transform)
reqs = Requirements(scripts=['./helper1.py'])
# using the AcumosSession created earlier:
session.push(model, 'my-model', reqs)
session.dump(model, 'my-model', '~/', reqs) # creates ~/my-model
Alternatively, all Python scripts within ``model_workspace/`` could be included
using:
.. code:: python
reqs = Requirements(scripts=['.'])
Custom Packages
---------------
Custom packages can be included by giving ``Requirements`` a sequence of paths to
Python packages, i.e. directories with an ``__init__.py`` file. Assuming that the
package ``~/repos/my_pkg`` contains:
::
my_pkg/
├── __init__.py
├── bar.py
└── foo.py
then you can bundle ``my_pkg`` with your model like so:
.. code:: python
from my_pkg.bar import do_thing
def transform(x: int) -> int:
'''Does the thing'''
return do_thing(x)
model = Model(transform=transform)
reqs = Requirements(packages=['~/repos/my_pkg'])
# using the AcumosSession created earlier:
session.push(model, 'my-model', reqs)
session.dump(model, 'my-model', '~/', reqs) # creates ~/my-model
Requirement Mapping
-------------------
Python packaging and `PyPI <https://pypi.org/>`__ aren’t
perfect, and sometimes the name of the Python package you import in your
code is different than the package name used to install it. One example
of this is the ``PIL`` package, which is commonly installed using `a fork
called pillow <https://pillow.readthedocs.io>`_ (i.e.
``pip install pillow`` will provide the ``PIL`` package).
To address this inconsistency, the ``Requirements``
class allows you to map Python package names to PyPI package names. When
your model is analyzed for dependencies by ``acumos``, this mapping is
used to ensure the correct PyPI packages will be used.
In the example below, the ``req_map`` parameter is used to declare a
requirements mapping from the ``PIL`` Python package to the ``pillow``
PyPI package:
.. code:: python
reqs = Requirements(req_map={'PIL': 'pillow'})
Declaring Options
=================
The ``acumos.metadata.Options`` class is a collection of options that users may
wish to specify along with their Acumos model. If an ``Options`` instance is not
provided to ``AcumosSession.push``, then default options are applied. See the
class docstring for more details.
Below, we demonstrate how options can be used to include additional model metadata
and influence the behavior of the Acumos platform. For example, a license can be
included with a model via the ``license`` parameter, either by providing a license
string or a path to a license file. Likewise, we can specify whether or not the Acumos
platform should eagerly build the model microservice via the ``create_microservice``
parameter. Then thanks to the ``deploy`` parameter you can specifiy if you want to deploy
this microservice automatically. (Please refer to the appropriate documentation on Acumos
wiki to use this functionality based on an external jenkins server). if ``create_microservice``=True,
``deploy`` can be True or False. But if ``create_microservice``=False, ``deploy`` must be set to False
if not, ``create_microservice`` will be force to True to create the micro-service and deploy it.
.. code:: python
from acumos.metadata import Options
opts = Options(license="Apache 2.0", # "./path/to/license_file" also works
create_microservice=True, # Build the microservice just after the on-boarding
deploy=True) # Deploy the microservice based on an external Jenkins server
session.push(model, 'my-model', options=opts)
Keras and TensorFlow
====================
Check out the Keras and TensorFlow examples in the ``examples/`` directory of
the `Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>`__.
Testing Models
==============
The ``acumos.modeling.Model`` class wraps your custom functions and
produces corresponding input and output types. This section shows how to
access those types for the purpose of testing. For simplicity, we’ll
create a model using the ``add_numbers`` function again:
.. code:: python
def add_numbers(x: int, y: int) -> int:
'''Returns the sum of x and y'''
return x + y
model = Model(add=add_numbers)
The ``model`` object now has an ``add`` attribute, which acts as a
wrapper around ``add_numbers``. The ``add_numbers`` function can be
invoked like so:
.. code:: python
result = model.add.inner(1, 2)
print(result) # 3
The ``model.add`` object also has a corresponding *wrapped* function
that is generated by ``acumos.modeling.Model``. The wrapped function is
the primary way your model will be used within Acumos.
We can access the ``input_type`` and ``output_type`` attributes to test
that the function works as expected:
.. code:: python
AddIn = model.add.input_type
AddOut = model.add.output_type
add_in = AddIn(1, 2)
print(add_in) # AddIn(x=1, y=2)
add_out = AddOut(3)
print(add_out) # AddOut(value=3)
model.add.wrapped(add_in) == add_out # True
More Examples
=============
Below are some additional function examples. Note how ``numpy`` types
can even be used in type hints, as shown in the ``numpy_sum`` function.
.. code:: python
from collections import Counter
import numpy as np
def list_sum(x: List[int]) -> int:
'''Computes the sum of a sequence of integers'''
return sum(x)
def numpy_sum(x: List[np.int32]) -> np.int32:
'''Uses numpy to compute a vectorized sum over x'''
return np.sum(x)
def count_strings(x: List[str]) -> Dict[str, int]:
'''Returns a count mapping from a sequence of strings'''
return Counter(x)
.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================
==================================
Acumos Python Client Release Notes
==================================
v1.0.1, 27 April 2021
=====================
* use acumos-python-client > 0.8.0 with Acumos clio `ACUMOS-4330 <https://jira.acumos.org/browse/ACUMOS-4330>`_
v1.0.0, 13 April 2021
=====================
* Fix Type issue with python 3.9 `ACUMOS-4323 <https://jira.acumos.org/browse/ACUMOS-4323>`_
v0.9.9, 13 April 2021
=====================
* Take into account "deploy" parameter in acumos python client `ACUMOS-4303 <https://jira.acumos.org/browse/ACUMOS-4303>`_
v0.9.8, 06 November 2020
========================
* Return docker URI & added an optional flag to replace and existing model when dumping `ACUMOS-4298 <https://jira.acumos.org/browse/ACUMOS-4298>`_
* The model bundle can now be dumped directly as a zip file `ACUMOS-4273 <https://jira.acumos.org/browse/ACUMOS-4273>`_
* Allow installation on python 3.9 `ACUMOS-4123 <https://jira.acumos.org/browse/ACUMOS-4123>`_
v0.9.7, 27 August 2020
======================
* Add support of python 3.7 & 3.8 `ACUMOS-4123 <https://jira.acumos.org/browse/ACUMOS-4123>`_
* Display acumos logo on github `ACUMOS-4094 <https://jira.acumos.org/browse/ACUMOS-4094>`_
v0.9.4, 05 April 2020
=====================
* Give image tag URL from python client `ACUMOS-3961 <https://jira.acumos.org/browse/ACUMOS-3961>`_
v0.9.3, 30 Mar 2020
===================
* Modify unstructured type section in pypi `ACUMOS-3956 <https://jira.acumos.org/browse/ACUMOS-3956>`_
* Raise an Error when using asymetric type `ACUMOS-3956 <https://jira.acumos.org/browse/ACUMOS-3956>`_
v0.9.2, 31 Jan 2020
===================
* Remove support for python 3.5 `Gerrit-6275 <https://gerrit.acumos.org/r/c/acumos-python-client/+/6275>`_
v0.9.1
======
* add raw format support `ACUMOS-2712 <https://jira.acumos.org/browse/ACUMOS-2712>`_
* publish content type for long description `Gerrit-5504 <https://gerrit.acumos.org/r/c/acumos-python-client/+/5504>`_
v0.8.0
======
(This is the recommended version for the Clio release)
- Enhancements
- Users may now specify additional options when pushing their Acumos model. See the options section in the tutorial for more information.
- ``acumos`` now supports Keras models built with ``tensorflow.keras``
- Support changes
- ``acumos`` no longer supports Python 3.4
v0.7.2
======
- Bug fixes
- The deprecated authentication API is now considered optional
- A more portable path solution is now used when saving models, to avoid issues with models developed in Windows
v0.7.1
======
- Authentication
- Username and password authentication has been deprecated
- Users are now interactively prompted for an onboarding token, as opposed to a username and password
v0.7.0
======
- Requirements
- Python script dependencies can now be specified using a Requirements object
- Python script dependencies found during the introspection stage are now included with the model
v0.6.5
======
- Bug fixes
- Don't attempt to use an empty auth token (avoids blank strings to be set in environment)
v0.6.4
======
- Bug fixes
- The normalized path of the system base prefix is now used for identifying stdlib packages
v0.6.3
======
- Bug fixes
- Improved dependency inspection when using a virtualenv
- Removed custom packages from model metadata, as it caused image build failures
- Fixed Python 3.5.2 ordering bug in wrapped model usage
v0.6.2
======
- TensorFlow
- Fixed a serialization issue that occurred when using a frozen graph
v0.6.1
======
- Model upload
- The JWT is now cleared immediately after a failed upload
- Additional HTTP information is now included in the error message
v0.6.0
======
- Authentication token
- A new environment variable ``ACUMOS_TOKEN`` can be used to short-circuit
the authentication process
- Extra headers
- ``AcumosSession.push`` now accepts an optional ``extra_headers`` argument,
which will allow users and systems to include additional information when
pushing models to the onboarding server
v0.5.0
======
- Modeling
- Python 3.6 NamedTuple syntax support now tested
- User documentation includes example of new NamedTuple syntax
- Model wrapper
- Model wrapper now has APIs for consuming and producing Python
dicts and JSON strings
- Protobuf and protoc
- An explicit check for protoc is now made, which raises a more
informative error message
- User documentation is more clear about dependence on protoc, and
provides an easier way to install protoc via Anaconda
- Keras
- The active keras backend is now included as a tracked module
- keras_contrib layers are now supported
v0.4.0
======
- Replaced library-specific onboarding functions with “new-style”
models
- Support for arbitrary Python functions using type hints
- Support for custom user-defined types
- Support for TensorFlow models
- Improved dependency introspection
- Improved object serialization mechanisms
.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================
====================================
Acumos Python Client Developer Guide
====================================
Testing
=======
We use a combination of ``tox``, ``pytest``, and ``flake8`` to test
``acumos``. Code which is not PEP8 compliant (aside from E501) will be
considered a failing test. You can use tools like ``autopep8`` to
“clean” your code as follows:
.. code:: bash
$ pip install autopep8
$ cd acumos-python-client
$ autopep8 -r --in-place --ignore E501 acumos/ testing/ examples/
Run tox directly:
.. code:: bash
$ cd acumos-python-client
$ export WORKSPACE=$(pwd) # env var normally provided by Jenkins
$ tox
You can also specify certain tox environments to test:
.. code:: bash
$ tox -e py36 # only test against Python 3.6
$ tox -e flake8 # only lint code
A set of integration test is also available in ``acumos-package/testing/integration_tests``.
To run those, use ``acumos-package/testing/tox-integration.ini`` as tox config (-c flag),
onboarding tests will be ran with python 3.6 to 3.9.
You will need to set your user credentials and platform configuration in ``tox-integration.ini``.
.. code:: bash
$ tox -c acumos-package/testing/integration_tests
Packaging
=========
The RST files in the docs/ directory are used to publish HTML pages to
ReadTheDocs.io and to build the package long description in setup.py.
The symlink from the subdirectory acumos-package to the docs/ directory
is required for the Python packaging tools. Those tools build a source
distribution from files in the package root, the directory acumos-package.
The MANIFEST.in file directs the tools to pull files from directory docs/,
and the symlink makes it possible because the tools only look within the
package root.