# Conductor Core
Use the command-line uploader and downloader or develop your own tools using the Python API.
## Install
**To install the latest version.**
```bash
pip install --upgrade ciocore --target=$HOME/Conductor
```
**To install a specific version, for example 0.1.0.**
```bash
pip install --upgrade --force-reinstall ciocore==0.1.0 --target=$HOME/Conductor
```
Run the conductor command to confirm the package was installed.
```
$HOME/ConductorStaging/bin/conductor --help
```
To use the Python API, you must either set the PYTHONPATH or pip install, possibly to a virtualenv.
Option 1. PYTHONPATH
```bash
export PYTHONPATH=${HOME}/Conductor && python
```
Option 2. Pip install to venv.
```bash
# Create.
python3.9 -m venv ~/venvs/cioapi
# activate
. ~/venvs/cioapi/bin/activate
pip install --upgrade .
```
Example API usage from.
```
from ciocore import api_client
print api_client.request_software_packages()[0]
```
## Contributing
Clone the repo.
```
git clone git@github.com:ConductorTechnologies/ciocore.git
cd ciocore
```
Set up a clean virtual envirionment for development (optional).
```
python -m virtualenv venv
. ./venv/bin/activate
```
Install development dependencies
```
pip install -r requirements_dev.txt
```
Run the automated tests
```
tox
```
Build and install from this directory into to a convenient location for testing.
```
pip install --upgrade . --target=$HOME/ConductorStaging
```
See above for information on the conductor command and the Python API.
## License
[MIT](https://choosealicense.com/licenses/mit)
## Changelog
### Version:5.6.0 -- 6 April 2023
* Add support for CoreWeave Object Storage. [3cdfcea]
### Version:5.5.0 -- 10 Nov 2022
* Support KMS encryption for uploaded files in both local and daemon mode. [a1ba987]
### Version:5.4.2 -- 10 Aug 2022
* Adds CONDUCTOR_AUTH_PORT environment variable. [eeee8c5]
### Version:5.4.1 -- 27 Jul 2022
* Graft old uploader back in. [caedcce]
### Version:5.3.2 -- 25 Jul 2022
* Fix argparse bug https://bugs.python.org/issue16308 [27841c8]
### Version:5.3.1 -- 28 Jun 2022
* Adds a check if any errors occured before marking the upload as complete. [124bbf6]
### Version:5.3.0 -- 03 Jun 2022
* Adds the option to upload a list of explicit paths to the cli. [ba16fe4]
### Version:5.2.2 -- 23 May 2022
* A minor documentation change. [533c7ee]
### Version:5.2.0 -- 26 Apr 2022
* Reinstate frozen requirement on urllib version, since it broke submissions on Mac.
* Adds several doc strings.
* Core data now allows initialization with a list of host products to act as a whitelist. Legacy behavior still works.
* Remove reload code.
* Use manageable fixtures for tests.
* Refactor package tree to make docs clearer.
* Autodoc setup, including script to publish autogen doc to mkdocs.
### Version:5.1.1 -- 05 Apr 2022
* Ensures the base64 strings are represented as strings. [be95b83]
### Version:5.1.0 -- 05 Apr 2022
* Adds registering the uploader and downloader for the user agent. [18a0a7c]
* Adds the user-agent http header to all requests. [6b452c5]
### Version:5.0.1 -- 31 Mar 2022
* In developer mode we add X drive letter if submitting to a windows target. [bc91524]
* Adds Packagetree platform filtering.
* Add all=true to the r4equest for packages in order to get Windows packages included. [2735781]
* ciocore.data singleton ensures that no windows -packages are provided if there are no windows
instance types, and vice versa.
* Remove urllib constraint because pip complained. Might have to add this back if we get weird breakages. [4e34359]
### Version:5.0.0-rc.1 -- 18 Mar 2022
* This version contains breaking changes. Clients should be upgraded.
* PackageTree now supports platform suffixes.
* The singleton that brings together inst_types and packages now filter's each automatically based on platforms provided by the other.
* Improvements to PackageEnvironment, including a constraint to prevent a merge_policy change, and platform detection for correct path separator.
* Better validation for the value of the CONDUCTOR_URL variable.
* Several uploader improvements, including support for reporting progress to external threads and gracefull cancelling the upload process.
### Version:4.0.7 -- 10 Mar 2022
* Adds a flag to propogate log properties. [dc12f84]
### Version:4.0.6 -- 03 Mar 2022
* Removes automatic setup of logging. adds an option to disable console logging during setup. [9194b4b]
### Version:3.4.1 -- 03 Mar 2022
* Adds a centralized py2/3 compatible reload logic. [1f5db2a]
* Avoid validators being duplicated
### Version:3.2.4 -- 27 Jan 2022
* Corrects very poor downloader performance in Python3 when runing in daemon mode.
### Version:3.2.3 -- 13 Jan 2022
* Repaired broken uploader args resolution. Specifically, md5 caching was None by default, which was treated as OFF. [615bfbe]
### Version:3.1.0 -- 16 Nov 2021
* Adds a post_install script to reset the shebang. This avoids errors that could occur when the
Python installation in Companion is changed or removed. [3918011]
* Conductor cmd-line now asks for python 3. [093d9ea]
### Version:3.0.11 -- 21 Oct 2021
* Rename original downloader to downloader and remove linux daemon downloader version. [6b54566]
* Use a fallback folder for downloads if the output folder doesn't exist. Also support a comma
separated task id list. [fb40e8d]
* conductor command finds python on Windows in a cleaner way now. [0ab3261]
### Version:3.0.10 -- 01 Sep 2021
* The Windows bat command is simplified. It just tries to use python.exe in the path, and if it fails it provides some help. [0ab3261]
### Version:3.0.9 -- 27 Aug 2021
* Fixes a bug where DCCs on Windows using Python2 would fail to upload because the result of a file
stat size was a long, and long was erroneousl removed from the type check while converting to python 3 compatibility.
### Version:3.0.7 -- 03 Aug 2021
* Fix urllib version because of an error when using proxies. See https://stackoverflow.com/a/66669839/179412 [b89593e]
* Keep auth_url for backwards compat. [1bfe5ed]
* Eliminate use of project_url such as atomic-light-001. [b687f7a]
### Version:3.0.6 -- 20 Jul 2021
* Fixed a bug where filenames and md5s were failing assertions in py2 based hosts, like C4DR21 and Maya2019. [0ad6a88]
### Version:3.0.5 -- 19 Jul 2021
* Requests exceptions attrs changed, so we now use a try except block for both eventualities. [5f198c8]
* Keep requests under 2.25.1 for m19 because chardet is replaced in 2.26.0 with a buggy replacement charset_normalizer. [f5ebadb]
### Version:3.0.4 -- 17 Jul 2021
* Fixed bug where server.py global vars were not reset at start of an auth run(). This caused a
problem when credentials were manually deleted during a single python session, which could happen
if we suggest that course of action in a support session. [6b4132c]
### Version:3.0.3 -- 12 Jul 2021
* Threading Isalive function was renamed to is_alive in py 3.9, so we now try both. [0070709]
* Disable multiprocessing cached bearer token in order to work around a bug in c4d24, where the app
itself would respawn recursively if our plugin was registered.
[27afab9]
* Use any python version to run the conductor command. Shebang looks for python, not python2. [fd13011]
* Fix bug in pkgs and add windows path support. [fb7a71e]
### Version:3.0.2 -- 27 Jun 2021
* Remove the index.html file in favour of an inline string. [05fab98]
### Version:3.0.1 -- 20 Jun 2021
* Removes dependency on builtins which caused an error when running through Python2.7 [a37b118]
* Better spawning for conductor command on Windows. It now searches the path for Python versions
with a preference for Python3 [ed11102]
* Config is no longer a singleton instantiated on initialization. It can be force refreshed,
although that functionality is not yet utilized. [1407582]
### Version:2.0.1 -- 10 Jun 2021
* Adds Python 3 support for VFX Platform (CY2020) Maya2022, C4d23 etc. [fa5005b]
### Version:1.0.3 -- 10 Jun 2021
* Restored the please-close-your-browser index.html page that appears after authentication. [ba11cb1]
### Version:1.0.2 -- 29 Apr 2021
* Fix logging - use const for logger name. [38111bf]
* Adds get_api_key_from_variable function. [cd57428]
### Version:1.0.0 -- 07 Apr 2021
* Version 1.0.0 is not backward compatible with older versions. The functionality from sequence, gpath, and expander has been extracted to separate Py3 compatible packages. [25c64c6]
### Version:0.3.0 -- 23 Mar 2021
* Gpath supports dollar brace variables for context expansion. [ff2fae9]
### Version:0.2.26 -- 11 Mar 2021
* Better local dev flow and instructions. [4566a0b]
### Version:0.2.25 -- 11 Mar 2021
* Adds skulk to deps. [4c2dc89]
* Developer install instructions.. [b9afd1d]
### Version:0.2.20 -- 11 Mar 2021
* Set up repo for Circle CI deploy. [e111736]
### Version:0.2.19 -- 24 Feb 2021
* Adds basename modifiers to expander. Example, use `<basename scenefile>` instead of `<scenename>` [daf1af3]
### Version:0.2.18 -- 15 Jan 2021
* Adds expression resolution to expander class. [cbeb64e]
* Unc support (#16)
### Version:0.2.16 -- 07 Jan 2021
* Gpath supports UNC paths. [7333433]
### Version:0.2.15 -- 06 Nov 2020
* Developer can switch between using test-fixtures and the real API within one session by using data.set_fixtures_dir() method. As a result, the DISK_CACHE env var is no longer used and has been deleted. [226b860]
### Version:0.2.14 -- 21 Oct 2020
* Repair windows command line executable. Added a bat file so that it is no longer necessary to call python explicitly. [cf9f19f]
### Version:0.2.13 -- 08 Oct 2020
* Adds README and CHANGELOG to PyPi description. [97a07dd]
### Version:0.2.12 -- 04 Oct 2020
* Catch unimplemented thread_count. [c64ad79]
### Version:0.2.11 -- 01 Oct 2020
* Package environment now allows exclusive override. [481851b]
* Allow relative paths that start with letter, colon - it could happen. [241fc68]
### Version:0.2.10 -- 08 Sep 2020
* Gpath allow colons in paths. [9c19d3a]
### Version:0.2.9 -- 06 Sep 2020
* Sequence consumers must use factory. [01a7dd7]
### Version:0.2.8 -- 03 Sep 2020
* Tidy and remove dependency on future. [0c2f72c]
### Version:0.2.7 -- 28 Aug 2020
* Adds cycle_progressions chunk strategy for improved scout frame distribution. [9b42288]
* Remove yaml config (#8)
* Remove config file references
* Adds jwt domain validation for credentials file so token is renewed when switching Google projects.
* Api_url defaults to same domain as auth_url
### Version:0.2.6 -- 18 Aug 2020
* Adds jwt domain validation for creds file. [0042b2b]
* Remove config file references. [451896b]
### Version:0.2.5 -- 11 Aug 2020
* Production release
### Version:0.2.4 -- 11 Aug 2020
* Removed redshift package ids hack. [8d11367]
### Version:0.2.3 -- 10 Aug 2020
* Adds remove_missing files method to Gpath. [3b96796]
* Adds validators base class. [c91ffc9] [b074cce]
### Version:0.2.2 -- 08 Aug 2020
* Temp workarounds for missing plugin-host links in packages
* Remove "shared" request.session object. (#293)
* Update httpbatchworker docstring (#291)
* File api refactor multipart and tcp connection pooling (#290)
* Multipart updates to workers
* Add make_preapred_request for s3 calls to remove transfer-encoding header being added. update descriptions and v2 endpoints. Add content-length for s3 calls. don't return response object on s3 calls which can cause a build up of memory due to jobworker, return headers or none.
* Add metric_store increments for aws presigned and multipart, cannot use chunked reader since generator does not have len function.
* Use httpbatchworker response to avoid additional os.stat calls. [7c76c46]
### Version:0.2.0 -- 01 Aug 2020
* Use explicit entry point script. [dfd9da3]
* The coredata singleton that holds instance_types, projects, and software must now be initialized with software product. This is to avoid having to specify the product on every call to data(). [db0fcb0]
* Expander context made public to indicate to other objects that they may retrieve the context. [a457aa1]
### Version:0.1.15 -- 27 Jul 2020
* Made post install script runnable. [8dce89f]
### Version:0.1.14 -- 27 Jul 2020
* Simplified post install script - no longer interactive. [6bba8af]
### Version:0.1.13 -- 26 Jul 2020
* Refactored post install setup script. [3fbba5e]
### Version:0.1.12 -- 25 Jul 2020
* Setup can accept cmdline args for non-wizard mode. [f195b7d]
* Adds safe mode to expander. [95f6cfd]
### Version:0.1.11 -- 22 Jul 2020
* Adds Setup wizard to make installation easier. [a06d610]
### Version:0.1.10 -- 21 Jul 2020
* Use cio to differentiate from conductor. [3f49451]
* Projects and instance types sorted for fixtures. [4a0f833]
### Version:0.1.9 -- 10 Jul 2020
* Implement data singletons here and remove from conductor-maya. [d31aeaa]
### Version:0.1.8 -- 06 Jul 2020
* Repair some path list issues found while adding remove method. [7368cc2]
### Version:0.1.7 -- 04 Jul 2020
* Adds a remove method to pathlist. [5712d2c]
* Wip configure script. [e7f6ea9]
### Version:0.1.6 -- 29 Jun 2020
* Fix bad path to conductor cmd. [5be12af]
* Packages: Remove tree property and make _tree -> tree. [1e6ae34]
### Version:0.1.5 -- 22 Jun 2020
* Remove src. [a366047]
* Newline in requirements. [a4cd9d3]
* Missing member. [b2bfb6d]
* Fix tests. [c9498c9]
* Imports ciocore. [916651e]
* Flatten. [c88afae]
* Flatter hierarchy. [6906e23]
* Slug means local slug. [0db16e0]
* Init declare namespace. [84d16b3]
* Use underscore name. [1aae967]
* Correct namespace config. [22a09f3]
### Version:0.1.4 -- 14 Jun 2020
* Ignore build dir. [8080df5]