معرفی شرکت ها

تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

توضیحات

A testing framework for decision optimization model learning algorithms.

ویژگی	مقدار
سیستم عامل	-
نام فایل	doframework-0.2.1
نام	doframework
نسخه کتابخانه	0.2.1
نگهدارنده	[]
ایمیل نگهدارنده	[]
نویسنده	Orit Davidovich
ایمیل نویسنده	orit.davidovich@ibm.com
آدرس صفحه اصلی	https://github.com/IBM/doframework
آدرس اینترنتی	https://pypi.org/project/doframework/
مجوز	Apache-2.0

# DOFramework `doframework` is a testing framework for decision-optimization model learning algorithms. Such algorithms learn part or all of a decision-optimization model from data and solve the model to produce a predicted optimal solution. `doframework` randomly generates multiple optimization problems (f,O,D,x*) for your algorithm to learn and solve: * f is a continuous piece-wise linear function defined over a domain in d-dimensional space (d>1), * O is a feasibility region in dom(f) defined by linear constraints, * D = (X,y) is a dataset derived from f, * x* is the true optimum of f in O (minimum or maximum). `doframework` feeds your algorithm constraints and data (O,D) and collects its predicted optimum. The algorithm's predicted optimal value can then be compared to the true optimal value f(x*). By comparing the two over multiple randomly generated optimization problems, `doframework` produces a **prediction profile** for your algorithm. `doframework` integrates with your algorithm (written in Python). # Design `doframework` was designed for optimal cloud distribution following an event-driven approach. `doframework` was built on top of [ray](https://www.ray.io/ "Ray -- fast and simple distributed computing") for cloud distribution and [rayvens](https://github.com/project-codeflare/rayvens "Rayvens augments Ray with events") for event driven management. # Requirements `doframework` was written for Python version >= 3.8.0. `doframework` can run either locally or remotely. For optimal performance, run it on a Kubernetes cluster. Cloud configuration is currently available for AWS and IBM Cloud [OpenShift](https://docs.openshift.com/ "RedHat OpenShift Documentation") clusters. The framework uses storage (local or S3) to interact with simulation products. Configuration is currently available for [AWS](https://aws.amazon.com/s3/ "AWS S3") or [IBM Cloud Object Storage COS](https://www.ibm.com/cloud/object-storage "IBM Cloud Object Storage"). # Install To run `doframework` locally, install with ``` $ pip install doframework ``` # Configs Storage specifications are provided in a `configs.yaml`. You'll find examples under `./configs/*`. The `configs.yaml` includes the list of source and target bucket names (under `buckets`). If necessary, S3 credentials are added under designated fields. Here is the format of the `configs.yaml` either for local storage ``` local: buckets: inputs: '<inputs-folder>' inputs_dest: '<inputs-dest-folder>' objectives: '<objectives-folder>' objectives_dest: '<objectives-dest-folder>' data: '<data-folder>' data_dest: '<data-dest-folder>' solutions: '<solutions-folder>' ``` or S3 ``` s3: buckets: inputs: '<inputs-bucket>' inputs_dest: '<inputs-dest-bucket>' objectives: '<objectives-bucket>' objectives_dest: '<objectives-dest-bucket>' data: '<data-bucket>' data_dest: '<data-dest-bucket>' solutions: '<solutions-bucket>' aws_secret_access_key: 'xxxx' aws_access_key_id: 'xxxx' endpoint_url: 'https://xxx.xxx.xxx' region: 'xx-xxxx' cloud_service_provider: 'aws' ``` Currently, two S3 providers are available under `s3:cloud_service_provider`: either `aws` or `ibm`. The `endpoint_url` is _optional_ for AWS. **Bucket / folder names must be distinct**. # Inputs `input.json` files provide the necessary metadata for the random genration of optimization problems. `doframework` will run end to end, once `input.json` files are uploaded to `<inputs-bucket>` / `<inputs-folder>`. The jupyter notebook `./notebooks/inputs.ipynb` allows you to automatically generate input files and upload them to `<inputs-bucket>`. Here is an example of an input file (see input samples `input_basic.json` under `./inputs`). ``` { "f": { "vertices": { "num": 7, "range": [[5.0,20.0],[0.0,10.0]], }, "values": { "range": [0.0,5.0] }, }, "omega" : { "ratio": 0.8 }, "data" : { "N": 750, "noise": 0.01, "policy_num": 2, "scale": 0.4 }, "input_file_name": "input_basic.json" } ``` `f:vertices:num`: number of vertices in the piece-wise linear graph of f. `f:vertices:range`: f domain will be inside this range. `f:values:range`: range of f values. `omega:ratio`: vol(O) / vol(dom(f)) >= ratio. `data:N`: number of data points to sample. `data:noise`: response variable noise. `data:policy_num`: number of centers in Gaussian mix distribution of data. `data:scale`: max STD of Gaussian mix distribution of data (as a ratio of domain diameter). It's a good idea to start experimenting on low-dimensional problems. # User App Integration Your algorithm will be integrated into `doframework` once it is decorated with `doframework.resolve`. A `doframework` experiment runs with `doframework.run()`. The `run()` utility accepts the decorated model and an absolute path to the `configs.yaml`. Here is an example a user application `module.py`. ``` import doframework as dof @dof.resolve def alg(data: np.array, constraints: np.array, **kwargs): ... return optimal_arg, optimal_val, regression_model if __name__ == '__main__': dof.run(alg, 'configs.yaml', objectives=5, datasets=3, **kwargs) ``` `doframework` provides the following inputs to your algorithm: `data`: 2D np.array with features X = data[ : , :-1] and response variable y = data[ : ,-1]. `constraints`: linear constraints as a 2D numpy array A. A data point x satisfies the constraints when A[ : , :-1]*x + A[ : ,-1] <= 0. It feeds your algorithm additional inputs in kwargs: `lower_bound`: lower bound per feature variable. `upper_bound`: upper bound per feature variable. `init_value`: optional initial value. The `run()` utility accepts the arguments: `objectives`: number of objective targets to generate per input file. `datasets`: number of datasets to generate per objective target. `distribute`: True to run distributively, False to run sequentially. `logger`: True to see `doframework` logs, False otherwise. `after_idle_for`: stop running when event stream is idle after this many seconds. `alg_num_cpus`: number of CPUs to dedicate to your algorithm on each optimization task. `data_num_cpus`: number of CPUs to dedicate to data generation (useful in high dimensions). # Algorithm Prediction Profile Once you are done running a `doframework` experiment, run the notebook `notebooks/profile.ipynb`. It will fetch the relevant experiment products from the target buckets and produce the algorithm's prediction profile and prediction probabilities. `doframework` produces three types of experiment product files: * `objective.json`: containing information on (f,O,x*) * `data.csv`: containing the dataset the algorithm accepts as input * `solution.json`: containing the algorithm's predicted optimum See sample files under `./outputs`. # Kubernetes Cluster To run `doframework` on a K8S cluster, make sure you are on the cluster's local `kubectl` context. Log into your cluster, if necessary (applicable to OpenShift, see `./doc/openshift.md`). You can check your local `kubectl` context and change it if necessary with ``` $ kubectl config current-context $ kubectl config get-contexts $ kubectl config use-context cluster_name >> Switched to context "cluster_name". ``` Now `cd` into your project's folder and run the setup bash script `doframework-setup.sh`. The setup script will generate the cluster configuration file `doframework.yaml` in your project's folder. The setup script requires the absolute path to your `configs.yaml`. Running the setup `.sh` script will establish the `ray` cluster. ``` $ cd <user_project_folder> $ doframework-setup.sh --configs ~/path/to/configs.yaml ``` You have the option to adapt `doframework.yaml` to fit your application. Use the flag `--project-requirements` to specify the absolute path to your `requirements.txt` file. It will be `pip install -r requirements.txt` on your cluster nodes. Use the flag `--project-dir` to specify the absolute path to your project. It will be pip installed on your cluster nodes. ``` $ doframework-setup.sh --configs ~/path/to/configs.yaml --project-requirements <absolute_requirements_path> --project-dir <absolute_project_path> ``` Use the `--skip` flag to skip re-generating the `doframework.yaml`. ``` $ doframework-setup.sh --skip ``` Or, in case you are familiar with `ray`, run instead ``` $ ray up doframework.yaml --no-config-cache --yes ``` Upload `input.json` file(s) to your `<inputs_bucket>`. Now you can submit your application `module.py` to the cluster ``` $ ray submit doframework.yaml module.py ``` # Ray Cluster To observe the `ray` dashboard, connect to `http://localhost:8265` in your browser. See `./doc/openshift.md` for OpenShift-specific instructions. Some useful health-check commands: * Check the status of `ray` pods ``` $ kubectl get pods -n ray ``` * Check the status of the `ray` head node ``` $ kubectl describe pod rayvens-cluster-head-xxxxx -n ray ``` * Monitor autoscaling with ``` $ ray exec doframework.yaml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*' ``` * Connect to a terminal on the head node ``` $ ray attach doframework.yaml $ ... $ exit ``` * Get a remote shell to the cluster manually (find the head node ID with `kubectl describe`) ``` $ kubectl -n ray exec -it rayvens-cluster-head-z97wc -- bash ``` After introducing manual changes to `doframework.yaml`, update with ``` $ ray up doframework.yaml --no-config-cache --yes ``` Shutdown the `ray` cluster with ``` $ ray down -y doframework.yaml ``` # Test Run the setup bash script `doframework-setup.sh` with the `--example` flag to generate the test script `doframework_example.py` in your project folder. ``` $ cd <user_project_folder> $ doframework-setup.sh --configs ~/path/to/configs.yaml --example ``` To run the test script locally, use ``` $ python doframework_example.py --configs ~/path/to/configs.yaml ``` To run the test script on your K8S cluster, use ``` $ ray submit doframework.yaml doframework_example.py --configs configs.yaml ``` [NOTE: we are using the path to the `configs.yaml` that was mounted on cluster nodes under `$HOME`.] Make sure to upload input json files to `<inputs-bucket>` / `<inputs-folder>` once you run `doframework_example.py`.

نیازمندی

مقدار	نام
-	numpy
-	pandas
-	matplotlib
>=1.13.0	ray[default,k8s,serve]
>=0.7.0	rayvens
>=2.10.0	ibm-cos-sdk
>=1.17.110	boto3
>=3.7.4	aiohttp
>=1.3.1	aioredis
>=0.24.1	scikit-learn
-	scipy
>=2.4	PuLP
>=1.9.9	GPy

زبان مورد نیاز

مقدار	نام
>=3.8	Python

نحوه نصب

نصب پکیج whl doframework-0.2.1:

pip install doframework-0.2.1.whl

نصب پکیج tar.gz doframework-0.2.1:

pip install doframework-0.2.1.tar.gz