معرفی شرکت ها
ec2-cluster-0.6.0rc4
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
A tool for launching and running commands on multiple EC2 instances
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | ec2-cluster-0.6.0rc4 |
| نام | ec2-cluster |
| نسخه کتابخانه | 0.6.0rc4 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Armand McQueen |
| ایمیل نویسنده | armandmcqueen@gmail.com |
| آدرس صفحه اصلی | https://github.com/armandmcqueen/ec2-cluster |
| آدرس اینترنتی | https://pypi.org/project/ec2-cluster/ |
| مجوز | - |
# ec2-cluster
Simple CLI and Python library to spin up and run shell commands on clusters of EC2 instances using [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) and [`fabric`](https://github.com/fabric/fabric). Multi-purpose, but created to make deep learning distributed training infrastructure easier. Also very useful for running performance tests across multiple EC2 instance types.
## Quickstart
This code will launch a cluster of EC2 instances, run the command `hostname` on all of them, return the results of the command and then tear down the cluster.
```python
import ec2_cluster as ec3
with ec3.infra.ConfigCluster("cluster.yaml") as cluster:
sh = cluster.get_shell()
results = sh.run_on_all("hostname")
hostnames = [result.stdout for result in results]
```
## Long-running tasks
`ec2-cluster` is also designed for long-running tasks where you may not want to keep your local machine awake for the full duration. The library relies on EC2 tags to keep track of EC2 instances, letting you interact with a cluster across sessions without needing an always-on control plane.
Below is one way to launch complicated, long-running jobs and download the results at some later time. `check_on_job()` can be run in a different session or even a different machine than `launch_job()` as long as they have the same `cluster.yaml` and are using the same AWS account.
```python
import ec2_cluster as ec3
def launch_job():
cluster = ec3.infra.ConfigCluster("cluster.yaml")
cluster.launch(verbose=True)
sh = cluster.get_shell()
sh.copy_from_local_to_all("job_script.py", "job_script.py")
sh.run_on_all("python job_script.py > job.log 2>&1 &") # Launch script as background process
def check_on_job():
cluster = ec3.infra.ConfigCluster("cluster.yaml")
sh = cluster.get_shell()
statuses = sh.run_on_all("tail -n 1 job.log", hide=True)
for status in statuses:
if status.stdout.rstrip("\n") != "job_script.py complete":
print("Job is not yet complete on all instances")
return
print("Job is finally done on all instances!")
sh.copy_from_all_to_local("job.log", "./results/")
cluster.terminate(verbose=True)
```
This will create a directory on your local machine:
```
results/
├── 0
│ ├── ip.txt
│ └── job.log
├── 1
│ ├── ip.txt
│ └── job.log
└── 3
├── ip.txt
└── job.log
```
## Overview
`ec2-cluster` is designed for simple distributed tasks where Kubernetes is overkill. There is no setup required other than the ability to launch EC2 instances with `boto3` and the ability to SSH to those instances (only a requirement if you want to run commands on them).
`ec2-cluster` provides the ability to launch a cluster, to retrieve IP addresses for all nodes/nodes in the cluster, to delete the cluster and to execute commands on some or all of the instances.
Unlike like most cluster management tools, all cluster management in `ec2-cluster` happens client-side. `ec2-cluster` creates EC2 instances and gives them globally unique names (using the EC2 Name tag) which is later used to identify which instances are part of a given cluster. `ec2-cluster` then provides convenience classes to run commands on some or all of the instances via SSH (using the [`fabric`](https://github.com/fabric/fabric) library).
## Other benefits
- Resilient to EC2 capacity limits. If instances are not available, `ec2-cluster` will retry until the all nodes in the cluster are created or until the user-set timeout is reached.
- Easy to quickly launch duplicate clusters for parallel training runs.
- Can write orchestration logic that needs to be run when launching a cluster, e.g. enabling passwordless ssh between all instances for Horovod-based training
- Iterative, not disruptive, improvements on the common methodology of manually launching EC2 instances, ssh-ing to them, configuring environments by hand and running scripts
### Usage
`ec2-cluster` can be consumed in two ways:
- A CLI for launching, describing and deleting clusters.
- A python library for scripting.
This library has three main components:L
- **infra**: creating cluster infrastructure
- **orch**: orchestrating simple runtime cluster configuration (e.g. generate a hostfile with runtime IPs)
- **control**: running commands on the cluster
### CLI Quick Start
### Library Quick Start
نیازمندی
| مقدار | نام |
|---|---|
| <=2.5.1 | paramiko |
| >2.3.0 | fabric2 |
| - | boto3 |
| - | pyyaml |
| - | tabulate |
نحوه نصب
نصب پکیج whl ec2-cluster-0.6.0rc4:
pip install ec2-cluster-0.6.0rc4.whl
نصب پکیج tar.gz ec2-cluster-0.6.0rc4:
pip install ec2-cluster-0.6.0rc4.tar.gz