معرفی شرکت ها
bagua-cuda116-0.9.3.dev5
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Bagua is a deep learning training acceleration framework for PyTorch. It provides a one-stop training acceleration solution, including faster distributed training compared to PyTorch DDP, faster dataloader, kernel fusion, and more.
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | bagua-cuda116-0.9.3.dev5 |
| نام | bagua-cuda116 |
| نسخه کتابخانه | 0.9.3.dev5 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Kuaishou AI Platform & DS3 Lab |
| ایمیل نویسنده | admin@mail.xrlian.com |
| آدرس صفحه اصلی | https://github.com/BaguaSys/bagua |
| آدرس اینترنتی | https://pypi.org/project/bagua-cuda116/ |
| مجوز | - |
<p align="center">
<img width="500px" src="https://user-images.githubusercontent.com/18649508/141055752-2f429618-2707-4feb-abe3-74767c2b1976.gif"/>
</p>
<hr/>
<div align="center">
<a href="https://tutorials.baguasys.com/"><img src="https://img.shields.io/badge/tutorials-passing-green" alt="tutorials"></a> <a href="http://bagua.readthedocs.io/?badge=latest"><img src="https://readthedocs.org/projects/bagua/badge/?version=latest" alt="Documentation Status"></a> <a href="https://pypi.org/project/bagua/"><img src="https://pepy.tech/badge/bagua/month" alt="Downloads"></a> <a href="https://hub.docker.com/r/baguasys/bagua"><img src="https://img.shields.io/docker/pulls/baguasys/bagua" alt="Docker Pulls"></a> <a href="https://hub.docker.com/r/baguasys/bagua"><img src="https://img.shields.io/docker/cloud/build/baguasys/bagua" alt="Docker Cloud Build Status"></a> <a href="https://github.com/BaguaSys/bagua/blob/master/LICENSE"><img src="https://img.shields.io/github/license/BaguaSys/bagua" alt="GitHub license"></a>
</div>
<br/>
*WARNING: THIS PROJECT IS CURRENTLY IN MAINTENANCE MODE, DUE TO COMPANY REORGANIZATION.*
Bagua is a deep learning training acceleration framework for PyTorch developed by [AI platform@Kuaishou Technology](https://www.kuaishou.com/en) and [DS3 Lab@ETH Zürich](https://ds3lab.inf.ethz.ch/). Bagua currently supports:
- **Advanced Distributed Training Algorithms**: Users can extend the training on a single GPU to multi-GPUs (may across multiple machines) by simply adding a few lines of code (optionally in [elastic mode](https://tutorials.baguasys.com/elastic-training/)). One prominent feature of Bagua is to provide a flexible system abstraction that supports state-of-the-art system relaxation techniques of distributed training. So far, Bagua has integrated communication primitives including
- Centralized Synchronous Communication (e.g. [Gradient AllReduce](https://tutorials.baguasys.com/algorithms/gradient-allreduce))
- Decentralized Synchronous Communication (e.g. [Decentralized SGD](https://tutorials.baguasys.com/algorithms/decentralized))
- Low Precision Communication (e.g. [ByteGrad](https://tutorials.baguasys.com/algorithms/bytegrad))
- Asynchronous Communication (e.g. [Async Model Average](https://tutorials.baguasys.com/algorithms/async-model-average))
- [**Cached Dataset**](https://tutorials.baguasys.com/more-optimizations/cached-dataset): When data loading is slow or data preprocessing is tedious, they could become a major bottleneck of the whole training process. Bagua provides cached dataset to speedup this process by caching data samples in memory, so that reading these samples after the first time becomes much faster.
- [**TCP Communication Acceleration (Bagua-Net)**](https://tutorials.baguasys.com/more-optimizations/bagua-net): Bagua-Net is a low level communication acceleration feature provided by Bagua. It can greatly improve the throughput of AllReduce on TCP network. You can enable Bagua-Net optimization on any distributed training job that uses NCCL to do GPU communication (this includes PyTorch-DDP, Horovod, DeepSpeed, and more).
- [**Performance Autotuning**](https://tutorials.baguasys.com/performance-autotuning/): Bagua can automatically tune system parameters to achieve the highest throughput.
- [**Generic Fused Optimizer**](https://tutorials.baguasys.com/more-optimizations/generic-fused-optimizer): Bagua provides generic fused optimizer which improve the performance of optimizers by fusing the optimizer `.step()` operation on multiple layers. It can be applied to arbitrary PyTorch optimizer, in contrast to [NVIDIA Apex](https://nvidia.github.io/apex/optimizers.html)'s approach, where only some specific optimizers are implemented.
- [**Load Balanced Data Loader**](https://bagua.readthedocs.io/en/latest/autoapi/bagua/torch_api/contrib/load_balancing_data_loader/index.html): When the computation complexity of samples in training data are different, for example in NLP and speech tasks, where each sample have different lengths, distributed training throughput can be greatly improved by using Bagua's load balanced data loader, which distributes samples in a way that each worker's workload are similar.
- [**Integration with PyTorch Lightning**](https://pytorch-lightning.readthedocs.io/en/latest/accelerators/gpu.html#bagua): Are you using [PyTorch Lightning](https://www.pytorchlightning.ai/) for your distributed training job? Now you can use Bagua in PyTorch Lightning by simply set `strategy=BaguaStrategy` in your Trainer. This enables you to take advantage of a range of advanced training algorithms, including decentralized methods, asynchronous methods, communication compression, and their combinations!
Its effectiveness has been evaluated in various scenarios, including VGG and ResNet on ImageNet, BERT Large and many industrial applications at Kuaishou.
## Links
* [Bagua Main Git Repo](https://github.com/BaguaSys/bagua)
* [Bagua Tutorials](https://tutorials.baguasys.com/)
* [Bagua Examples](https://github.com/BaguaSys/bagua/tree/master/examples)
* [Bagua API Documentation](https://bagua.readthedocs.io/)
## Performance
<p align="center">
<img src="https://tutorials.baguasys.com/benchmark/figures/e2e_vgg16_128.png" width="600"/>
</p>
<p align="center">
The performance of different systems and algorithms on VGG16 with 128 GPUs under different network bandwidth.
</p>
<br/>
<br/>
<p align="center">
<img src="https://tutorials.baguasys.com/benchmark/figures/tradeoff_network_bert-large-bandwidth.png" width="250"/><img src="https://tutorials.baguasys.com/benchmark/figures/tradeoff_network_bert-large-latency.png" width="250"/><img src="https://tutorials.baguasys.com/benchmark/figures/tradeoff_network_legend.png" width="260"/>
</p>
<p align="center">
Epoch time of BERT-Large Finetune under different network conditions for different systems.
</p>
For more comprehensive and up to date results, refer to [Bagua benchmark page](https://tutorials.baguasys.com/benchmark/index.html).
## Installation
Wheels (precompiled binary packages) are available for Linux (x86_64). Package names are different depending on your CUDA Toolkit version (CUDA Toolkit version is shown in `nvcc --version`).
| CUDA Toolkit version | Installation command |
|----------------------|-----------------------------|
| >= v10.2 | `pip install bagua-cuda102` |
| >= v11.1 | `pip install bagua-cuda111` |
| >= v11.3 | `pip install bagua-cuda113` |
| >= v11.5 | `pip install bagua-cuda115` |
| >= v11.6 | `pip install bagua-cuda116` |
Add `--pre` to `pip install` commands to install pre-release (development) versions. See [Bagua tutorials](https://tutorials.baguasys.com/getting-started/) for quick start guide and more installation options.
## Quick Start on AWS
Thanks to the [Amazon Machine Images (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html), we can provide users an easy way to deploy and run Bagua on AWS EC2 clusters with flexible size of machines and a wide range of GPU types. Users can find our pre-installed Bagua image on EC2 by the unique AMI-ID that we publish here. Note that AMI is a regional resource, so please make sure you are using the machines in same reginon as our AMI.
| Bagua version | AMI ID | Region |
|---|---|---|
| 0.6.3 | ami-0e719d0e3e42b397e | us-east-1 |
| 0.9.0 | ami-0f01fd14e9a742624 | us-east-1 |
To manage the EC2 cluster more efficiently, we use [Starcluster](http://star.mit.edu/cluster/) as a toolkit to manipulate the cluster. In the `config` file of Starcluster, there are a few configurations that need to be set up by users, including AWS credentials, cluster settings, etc. More information regarding the Starcluster configuration can be found in this [tutorial](http://star.mit.edu/cluster/docs/latest/quickstart.html).
For example, we create a EC2 cluster with 4 machines, each of which has 8 V100 GPUs (`p3.16xlarge`). The cluster is based on the Bagua AMI we pre-installed in `us-east-1` region. Then the `config` file of Starcluster would be:
```yaml
# region of EC2 instances, here we choose us_east_1
AWS_REGION_NAME = us-east-1
AWS_REGION_HOST = ec2.us-east-1.amazonaws.com
# AMI ID of Bagua
NODE_IMAGE_ID = ami-0e719d0e3e42b397e
# number of instances
CLUSTER_SIZE = 4
# instance type
NODE_INSTANCE_TYPE = p3.16xlarge
```
With above setup, we created two identical clusters to benchmark a synthesized image classification task over Bagua and Horovod, respectively. Here is the screen recording video of this experiment.
<p align="center">
<a href="https://youtu.be/G8o5HVYZJvs"><img src="https://user-images.githubusercontent.com/18649508/136463585-ba911d20-9088-48b7-ab32-fc3e465c31b8.png" width="600"/></a>
</p>
## Cite Bagua
```bibtex
% System Overview
@misc{gan2021bagua,
title={BAGUA: Scaling up Distributed Learning with System Relaxations},
author={Shaoduo Gan and Xiangru Lian and Rui Wang and Jianbin Chang and Chengjun Liu and Hongmei Shi and Shengzhuo Zhang and Xianghong Li and Tengxu Sun and Jiawei Jiang and Binhang Yuan and Sen Yang and Ji Liu and Ce Zhang},
year={2021},
eprint={2107.01499},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
% Theory on System Relaxation Techniques
@book{liu2020distributed,
title={Distributed Learning Systems with First-Order Methods: An Introduction},
author={Liu, J. and Zhang, C.},
isbn={9781680837018},
series={Foundations and trends in databases},
url={https://books.google.com/books?id=vzQmzgEACAAJ},
year={2020},
publisher={now publishers}
}
```
## Contributors
<a href="https://github.com/BaguaSys/bagua/graphs/contributors">
<img src="https://contrib.rocks/image?repo=BaguaSys/bagua" />
</a>
نیازمندی
| مقدار | نام |
|---|---|
| - | setuptools-rust |
| - | colorama |
| - | tqdm |
| >=2.1 | deprecation |
| >=3.4 | pytest-benchmark |
| >=0.8.1 | scikit-optimize |
| !=1.0,<=1.0.1,>=0.24 | scikit-learn |
| - | numpy |
| >=2.0 | flask |
| >=0.11 | prometheus-client |
| ==2.9.1 | parallel-ssh |
| >=1.8 | pydantic |
| >=2.25 | requests |
| ==0.4.0 | gorilla |
| >=21.8 | gevent |
| >=2.0 | xxhash |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >=3.7 | Python |
نحوه نصب
نصب پکیج whl bagua-cuda116-0.9.3.dev5:
pip install bagua-cuda116-0.9.3.dev5.whl
نصب پکیج tar.gz bagua-cuda116-0.9.3.dev5:
pip install bagua-cuda116-0.9.3.dev5.tar.gz