معرفی شرکت ها
batchflow-0.8.1
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
A framework for fast data processing and ML models training
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | batchflow-0.8.1 |
| نام | batchflow |
| نسخه کتابخانه | 0.8.1 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Roman Kh at al |
| ایمیل نویسنده | rhudor@gmail.com |
| آدرس صفحه اصلی | https://github.com/analysiscenter/batchflow |
| آدرس اینترنتی | https://pypi.org/project/batchflow/ |
| مجوز | Apache License 2.0 |
[](https://www.apache.org/licenses/LICENSE-2.0)
[](https://python.org)
[](https://pytorch.org)
[](https://codecov.io/gh/analysiscenter/batchflow)
[](https://badge.fury.io/py/batchflow)
[](https://github.com/analysiscenter/batchflow/actions?query=workflow%3Astatus)
# BatchFlow
`BatchFlow` helps you conveniently work with random or sequential batches of your data
and define data processing and machine learning workflows even for datasets that do not fit into memory.
For more details see [the documentation and tutorials](https://analysiscenter.github.io/batchflow/).
Main features:
- flexible batch generaton
- deterministic and stochastic pipelines
- datasets and pipelines joins and merges
- data processing actions
- flexible model configuration
- within batch parallelism
- batch prefetching
- ready to use ML models and proven NN architectures
- convenient layers and helper functions to build custom models
- a powerful research engine with parallel model training and extended experiment logging.
## Basic usage
```python
my_workflow = my_dataset.pipeline()
.load('/some/path')
.do_something()
.do_something_else()
.some_additional_action()
.save('/to/other/path')
```
The trick here is that all the processing actions are lazy. They are not executed until their results are needed, e.g. when you request a preprocessed batch:
```python
my_workflow.run(BATCH_SIZE, shuffle=True, n_epochs=5)
```
or
```python
for batch in my_workflow.gen_batch(BATCH_SIZE, shuffle=True, n_epochs=5):
# only now the actions are fired and data is being changed with the workflow defined earlier
# actions are executed one by one and here you get a fully processed batch
```
or
```python
NUM_ITERS = 1000
for i in range(NUM_ITERS):
processed_batch = my_workflow.next_batch(BATCH_SIZE, shuffle=True, n_epochs=None)
# only now the actions are fired and data is changed with the workflow defined earlier
# actions are executed one by one and here you get a fully processed batch
```
## Train a neural network
`BatchFlow` includes ready-to-use proven architectures like VGG, Inception, ResNet and many others.
To apply them to your data just choose a model, specify the inputs (like the number of classes or images shape)
and call `train_model`. Of course, you can also choose a loss function, an optimizer and many other parameters, if you want.
```python
from batchflow.models.torch import ResNet34
my_workflow = my_dataset.pipeline()
.init_model('model', ResNet34, config={'loss': 'ce', 'classes': 10})
.load('/some/path')
.some_transform()
.another_transform()
.train_model('ResNet34', inputs=B.images, targets=B.labels)
.run(BATCH_SIZE, shuffle=True)
```
For more advanced cases and detailed API see [the documentation](https://analysiscenter.github.io/batchflow/).
## Installation
> `BatchFlow` module is in the beta stage. Your suggestions and improvements are very welcome.
> `BatchFlow` supports python 3.6 or higher.
### Stable python package
With modern [pipenv](https://docs.pipenv.org/)
```
pipenv install batchflow
```
With old-fashioned [pip](https://pip.pypa.io/en/stable/)
```
pip3 install batchflow
```
### Development version
With modern [pipenv](https://docs.pipenv.org/)
```
pipenv install git+https://github.com/analysiscenter/batchflow.git#egg=batchflow
```
With old-fashioned [pip](https://pip.pypa.io/en/stable/)
```
pip3 install git+https://github.com/analysiscenter/batchflow.git
```
After that just import `batchflow`:
```python
import batchflow as bf
```
### Git submodule
In many cases it might be more convenient to install `batchflow` as a submodule in your project repository than as a python package.
```
git submodule add https://github.com/analysiscenter/batchflow.git
git submodule init
git submodule update
```
If your python file is located in another directory, you might need to add a path to `batchflow`:
```python
import sys
sys.path.insert(0, "/path/to/batchflow")
import batchflow as bf
```
What is great about using a submodule that every commit in your project can be linked to its own commit of a submodule.
This is extremely convenient in a fast paced research environment.
Relative import is also possible:
```python
from .batchflow import Dataset
```
## Projects based on BatchFlow
- [SeismiQB](https://github.com/gazprom-neft/seismiqb) - ML for seismic interpretation
- [SeismicPro](https://github.com/gazprom-neft/SeismicPro) - ML for seismic processing
- [PetroFlow](https://github.com/gazprom-neft/petroflow) - ML for well interpretation
- [PyDEns](https://github.com/analysiscenter/pydens) - DL Solver for ODE and PDE
- [RadIO](https://github.com/analysiscenter/radio) - ML for CT imaging
- [CardIO](https://github.com/analysiscenter/cardio) - ML for heart signals
## Citing BatchFlow
Please cite BatchFlow in your publications if it helps your research.
[](https://doi.org/10.5281/zenodo.1041203)
```
Roman Khudorozhkov et al. BatchFlow library for fast ML workflows. 2017. doi:10.5281/zenodo.1041203
```
```
@misc{roman_kh_2017_1041203,
author = {Khudorozhkov, Roman and others},
title = {BatchFlow library for fast ML workflows},
year = 2017,
doi = {10.5281/zenodo.1041203},
url = {https://doi.org/10.5281/zenodo.1041203}
}
```
نیازمندی
| مقدار | نام |
|---|---|
| >=1.25 | urllib3 |
| >=1.10 | numpy |
| >=0.24 | pandas |
| >=0.2.7 | dill |
| >=4.19.7 | tqdm |
| >=0.19.1 | scipy |
| >=0.13.1 | scikit-image |
| >=3.0 | matplotlib |
| >=1.7 | torch |
| >=0.3 | einops |
نحوه نصب
نصب پکیج whl batchflow-0.8.1:
pip install batchflow-0.8.1.whl
نصب پکیج tar.gz batchflow-0.8.1:
pip install batchflow-0.8.1.tar.gz