معرفی شرکت ها


dataloop-upipe-0.2.0


Card image cap
تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر
Card image cap
تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر
Card image cap
تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر
Card image cap
تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر
Card image cap
تبلیغات ما

مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.

مشاهده بیشتر

توضیحات

Micro Pipelines for Dataloop platform
ویژگی مقدار
سیستم عامل -
نام فایل dataloop-upipe-0.2.0
نام dataloop-upipe
نسخه کتابخانه 0.2.0
نگهدارنده []
ایمیل نگهدارنده []
نویسنده Dataloop Team
ایمیل نویسنده info@dataloop.ai
آدرس صفحه اصلی -
آدرس اینترنتی https://pypi.org/project/dataloop-upipe/
مجوز -
# uPipe # [![pypi](https://img.shields.io/pypi/v/dataloop-upipe.svg)](https://pypi.org/project/dataloop-upipe/) [![versions](https://img.shields.io/pypi/pyversions/dataloop-upipe.svg)](https://github.com/dataloop-ai/dtlpy-upipe) Micro-pipe is asyncio based framework for running **simple**,high performance, parallel and synchronized python processes within a single machine. uPipe comes with auto-scaler for maximum throughput and hardware utilization and a monitoring tool. ```python a = Processor('a', entry='./a.py', config=config) b = Processor('b', entry='./b.py', config=config) pipe = Pipe('plus-one') pipe.add(a).add(b) await pipe.start() await pipe.wait_for_completion() print("Running") ``` ## Why uPipe ? ## In the world of machine learning python has taken the stage as the go-to programming language, yet python has few fundamental issues preventing it from being production friendly ### Parallel processing ### Python has a built-in weakness for parallel data processing, this weakness is sourced in the core of python at the [GIL](https://wiki.python.org/moin/GlobalInterpreterLock) The GlobalInterpreterLock prevents from different threads to access memory (and therefore variables) in parallel. This essentially means that a multi-threaded python program executes slower compared to single thread on CPU intensive workloads. Python also offers multiprocessing capabilities that are not blocked by GIL since the workloads are running in different processes, yet using multiprocessor does not allow sharing memory and communication between process implicitly uses [Pickling](https://docs.python.org/3/library/pickle.html) for data sharing, which in turn significantly degrades data throughput. ### Parallel optimization ### Any given processing workload is bounded by critical hardware resources where the leading resources are I/O (Network and disk), Memory and CPU. Machine learning also introduces additional resources like GPU cores, CPU memory and other AI specific accelerators. Optimizing the overall data on a given machine is a process of trial and error, often includes non-trivial technical challenges.uPipe implements [AutoScaling](https://en.wikipedia.org/wiki/Autoscaling) by process into the single machine with maximum pipeline throughput as goal. ### Simple ### uPipe exposes a simple API for creating data pipelines from existing packages and library's run with fews line of code. The basics entities: * DataFrame: Arbitrary binary data that is shared between processors. * Processor: A python code that can receive, process and generate(emit) data frames. Processor is a logical entities that can encapsulate many (os) processes of the same processor code. * Queue: Two processors will communicate DataFrames using queue, usually memory based queue. * Pipe: The Pipeline the defines the data flow between Processors and manages the processing execution. ### Read more ### * [uPipe basics](docs/basics/basics.md) * [uPipe API](docs/api/api.md) * [uPipe architecture](docs/architecture/architecture.md) * [uPipe design](docs/design/design.md) ### Install ### ```shell pip install dataloop-upipe ``` ### Import ### from dataloop.upipe import Processor, Pipe ## Usage ## ### Processor declaration ### Processor can be defined inline methods(_func_) or as external scripts (_entry_) in order to be used in pipeline definition, the name for each processor must be a unique string and used to identify the processor. ```python a = Processor('my name is A', func=processor_a) b = Processor('b', entry='./b.py') ``` ### Pipe definition ### Processor can be chained to pipe and to each other by the _add_ method ```python pipe = Pipe('some-pipe') pipe.add(a).add(b) #a,b are processors ``` ### Joining a Processor to a pipe### This code is declaring the Worker's code and _join_ in order to be able to receive and emit data frames. ```python proc = Worker("b") await proc.join() # Processor is now ready to ingest and emit data ``` ### Acquiring data by processor ### Worker can get data using the _get_ and _get_sync_ methods ```python proc = Worker("b") await proc.join() # Processor is now ready to ingest and emit data #get data frame, return None if no data available data_frame = await proc.get() #get data frame, wait until data available data_frame = await proc.get_sync() ``` ### Emitting data by processor ### Worker can emit data frame for the next Processor in Pipe using the _emit_ and _emit_sync_ methods ```python proc = Worker("b") await proc.join() # Processor is now ready to ingest and emit data #emit data frame, return None if no space (back pressure from next processor) await proc.emit() #emit data frame, wait if no space (back pressure) await proc.emit_sync() ``` ### start pipe ### ```python await pipe.start() await pipe.wait_for_completion() ``` ### terminate pipe ### ```python await pipe.terminate() await pipe.wait_for_completion() # wait for existing data to complete ```


نیازمندی

مقدار نام
<=1.22.* numpy
<=0.70.0 fastapi
<=0.15.0 uvicorn
<=0.20.0 httpx
<=10.0 websockets
<=1.8.2 pydantic
- python-multipart
<=3.7.4.post0 aiohttp
<=5.8.0 psutil
<=0.16.0 starlette
<=4.5.3.56 opencv-python
<=0.4.4 colorama
<=1.2.1 websocket-client
<=2.26.0 requests


زبان مورد نیاز

مقدار نام
>=3.8 Python


نحوه نصب


نصب پکیج whl dataloop-upipe-0.2.0:

    pip install dataloop-upipe-0.2.0.whl


نصب پکیج tar.gz dataloop-upipe-0.2.0:

    pip install dataloop-upipe-0.2.0.tar.gz