# as_a_service
A simple package that transforms a batch function {inputs->results} into a service that
- groups inputs into batches - you specify max batch size and time waiting
- processes them - and returns results back to whoever was asking
### Usage:
__[notebook version](https://github.com/justheuristic/as_a_service/blob/master/example.ipynb)__
Here's how it feels
```python
@as_batched_service(batch_size=3, max_delay=0.1)
def square(batch_xs):
print("processing...", batch_xs)
return [x_i ** 2 for x_i in batch_xs]
# submit many queries
futures = square.submit_many(range(10))
print([f.result() for f in futures])
```
This will print
```
processing... [0, 1, 2]
processing... [3, 4, 5]
processing... [6, 7, 8]
processing... [9]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
```
You can also use it as a drop-in replacement for a function that processes one input at a time
* `square(2.0)` will return 4.0 as if a normal function
* Under the hood, it submits a request and waits for it to finish
This package contains three objects
- BatchedService(batch_process_func, batch_size, max_delay) - main object
- @as_batched_service(batch_size, max_delay) - same thing as a decorator
- @as_service(max_delay) - decorator for a function without batches (single input/output)
Use help(BatchedService) and "Why should I care?" for more details.
### Install:
* ```pip install as_a_service```
* No dependencies apart from standard libraries
* Works with both python2 and python3 (pip3 install)
### Why should I care?
This primitive is useful for a number of scenarios like:
1) You are building a web-based demo around your neural network. You want your network to process
a stream of user queries, but doing so one query at a time is slow. Batch-parallel processing is way better.
```python
@as_batched_service(batch_size=32, max_delay=1.0)
def service_predict(input_images_list):
predictions_list = my_network_predict_batch(input_images_list)
return predictions_list
@my_web_framework.run_as_a_thread_for_every_query
def handle_user_query(query):
input_image = get_image(query)
return service_predict(input_image)
```
2) You are experimenting with a reinforcement learning agent. The agent itself is a neural network
that predicts actions. You want to play 100 parallel game sessions to train on.
Playing one session at a time is slow. If only we could run multiple sessions on one GPU
```python
my_network = make_keras_network_on_gpu()
service = BatchedService(my_network.predict, batch_size=32, max_delay=1.0)
threads = [
GamePlayingThread(predict_action=lambda x: service(x)) for i in range(100)
]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
service.stop()
```
And many other scenarios where you want to use a single resource
(GPU / device /DB) concurrently and utilize batch-parallelism
