معرفی شرکت ها
api-rate-limiter-0.1.1
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Implement client side API rate limiting
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | api-rate-limiter-0.1.1 |
| نام | api-rate-limiter |
| نسخه کتابخانه | 0.1.1 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Bradley Atkins |
| ایمیل نویسنده | bradley.atkinz@gmail.com |
| آدرس صفحه اصلی | https://github.com/museadmin/api-rate-limiter |
| آدرس اینترنتی | https://pypi.org/project/api-rate-limiter/ |
| مجوز | MIT |
# API Rate Limiter
> A simple Python object that allows you to implement client side API
rate limiting.
---
## Introduction
This project was created to address a specific issue – API rate limiting
when scanning an AWS platform.
The problem with AWS applying rate limiting to the services is that it
is indiscriminate in nature. So if you are running a multi-threaded
scanner such as ScoutSuite, while for example an Auto Scaling Group is
trying to scale something, the auto Scaling can fail because AWS does
not distinguish between the third party scanner and its own services.
This is a cause for frustration amongst the platform support team and
can also cause an outage, particularly in the early stages of
development of a new service, where resilience and high availability are
still but a twinkle in the architect's eye...
This is becoming more and more of an issue as consumers realise the
necessity of scanning their cloud platforms.
## NOTE!
Apologies for the lack of images if you are viewing this on PyPi. For
some reason PyPi insist on changing the URLs to
https://warehouse-test-camo.cmh1.psfhosted.org/..... I've no idea why.
You can read this README correctly on the home project on GitHub.
## History
I started looking into this issue when scanning my own client's platform
and hit the rate limiting issue when scanning some 15k of snapshots.
Initially I managed to mitigate this using the Network Link Conditioner
on the MAC I was using to run the scanner.
The NLC allows you to add latency to all outbound packets on a given
network like so:
This blanket approach to delaying all outbound packets did resolve the
issue of the rate limiting, but at a cost of causing the scanner to now
run for two hours in order to complete a scan of all of our development
environments.
Not only was this slow, but it's also not OS agnostic, so hardly a good
solution.
It occurred to me that if a mechanism could be created within boto3
itself to queue outbound API calls at a configurable rate, then this
might might prove to be a more general solution to the issue.
So I forked botocore [here](https://github.com/museadmin/botorate) into
a project that combined forks of botocore, boto3 and ScoutSuite.
This project has the queue implemented in it and boto3 has been
refactored to pass through the value of the API rate in ms to botocore.
ScoutSuite is included and has simply been hardcoded to apply the queue
to ec2 clients.
This has enabled me to conduct comparative runs between this solution
and using the Network Link Conditioner. With the former completing in 30
minutes and the latter in around 2 hours.
The queue was only applied to ec2 clients because I was only
experiencing rate limiting on the scanning of the snapshots, around 15k
+
At the time of writing botocore seem to be unwilling to accept the PR
for this as they feel that it is beyond the scope of their project.
c'est la vie.
## The Solution
Rather than cry over a missed opportunity I've now taken the rate
limiter and packaged it up as a stand alone utility that anyone can
consume in their own projects should they need to avoid server-side rate
limiting:
In the diagram above, each thread needing to make an API call using
a/the ec2 client calls a method that first enqueues the call in a FIFO
queue and then waits for it to reach the head of the queue. Thereby
translating the asynchronous calls from the multiple threads into a
synchronous stream of calls at a configurable frequency.
This approach allows each thread to continue to leverage parallel
processing of tasks while only waiting on the actual API call. So you
still see the increase in efficiency of the multi threaded approach.
By instantiating individual queues for each AWS service, each can be
individually configured with an appropriate rate for the consumer's
platform, or not rate limited at all.
e.g. in my case I only had to limit the EC2 client because of the
excessive number of snapshots being scanned.
## Installation
In the usual Python fashion:
```sh
import ApiRateLimiter
```
## Usage example
See "The Solution" above or look at the tests in the GitHub repo:
[Tests on GitHub](https://github.com/museadmin/api-rate-limiter/tree/master/api-rate-limiter/tests)
### Basic Usage
Setup:
* Instantiate the rate limiter
* Start it running in a background thread
* Call the enqueue() method to join the queue
* Poll the waiting state until false
* Make your API call
On close:
* Soft stop the rate limiter – Waits for background thread to exit or timeout
#### Example
```
rate_limiter = ApiRateLimiter(100)
rate_limiter.start()
...
def some_method()
waiter = self.rate_limiter.enqueue()
while waiter.waiting is True:
pass
client.describe_instances()
...
rate_limiter.stop(True)
```
## Release History
* 0.1.0
* CHANGE: Initial code commit
* ADD LICENCE
* ADD Detailed README
* FIX Error handling in integration test
* 0.1.1
* FIX Remove requires for multiprocessing as now in Python3 main lib
## Meta
Bradley Atkins – bradley.atkinz @ gmail.com
Distributed under the MIT license. See ``LICENSE`` for more information.
[This Project on GitHub](https://github.com/museadmin/api-rate-limiter)
## Contributing
1. Fork it (<https://github.com/museadmin/api-rate-limiter>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Commit your changes (`git commit -am 'Add some fooBar'`)
4. Push to the branch (`git push origin feature/fooBar`)
5. Create a new Pull Request
6. Email me if I don't notice!
نیازمندی
| مقدار | نام |
|---|---|
| - | queue |
| - | time |
| - | threading |
نحوه نصب
نصب پکیج whl api-rate-limiter-0.1.1:
pip install api-rate-limiter-0.1.1.whl
نصب پکیج tar.gz api-rate-limiter-0.1.1:
pip install api-rate-limiter-0.1.1.tar.gz