معرفی شرکت ها
cocrawler-0.1.8
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
A modern web crawler framework for Python
| ویژگی | مقدار |
|---|---|
| سیستم عامل | POSIX :: Linux |
| نام فایل | cocrawler-0.1.8 |
| نام | cocrawler |
| نسخه کتابخانه | 0.1.8 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Greg Lindahl and others |
| ایمیل نویسنده | lindahl@pbm.com |
| آدرس صفحه اصلی | https://github.com/cocrawler/cocrawler |
| آدرس اینترنتی | https://pypi.org/project/cocrawler/ |
| مجوز | Apache 2.0 |
# CoCrawler
[](https://dev.azure.com/cocrawler/cocrawler/_build/latest?definitionId=2&branchName=master) [](https://coveralls.io/github/cocrawler/cocrawler?branch=master) [](LICENSE)
CoCrawler is a versatile web crawler built using modern tools and
concurrency.
Crawling the web can be easy or hard, depending upon the details.
Mature crawlers like Nutch and Heritrix work great in many situations,
and fall short in others. Some of the most demanding crawl situations
include open-ended crawling of the whole web.
The object of this project is to create a modular crawler with
pluggable modules, capable of working well for a large variety of
crawl tasks. The core of the crawler is written in Python 3.5+ using
coroutines.
## Status
CoCrawler is pre-release, with major restructuring going on. It is
currently able to crawl at around 170 megabits / 170 pages/sec on a 4
core machine.
Screenshot: 
## Installing
We recommend that you use pyenv, because (1) CoCrawler requires Python
3.5+, and (2) requirements.txt specifies exact module versions.
```
git clone https://github.com/cocrawler/cocrawler.git
cd cocrawler
make init # will install requirements using pip
make pytest
make test_coverage
```
## Pluggable Modules
Pluggable modules make policy decisions, and use utility routines
to keep policy modules short and sweet.
An additional set of pluggable modules provide support for a variety
of databases. These databases are mostly used to orchestrate the
cooperation of multiple crawl processes, enabling the horizontal
scalability of the crawler over many cores and many nodes.
Crawled web assets are intended to be stored as WARC files, although
this interface should also pluggable.
## Ranking
Everyone knows that ranking is extremely important to search queries,
but it's also important to crawling. Crawling the most important stuff
is one of the best ways to avoid crawling too much webspam, soft 404s,
and crawler trap pages.
SEO is a multi-billion-dollar industry created to game search engine
ranking, and any crawl of a wide swath of the web is going to run into
poor-quality content attempting to appear to have high quality.
There's little chance that CoCrawler's algorithms will beat the most
sophisticated SEO techniques, but a little ranking goes a long way.
## Credits
CoCrawler draws on ideas from the Python 3.4 code in "500 Lines or
Less", which can be found at https://github.com/aosabook/500lines.
It is also heavily influenced by the experiences that Greg acquired
while working at blekko and the Internet Archive.
## License
Apache 2.0
نیازمندی
| مقدار | نام |
|---|---|
| - | uvloop |
| - | aiohttp |
| - | yarl |
| - | aiodns |
| - | pyyaml |
| - | cchardet |
| - | surt |
| - | reppy |
| - | cachetools |
| >=3 | tldextract |
| - | sortedcontainers |
| - | sortedcollections |
| - | psutil |
| - | hdrhistogram |
| - | beautifulsoup4 |
| - | lxml |
| - | extensions |
| - | warcio |
| - | geoip2 |
| - | objgraph |
| - | brotlipy |
| - | setuptools-scm |
| - | bottle |
| >=3.0.0 | pytest |
| - | pytest-cov |
| - | pytest-asyncio |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >=3.6.3 | Python |
نحوه نصب
نصب پکیج whl cocrawler-0.1.8:
pip install cocrawler-0.1.8.whl
نصب پکیج tar.gz cocrawler-0.1.8:
pip install cocrawler-0.1.8.tar.gz