معرفی شرکت ها
dlt-0.2.6a1
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
DLT is an open-source python-native scalable data loading framework that does not require any devops efforts to run.
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | dlt-0.2.6a1 |
| نام | dlt |
| نسخه کتابخانه | 0.2.6a1 |
| نگهدارنده | ['Marcin Rudolf'] |
| ایمیل نگهدارنده | ['marcin@dlthub.com'] |
| نویسنده | dltHub Inc. |
| ایمیل نویسنده | services@dlthub.com |
| آدرس صفحه اصلی | https://github.com/dlt-hub |
| آدرس اینترنتی | https://pypi.org/project/dlt/ |
| مجوز | Apache-2.0 |
<p align="center">
[](https://pypi.org/project/python-dlt/)
[](https://github.com/dlt-hub/dlt/actions/workflows/lint.yml)
[](https://github.com/dlt-hub/dlt/actions/workflows/test_common.yml)
[](https://github.com/dlt-hub/dlt/actions/workflows/test_destinations.yml)
[](https://github.com/dlt-hub/dlt/actions/workflows/test_destination_bigquery.yml)
[](https://github.com/dlt-hub/dlt/actions/workflows/test_dbt_runner.yml)
</p>
# data load tool (dlt)
**[Colab Demo](https://colab.research.google.com/drive/1BXvma_9R9MX8p_iSvHE4ebg90sUroty2)**
```python
import dlt
from chess import chess # a utility function that grabs data from the chess.com API
# create a dlt pipeline that will load chess game data to the DuckDB destination
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='duckdb',
dataset_name='games_data'
)
# use chess.com API to grab data about a few players
data = chess(['magnuscarlsen', 'rpragchess'], start_month='2022/11', end_month='2022/12')
# extract, normalize, and load the data
pipeline.run(data)
```
**data load tool (dlt)** is a simple, open source Python library that makes data loading easy
- Automatically turn the JSON returned by any API into a live dataset stored wherever you want it
- `pip install python-dlt` and then include `import dlt` to use it in your Python loading script
- The **dlt** library is licensed under the Apache License 2.0, so you can use it for free forever
Read more about it on the [dlt Docs](https://dlthub.com/docs)
# semantic versioning
`python-dlt` will follow the semantic versioning with [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern. Currently we do **pre-release versioning** with major version being 0.
- `minor` version change means breaking changes
- `patch` version change means new features that should be backward compatible
- any suffix change ie. `a10` -> `a11` is a patch
# development
`python-dlt` uses `poetry` to manage, build and version the package. It also uses `make` to automate tasks. To start
```sh
make install-poetry # will install poetry, to be run outside virtualenv
```
then
```sh
make dev # will install all deps including dev
```
Executing `poetry shell` and working in it is very convenient at this moment.
## python version
Use python 3.8 for development which is the lowest supported version for `python-dlt`. You'll need `distutils` and `venv`:
```shell
sudo apt-get install python3.8
sudo apt-get install python3.8-distutils
sudo apt install python3.8-venv
```
You may also use `pyenv` as [poetry](https://python-poetry.org/docs/managing-environments/) suggests.
## bumping version
Please use `poetry version prerelease` to bump patch and then `make build-library` to apply changes. The source of the version is `pyproject.toml` and we use poetry to manage it.
## testing and linting
`python-dlt` uses `mypy` and `flake8` with several plugins for linting. We do not reorder imports or reformat code.
`pytest` is used as test harness. `make test-common` will run tests of common components and does not require any external resources.
### testing destinations
To test destinations use `make test`. You will need following external resources
1. `BigQuery` project
2. `Redshift` cluster
3. `Postgres` instance. You can find a docker compose for postgres instance [here](tests/load/postgres/docker-compose.yml). When run the instance is configured to work with the tests.
```shell
cd tests/load/postgres/
docker-compose up --build -d
```
See `tests/.example.env` for the expected environment variables and command line example to run the tests. Then create `tests/.env` from it. You configure the tests as you would configure the dlt pipeline.
We'll provide you with access to the resources above if you wish to test locally.
To test local destinations (`duckdb` and `postgres`) run `make test-local`. You can run this tests without additional credentials (just copy `.example.env` into `.env`)
## publishing
1. Make sure that you are on `devel` branch and you have the newest code that passed all tests on CI.
2. Verify the current version with `poetry version`
3. You'll need `pypi` access token and use `poetry config pypi-token.pypi your-api-token` then
```
make publish-library
```
4. Make a release on github, use version and git tag as release name
## contributing
To contribute via pull request:
1. Create an issue with your idea for a feature etc.
2. Write your code and tests
3. Lint your code with `make lint`. Test the common modules with `make test-common`
4. If you work on a destination code then contact us to get access to test destinations
5. Create a pull request
نیازمندی
| مقدار | نام |
|---|---|
| >=2.26.0 | requests |
| >=2.1.2 | pendulum |
| >=3.17.5 | simplejson |
| >=5.4.1 | PyYAML |
| ==1.4.1rc0 | json-logging |
| >=0.11.0 | prometheus-client |
| >=2.13.0 | semver |
| >=1.4.3 | sentry-sdk |
| >=0.2.2 | hexbytes |
| >=5.2.0 | cachetools |
| >=2022.1 | tzdata |
| >=0.11.3 | tomlkit |
| >=3.10.5 | asyncstdlib |
| >=2.5.2 | pathvalidate |
| >=1.4.0 | SQLAlchemy |
| >=4.0.0 | typing-extensions |
| >=1.15.0 | makefun |
| >=7.1 | click |
| >=0.5.0 | requirements-parser |
| >=65.6.0 | setuptools |
| >=4.4.0 | humanize |
| >=1.6.3 | astunparse |
| >=1.2.32 | cron-descriptor |
| >=2.3.3 | pipdeptree |
| >=3.1.29 | gitpython |
| >=2022.6 | pytz |
| >=4.64.1 | tqdm |
| >=0.10.0 | giturlparse |
| >=3.8.6,<4.0.0) | orjson |
| >=2.9.1) | psycopg2-binary |
| >=2.9.0) | psycopg2cffi |
| >=1.50.0) | grpcio |
| >=2.26.0) | google-cloud-bigquery |
| >=8.0.0) | pyarrow |
| >=0.6.1,<0.8.0) | duckdb |
| >=1.3.0,<1.5.0) | dbt-core |
| >=1.3.0,<1.5.0) | dbt-redshift |
| >=1.3.0,<1.5.0) | dbt-bigquery |
| >=1.3.0,<1.5.0) | dbt-duckdb |
| >=8.2.2,<9.0.0 | tenacity |
| >=1.5.3,<2.0.0 | jsonpath-ng |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >=3.8,<4.0 | Python |
نحوه نصب
نصب پکیج whl dlt-0.2.6a1:
pip install dlt-0.2.6a1.whl
نصب پکیج tar.gz dlt-0.2.6a1:
pip install dlt-0.2.6a1.tar.gz