معرفی شرکت ها
ctodd-python-lib-avro-1.0.5
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Python utilities used for interacting with .avro/.avsc files
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | ctodd-python-lib-avro-1.0.5 |
| نام | ctodd-python-lib-avro |
| نسخه کتابخانه | 1.0.5 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Christopher H. Todd |
| ایمیل نویسنده | Christopher.Hayden.Todd@gmail.com |
| آدرس صفحه اصلی | https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro |
| آدرس اینترنتی | https://pypi.org/project/ctodd-python-lib-avro/ |
| مجوز | MIT |
# Christopher H. Todd's Python Lib for AVRO/AVSC
The ctodd-python-lib-avro project is responsible for interacting with [Apache AVRO](https://avro.apache.org/docs/1.8.2/gettingstartedpython.html). This includes converting to byte arrays and from byte arrays, writing and reading .avro files, writing and reading .avsc files, and other minor quality of life wrappers.
The library relies on Python's avro-python3 package, and is wrapped with custom/specific exception handling, simpler interactions, and a more functional style to reduce code in projects dealing with AVRO
## Table of Contents
- [Dependencies](#dependencies)
- [Libraries](#libraries)
- [Example Scripts](#example-scripts)
- [Notes](#notes)
- [TODO](#todo)
## Dependencies
### Python Packages
- avro-python3>=1.8.2
- simplejson>=3.16.0
## Libraries
### [avro_converter_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_converter_helpers.py)
This library is used to convert avro to other formats (first .json)
Functions:
```
def convert_avro_file_to_json(avro_filename, json_filename=None):
"""
Purpose:
Convert an .avro file into a .json file
Args:
avro_filename (String): Path/filename of the .avro file to convert to .json
json_filename (String): Path/filename of the .json file to generate. if none
is specified, just use the same .avro path and change the extension
Yields:
json_filename (String): Path/filename of the .json file generated
"""
```
### [avro_exceptions.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_exceptions.py)
File for holding custom exception types that will be generated by the avro_helpers libraries
Exception Types:
```
class AvroTestException(Exception):
"""
Purpose:
The AvscInvalid will be raised when reading the .avsc raises an exception
"""
```
```
class AvscInvalid(Exception):
"""
Purpose:
The AvscInvalid will be raised when reading the .avsc raises an exception
"""
```
```
class AvscNotFound(Exception):
"""
Purpose:
The AvscNotFound will be raised when trying to Read a .avsc file
that cannot be found.
"""
```
```
class AvroNotFound(Exception):
"""
Purpose:
The AvroNotFound will be raised when trying to Read a .avro file
that cannot be found.
"""
```
### [avro_general_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_general_helpers.py)
Avro General Helpers. This library is used to interact with .avro files not specificlly related to reading or writing them.
Functions:
*N/A*
### [avro_reading_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_reading_helpers.py)
Avro Reading Helpers. This library is used to aid in the task of reading .avro files
Functions:
```
def get_record_from_avro_generator(avro_filename):
"""
Purpose:
Generator of records from a .avro filename (with path in the filename)
Args:
avro_filename (String): Path/filename of the .avro file to get records from
Yields:
avro_record (Record Obj from .avro): Record read from the .avro file
"""
```
```
def get_record_from_avro_buffered(avro_filename):
"""
Purpose:
Buffered Get records from a .avro filename (with path in the filename)
Args:
avro_filename (String): Path/filename of the .avro file to get records from
Returns:
avro_records (List of Record Objs from .avro): List of Records read from
the .avro file
"""
```
### [avro_schema_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_schema_helpers.py)
Avro Schema Helpers. This library is used to interact with .avsc files
Functions:
```
def get_schema_from_avsc_file(avsc_filename):
"""
Purpose:
Get the file schema from an .avsc filename (with path in the filename)
Args:
avsc_filename (String): Path/filename of the .avsc file to get the schema from
Return:
avro_schema (AVRO Schema Object): Schema object from the avro library
"""
```
### [avro_writing_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/avro_helpers/avro_writing_helpers.py)
Avro Writing Helpers. This library is used to aid in the task of writing .avro files
Functions:
```
def write_raw_records_to_avro(raw_records, avro_filename, avro_schema):
"""
Purpose:
Write Records to .avro File
Args:
raw_records (List of Dicts): List of Recrods to Write to AVRO as Bytes
avro_filename (String): Filename and path of .avro to write
avro_schema (AVRO Schema Object): Schema object from the avro library
Returns:
N/A
"""
```
```
def serialize_data(raw_records, avro_schema):
"""
Purpose:
Serialize a record as bytes
Args:
raw_records (List of Dicts): List of Records to Serialize
avro_schema (AVRO Schema Object): Schema object from the avro library
Return:
serialized_records (List of Byte Array): Records Serialized into Byte-Array
"""
```
## Example Scripts
Example executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.
### [read_avro_file.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/example_usage/read_avro_file.py)
```
Purpose:
Read an .avro File
Steps:
- Either
- Read .avro File as Buffered List
- Read .avro File as Generator
function call:
python3 read_avsc_file.py {--avro=avro_filename}
example call:
python3 read_avsc_file.py --avro="./data/test_data.avro"
```
### [read_avsc_file.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/example_usage/read_avsc_file.py)
```
Purpose:
Read an .avsc File to get the schema
Steps:
- Read .avsc Schema
function call:
python3 read_avsc_file.py {--avsc=avsc_filename}
example call:
python3 read_avsc_file.py --avsc="./avsc/test_schema.avsc"
```
### [write_avro_file.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-avro/blob/master/example_usage/write_avro_file.py)
```
Purpose:
Write an .avro File
Steps:
- Either
- Write .avro File
function call:
python3.6 write_avro_file.py {--avro=avro_filename} \
{--avsc=avsc_filename}
example call:
python3.6 write_avro_file.py --avro="./data/generated_data.avro" \
--avsc="./avsc/test_schema.avsc"
```
## Notes
- Relies on f-string notation, which is limited to Python3.6. A refactor to remove these could allow for development with Python3.0.x through 3.5.x
## TODO
- Unittest framework in place, but lacking tests
نیازمندی
| مقدار | نام |
|---|---|
| >=1.8.2 | avro-python3 |
| >=3.16.0 | simplejson |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >3.6 | Python |
نحوه نصب
نصب پکیج whl ctodd-python-lib-avro-1.0.5:
pip install ctodd-python-lib-avro-1.0.5.whl
نصب پکیج tar.gz ctodd-python-lib-avro-1.0.5:
pip install ctodd-python-lib-avro-1.0.5.tar.gz