********
Overview
********
Library to handle sparse bytes within a virtual memory space.
* Free software: BSD 2-Clause License
Objectives
==========
This library aims to provide utilities to work with a `virtual memory`, which
constsis in a virtual addressing space where sparse `chunks` of data can be
stored.
In order to be easy to use, its interface should be close to that of a
``bytearray``, which is the closest pythonic way to store dynamic data.
The main downside of a ``bytearray`` is that it requires a contiguous data
allocation starting from address 0. This is not good when sparse data have to
be stored, such as when emulating the addressing space of a generic
microcontroller.
The main idea is to provide a ``bytearray``-like class with the possibility to
internally hold the sparse `blocks` of data.
A `block` is ideally a tuple ``(start, data)`` where `start` is the start
address and `data` is the container of data items (e.g. ``bytearray``).
The length of the block is ``len(data)``.
Those blocks are usually not overlapping nor contiguous, and sorted by start
address.
Python implementation
=====================
This library provides a pure Python implementation, for maximum compatibility.
Its implementation should be correct and robust, whilst trying to be as fast
as common sense suggests. This means that the code should be reasonably
optimized for general use, while still providing features that are less likely
to be used, yet compatible with the existing Python API (e.g. ``bytearray`` or
``dict``).
The Python implementation can also exploit the capabilities of its powerful
``int`` type, so that a virtually infinite addressing space can be used,
even with negative addresses!
Data chunks are stored as common mutable ``bytearray`` objects, whose size is
limited by the Python engine (e.g. that of ``size_t``).
More details can be found within ``bytesparse.inplace``.
Cython implementation
=====================
The library also provides an experimental `Cython` implementation. It tries to
mimic the same algorithms of the Python implementation, while exploiting the
speedup of compiled `C` code.
Please refer to the ``cbytesparse`` Python package for more details.
Background
==========
This library started as a spin-off of ``hexrec.blocks.Memory``.
That is based on a simple Python implementation using immutable objects (i.e.
``tuple`` and ``bytes``). While good enough to handle common hexadecimal files,
it is totally unsuited for dynamic/interactive environments, such as emulators,
IDEs, data editors, and so on.
Instead, ``bytesparse`` should be more flexible and faster, hopefully
suitable for generic usage.
While developing the Python implementation, why not also jump on the Cython
bandwagon, which permits even faster algorithms? Moreover, Cython itself is
an interesting intermediate language, which brings to the speed of C, whilst
being close enough to Python for the like.
Too bad, one great downside is that debugging Cython-compiled code is quite an
hassle -- that is why I debugged it in a crude way I cannot even mention, and
the reason why there must be dozens of bugs hidden around there, despite the
test suite :-) Moreover, the Cython implementation is still experimental, with
some features yet to be polished (e.g. reference counting).
Documentation
=============
For the full documentation, please refer to:
https://bytesparse.readthedocs.io/
Installation
============
From PyPI (might not be the latest version found on *github*):
.. code-block:: sh
$ pip install bytesparse
From the source code root directory:
.. code-block:: sh
$ pip install .
Development
===========
To run the all the tests:
.. code-block:: sh
$ pip install tox
$ tox
Changelog
=========
0.0.5 (2022-02-22)
------------------
* Added ``bytesparse`` class, closer to ``bytearray`` than ``Memory``.
* Added missing abstract and ported methods.
* Added cut feature.
* Added more helper methods.
* Fixed values iteration.
* Improved extraction performance.
* Improved testing.
0.0.4 (2022-01-09)
------------------
* Refactored current implementation as the ``inplace`` sub-module.
* Added abstract base classes and base types into the ``base`` sub-module.
* Removed experimental backup feature.
* Added dedicated methods to backup/restore mutated state.
* Fixed some write/insert bugs.
* Fixed some trim/bound bugs.
* Methods sorted by name.
* Removed useless functions.
0.0.3 (2022-01-03)
------------------
* Using explicit factory methods instead of constructor arguments.
* Added block collapsing helper function.
* Minor fixes.
* Improved test suite.
0.0.2 (2021-12-27)
------------------
* Cython implementation moved to its own ``cbytesparse`` Python package.
* Remote testing moved to GitHub Actions.
0.0.1 (2021-04-04)
------------------
* First release on PyPI.