amo2kinto
#########
.. image:: https://img.shields.io/travis/mozilla-services/amo2kinto/master.svg
:target: https://travis-ci.org/mozilla-services/amo2kinto
.. image:: https://img.shields.io/pypi/v/amo2kinto.svg
:target: https://pypi.python.org/pypi/amo2kinto
.. image:: https://coveralls.io/repos/mozilla-services/amo2kinto/badge.svg?branch=master
:target: https://coveralls.io/r/mozilla-services/amo2kinto
- **kinto2xml**: Export the blocklist in the legacy XML format
- **blockpages-generator**: Generate the blocklist documentation pages.
Installation
============
To install the release::
pip install amo2kinto
To install the development environment::
make install
System dependencies
-------------------
In order to build ``amo2kinto`` dependencies you may need the following libraries:
- Building **lxml** will require: ``libxml2-dev libxslt-dev python-dev``
Otherwise you will run into errors like:
- ``libxml/xmlversion.h: File not found``
Generate a blocklist.xml file from Kinto collections
====================================================
If you want to export blocklists stored in Kinto in the addons server XML export
format, you can use the ``kinto2xml`` script::
kinto2xml -s http://localhost:8888/v1
**kinto2xml** gives you the ability to configure what you want to
export and in which bucket and collection are the data stored.
.. code-block::
usage: kinto2xml [-h] [-s SERVER] [-a AUTH] [-v] [-q] [-D]
[--certificates-bucket CERTIFICATES_BUCKET]
[--certificates-collection CERTIFICATES_COLLECTION]
[--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
[--addons-bucket ADDONS_BUCKET]
[--addons-collection ADDONS_COLLECTION]
[--plugins-bucket PLUGINS_BUCKET]
[--plugins-collection PLUGINS_COLLECTION]
[--api-version API_VERSION] [--app APP]
[--app-version APP_VERSION] [-o OUT]
Build a blocklists.xml file from Kinto blocklists.
optional arguments:
-h, --help show this help message and exit
-s SERVER, --server SERVER
The location of the remote server (with prefix)
-a AUTH, --auth AUTH BasicAuth token:my-secret
-v, --verbose Show all messages.
-q, --quiet Show only critical errors.
-D, --debug Show all messages, including debug messages.
--certificates-bucket CERTIFICATES_BUCKET
Bucket name for certificates
--certificates-collection CERTIFICATES_COLLECTION
Collection name for certificates
--gfx-bucket GFX_BUCKET
Bucket name for gfx
--gfx-collection GFX_COLLECTION
Collection name for gfx
--addons-bucket ADDONS_BUCKET
Bucket name for addons
--addons-collection ADDONS_COLLECTION
Collection name for addon
--plugins-bucket PLUGINS_BUCKET
Bucket name for plugins
--plugins-collection PLUGINS_COLLECTION
Collection name for plugin
--api-version API_VERSION
Targeted blocklists.xml APP id
--app APP Targeted blocklists.xml APP id
--app-version APP_VERSION
The targetted app version
-o OUT, --out OUT Output XML file.
Generate blocked addons and plugins description pages
=====================================================
You might want to export the human readable list and description of
add-ons and plugins that were blocked.
You can do that using the ``blockpages-generator`` cli tool:
.. code-block::
usage: blockpages-generator [-h] [-s SERVER] [-a AUTH] [-b BUCKET] [-v] [-q]
[-D] [--addons-collection ADDONS_COLLECTION]
[--plugins-collection PLUGINS_COLLECTION]
[-d TARGET_DIR]
Generate blocked item description files.
optional arguments:
-h, --help show this help message and exit
-s SERVER, --server SERVER
The location of the remote server (with prefix)
-a AUTH, --auth AUTH BasicAuth token:my-secret
-b BUCKET, --bucket BUCKET
Bucket name.
-v, --verbose Show all messages.
-q, --quiet Show only critical errors.
-D, --debug Show all messages, including debug messages.
--addons-collection ADDONS_COLLECTION
Collection name for addon
--plugins-collection PLUGINS_COLLECTION
Collection name for plugin
-d TARGET_DIR, --target-dir TARGET_DIR
Destination directory to write files in.
It will generate an ``index.html`` file with the list of records
present in the ``addons`` and ``plugins`` collections in the
``target-dir`` directory.
It will also generate a file per ``add-on`` and ``plugin`` using the
``blockID`` or the ``id``. e.g ``i487.html`` or
``08db5018-2c80-4c4d-aa98-dafe6aacc28c.html``
CHANGELOG
#########
This document describes changes between each past release.
4.0.2 (2020-05-18)
==================
**Bug fix**
- Fix crash when ``prefs`` field is missing from addons entry
4.0.1 (2018-07-04)
==================
**Bug fix**
- Fix the XML item ID when squashing entries by addon ID
4.0.0 (2018-07-04)
==================
**Bug fix**
- Fix the affected users section (#87), thanks @rctgamer3!
**Breaking changes**
- Removed code in charge of updating the collection schema (#85)
3.2.1 (2018-02-28)
==================
- Fix bug with Python 3 and writing files (#80).
3.2.0 (2018-02-06)
==================
- Filter add-ons and plugins in v3 based on the requesting application and version. (#74)
- Stop exporting cert items to Firefox 58 and above, where they aren't used. (#75)
3.1.0 (2017-10-03)
==================
- Add support for cert items subject and pubKeyHash attributes. (#70)
3.0.0 (2017-09-14)
==================
- Remove json2kinto importer
- Remove xml verifier
2.0.1 (2017-06-02)
==================
**Bug fix**
- Fix synchronize kinto-http parameters. (#67)
2.0.0 (2017-06-02)
==================
**Breaking changes**
- Upgrade to kinto-http.py 8.0 and drop support for Python < 3.5
1.7.2 (2017-02-20)
==================
- Fix XML exporter on missing blockID. (#63)
1.7.1 (2016-11-24)
==================
**Bug fix**
- Allow the importer to work even without the permission to create collections. (#56)
- Use ``PUT`` instead of ``PATCH`` to fully overwrite the destination with the source (#58)
1.7.0 (2016-11-21)
==================
**New features**
- Retry downloading the XML in case it fails the first time. (#50)
**Bug fix**
- Remove redundant footer when generating pages. (#51)
- Allow the importer to work even without the permission to create buckets. (#53)
**Internal changes**
- Improve the way the Makefile works (#52)
1.6.0 (2016-10-04)
==================
**New features**
- Add a configurable signoff workflow to-review → to-sign (#48)
1.5.1 (2016-09-08)
==================
**Bug fix**
- session_kwargs is not accessible anymore with last kinto-http client release. (#45)
1.5.0 (2016-08-25)
==================
- Add the ``blockpages-generator`` CLI tool (#43)
1.4.1 (2016-08-02)
==================
- Fix blockID ordering to make it the same as the addons-server.
1.4.0 (2016-07-18)
==================
**New features**
- Add version ranges for GFX items (#39) â `Bug 1283601`_
.. _`Bug 1283601`: https://bugzilla.mozilla.org/show_bug.cgi?id=1283601
1.3.1 (2016-07-06)
==================
**Bug fix**
- Fix patch_records properties.
1.3.0 (2016-07-06)
==================
- Update records that were changed (#37)
1.2.1 (2016-05-26)
==================
**Bug fix**
- kinto_client.delete_records API changed. (#35)
1.2.0 (2016-05-26)
==================
**New features**
- Let people specify ``--app-version`` and ``--api-version`` while using kinto2xml (#33)
- Add functional tests for both version of the XML file format. (#33)
- Configure the kinto-admin as well as the JSONSchema (#32)
**Bug fix**
- Group addons by the biggest guid (#33)
1.1.0 (2016-05-18)
==================
**Bug fix**
- In case there is a diff using xml-verifier, the command now returns an error code (#28)
**Internal changes**
- ``json2kinto`` does not set destination collections as publicly readable (#27)
1.0.0 (2016-05-12)
==================
**Breaking changes**
- kinto2xml was renamed amo2kinto (#21)
- New JSON Schema file format with a "collection" prefix (#22)
**New features**
- Use the schema to validate AMO records in the importer script. (#5)
- Warn if the server does not have the schema capability (#24)
**Internal changes**
- Document amo2kinto commands: json2kinto, kinto2xml and verifier. (#23)
0.1.0 (2016-04-27)
==================
**Initial version**
- Create collection with the definition of the JSON schema.
- Fetch AMO blocklists information from the /blocked/blocklists.json AMO endpoint.
- Handle import configuration on the CLI.
- Bucket / Collection names
- Remote AMO endpoint configuration
- Schema file path configuration
- Schema or not schema
- Verbosity level
- Server selection
- Auth credentials
- Importation type selection
- Support for kinto-signer triggering
- Full SSL support for Python 2.7
- Full Python 2.7 and Python 3.4/3.5 support
- Handle the enabled flag to activate records
- Makefile rule to update the schema definition
- Export kinto blocklists in XML blocklist file version 2
- Export kinto blocklists in XML blocklist file version 3
- XML verifier that create a diff of two XML files