=======================================================
DAccuracy: Detection and Segmentation Accuracy Measures
=======================================================
.. sectnum::
:suffix: -
Brief Description
=================
``DAccuracy`` (Detection Accuracy) allows to compute
- some accuracy measures
- on an N-dimensional detection or segmentation image
- when the ground-truth is available as a `CSV file <https://en.wikipedia.org/wiki/Comma-separated_values>`_, an image, or a `Numpy <https://numpy.org/>`_ file.
It works in 3 contexts:
- one-to-one: single ground-truth, single detection image;
- one-to-many: unique ground-truth, several detection images (typically obtained by various methods);
- many-to-many: set of "(ground-truth, detection image)" pairs.
.. note::
**With 8-bit image formats**, ground-truth and detection **cannot contain more than 255 objects**. If they do, they could be saved using higher-depth formats. However, it is recommended to save them in `NPY <https://numpy.org/doc/stable/reference/generated/numpy.save.html>`_ or `NPZ <https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html>`_ Numpy formats instead. Note that using Numpy arrays does not remove the limit on the number of objects, but with the `uint64 dtype <https://numpy.org/doc/stable/reference/arrays.scalars.html#numpy.uint64>`_, the limit goes up to close to 2e19 objects. See a note below as to why ground-truth and detection are requested to be labeled (which raises this issue), as opposed to binary (which a priori would not).
Example console output (accuracy measures can also be written to a CSV file)::
Ground truth = ground-truth.csv
Detection = detection.png
N ground truths = 55
N detections = 47
True_positive = 43
False_positive = 4
False_negative = 12
Precision = 0.9148936170212766
Recall = 0.7818181818181819
F1_score = 0.8431372549019609
Froc_sample = (4, 0.7818181818181819)
Check_tp_fn_equal_gt = 55
Check_tp_fp_equal_dn = 47
Installation
============
The ``DAccuracy`` project is published on the `Python Package Index (PyPI) <https://pypi.org>`_ at: `https://pypi.org/project/daccuracy <https://pypi.org/project/daccuracy>`_. It requires version 3.8, or newer, of the interpreter. It should be installable from Python distribution platforms or Integrated Development Environments (IDEs). Otherwise, it can be installed from a command-line console:
- For all users, after acquiring administrative rights:
- First installation: ``pip install daccuracy``
- Installation update: ``pip install --upgrade daccuracy``
- For the current user (no administrative rights required):
- First installation: ``pip install --user daccuracy``
- Installation update: ``pip install --user --upgrade daccuracy``
Documentation
=============
After installation, the ``daccuracy`` command should be available from a command-line console. The usage help is obtained with ``daccuracy --help`` (see output below).
Input Formats
-------------
The ground-truth can be specified through a CSV file, a labeled image, or a labeled Numpy array. The detection can be specified through a labeled image or a labeled Numpy array. A labeled image or Numpy array must have the background labeled with zero, with the objects labeled consecutively from 1.
.. note::
The input images are requested to be labeled (as opposed to binary: zero for the background and 1 for any of the objects) in order to be able to deal with tangent objects. With a binary image, there is no way to distinguish a unique object from a set of tangent objects.
In CSV format, the ground-truth must be specified as one row per object where ``n`` columns (the first ``n`` ones by default) correspond to the row, column, and remaining ``n-2`` coordinates of the object center. Note that these coordinates can have floating-point values (as opposed to being restricted to integers). See the usage help below for details.
Example CSV::
1.2, 2.3
3.4, 4.5
This specifies two ground-truth object centers in dimension 2, the first one being at row 1.2 and column 2.3. Alternatively, the center coordinates can be passed in x/y coordinate system. See the usage help below for details.
Accuracy Measures
-----------------
The following accuracy measures are computed:
- Number of ground-truth objects
- Number of detected objects
- Number of true positives, false positives, and false negatives
- Precision, recall, and F1 score
- Free-response Receiver Operating Characteristic (FROC) curve sample: named ``froc_sample`` and corresponding to the tuple (false positives, true positive rate)
- Values for measure correctness checking: ``check_tp_fn_equal_gt`` (true_positives + false_negatives ?=? ground-truths) and ``check_tp_fp_equal_dn`` (true_positives + false_positives ?=? detections)
Additionally, if the ground-truth has been passed as an image or a Numpy array, the mean, standard deviation, minimum, and maximum of the following measures are also computed:
- Ground-truth/detection overlap (as a percentage with respect to the smaller region among ground-truth and detection)
- Ground-truth/detection Jaccard index
- Pixel-wise precision, recall, and F1 score
Output Formats
--------------
See usage help below.
Usage Help (``daccuracy --help``)
---------------------------------
Usage Help::
usage: daccuracy [-h] --gt ground_truth --dn detection [--shifts Dn_shift Dn_shift] [-e] [-t TOLERANCE] [-f {csv,nev}]
[-o Output file] [-s]
3 modes:
- one-to-one: one ground-truth (csv, image, or Numpy array) vs. one detection (image or Numpy array)
- one-to-many: one ground-truth vs. several detections (folder of detections)
- many-to-many: several ground-truths (folder of ground-truths) vs. corresponding detections (folder of detections)
In many-to-many mode, each detection file must have a counterpart ground-truth file with the same name, but not
necessarily the same extension.
With 8-bit image formats, ground-truth and detection cannot contain more than 255 objects. If they do, they could be
saved using higher-depth formats. However, it is recommended to save them in NPY or NPZ Numpy formats instead.
optional arguments:
-h, --help show this help message and exit
--gt ground_truth Ground-truth CSV file of centers or labeled image or labeled Numpy array, or ground-truth folder;
If CSV, --rAcB (or --xAyB) can be passed additionally to indicate that columns A and B contain
the centers' rows and cols, respectively (or x's and y's in x/y mode). Columns must be specified
as (possibly sequences of) uppercase letters, as is usual in spreadsheet applications. For
ground-truths of dimension "n" higher than 2, the symbol "+" must be used for the remaining
"n-2" dimensions. For example, --rAcB+C+D in dimension 4.
--relabel-gt {seq,full}
If present, this option instructs to relabel the ground-truth
sequentially.
--dn detection Detection labeled image or labeled Numpy array, or detection folder.
--relabel-gt {seq,full}
If present, this option instructs to relabel the ground-truth
sequentially.
--shifts Dn_shift [Dn_shift ...]
Vertical (row), horizontal (col), and higher dimension shifts to apply to detection. Default:
all zeroes.
-e, --exclude-border If present, this option instructs to discard objects touching image border, both in ground-truth
and detection.
-t TOLERANCE, --tol TOLERANCE, --tolerance TOLERANCE
Max ground-truth-to-detection distance to count as a hit (meant to be used when ground-truth is
a CSV file of centers). Default: zero.
-f {csv,nev}, --format {csv,nev}
nev: one "Name = Value"-row per measure; csv: one CSV-row per ground-truth/detection pairs.
Default: "nev".
-o Output file CSV file to store the computed measures or "-" for console output. Default: console output.
-s, --show-image If present, this option instructs to show an image superimposing ground-truth onto detection.
It is actually done only for 2-dimensional images.
Thanks
======
The project is developed with `PyCharm Community <https://www.jetbrains.com/pycharm>`_.
The development relies on several open-source packages (see ``install_requires`` in ``setup.py``).
The code is formatted by `Black <https://github.com/psf/black>`_, *The Uncompromising Code Formatter*.
The imports are ordered by `isort <https://github.com/timothycrosley/isort>`_... *your imports, so you don't have to*.