===========================================================
Netzob : Protocol Reverse Engineering, Modeling and Fuzzing
===========================================================
.. image:: https://travis-ci.org/netzob/netzob.svg?branch=next
:target: https://travis-ci.org/netzob/netzob
:alt: Continuous integration
.. image:: https://coveralls.io/repos/github/netzob/netzob/badge.svg?branch=next
:target: https://coveralls.io/github/netzob/netzob?branch=next
:alt: Code coverage
.. image:: https://landscape.io/github/netzob/netzob/next/landscape.svg?style=flat
:target: https://landscape.io/github/netzob/netzob/next
:alt: Code health
.. image:: https://readthedocs.org/projects/gef/badge/?version=latest
:target: https://netzob.readthedocs.org/en/latest/
:alt: Doc
.. image:: https://img.shields.io/badge/Python-3-brightgreen.svg
:target: https://github.com/netzob/netzob
:alt: Python3
.. image:: https://img.shields.io/badge/freenode-%23netzob-yellowgreen.svg
:target: https://webchat.freenode.net/?channels=#netzob
:alt: IRC
About Netzob
============
Functional Description
-----------------------
Netzob is an opensource tool for reverse engineering, traffic generation
and fuzzing of communication protocols. This tool allows to infer the message format (vocabulary)
and the state machine (grammar) of a protocol through passive and active processes.
Its objective is to bring state of art academic researches to the operational field,
by leveraging bio-informatic and grammatical inferring algorithms in a semi-automatic manner.
Netzob is suitable for reversing network protocols, structured files and system and
process flows (IPC and communication with drivers and devices).
Once inferred, a protocol model can be used in our traffic generation engine, to allow simulation of realistic
and controllable communication endpoints and flows.
Netzob handles different types of protocols: text protocols (like HTTP and IRC), delimiter-based protocols,
fixed fields protocols (like IP and TCP) and variable-length fields protocols (like TLV-based protocols).
Technical Description
---------------------
This version of Netzob must be used as a Python 3 library. It can either be imported in your scripts
or in your favorite interactive shell (ipython?).
Once installed, we recommend the following statement to import Netzob::
from netzob.all import *
Netzob's source code is mostly made of Python (90%) with some specific extensions in C (6%).
More Information
----------------
:Website: `http://www.netzob.org <http://www.netzob.org>`_
:Email: `contact@netzob.org <contact@netzob.org>`_
:Mailing list: Two lists are available, use the `SYMPA web interface <https://lists.netzob.org/wws>`_ to register.
:IRC: You can hang-out with us on Freenode's IRC channel #netzob @ freenode.org.
:Wiki: Discuss strategy on `Netzob's wiki <https://dev.netzob.org/projects/netzob/wiki>`_
:Twitter: Follow Netzob's official accounts (@Netzob)
Get Started with Netzob
=======================
Install it
----------
First thing to do is to check the version of your python3 interpretor.
Netzob requires python 3::
$ python3 --version
Python 3.4.2
As a 'classic' python project, Netzob is provided with its
``setup.py``. This file defines what and how to install the project on a
python hosting OS.
This file depends on ``setuptools`` which like few other modules cannot be
automatically installed. The reason why, you have to manually install the
following bunch of prerequisites before initiating Netzob's install process.
* python3
* python3-dev
* python3-setuptools
* build-essential
We also highly recommend to install the following additional dependencies:
* python-sphinx (for the documentation)
Once the required dependencies are installed, you can build and install Netzob::
# python3 setup.py install
Or if you prefer a more developer-friendly install::
$ python3 setup.py develop --user
Docker container
^^^^^^^^^^^^^^^^
A docker build is offered from the docker registry repository. You can download
it from command line with the following command::
$ docker pull netzob/netzob
Start it
--------
Once installed, running Netzob is as simple as executing the provided script::
$ ./netzob
This script is in Python's path if you've installed Netzob, otherwise
(in developer mode), it's located in the top distribution directory.
Docker container
^^^^^^^^^^^^^^^^
If you used the docker container, the following command will allow you to start
netzob with your current directory attached to ``/data`` into the container::
$ docker run --rm -it -v $(pwd):/data netzob/netzob
Miscellaneous
-------------
Configuration of Log Level
^^^^^^^^^^^^^^^^^^^^^^^^^^
Environment variable ```NETZOB_LOG_VERBOSITY``` can be use to set the logging level. The numeric values of logging levels are given in the Python Documentation of the `Logging Module <https://docs.python.org/3.5/library/logging.html#levels>`_. For example, the following command starts netzob in *DEBUG* mode::
$ NETZOB_LOG_LEVEL=10 ./netzob
Configuration requirements for Network and PCAP input
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Capturing data from network interfaces often requires admin privileges.
Before we provide a cleaner and secure way (see issue 425 on the bugtracker for updated information - https://dev.netzob.org/issues/425), a possible *HACK* is to provide additional capabilities to the python binary::
$ sudo setcap cap_net_raw=ep /usr/bin/python3.XX
Configuration requirements for IPC input on Ubuntu
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following command must be triggered before collecting IPC exchanges with Netzob on Ubuntu (see https://www.kernel.org/doc/Documentation/security/Yama.txt)::
$ sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"
Documentation
=============
The folder ``doc/documentation`` contains all the documentation of Netzob.
The user manual can be generated based on RST sources located in folder
``doc/documentation/source`` with the following commands::
$ sphinx-apidoc -T -e -f -o doc/documentation/source/developer_guide/API/ src/netzob/
$ find doc/documentation/source/developer_guide/API/ -type f -exec sed -i ':a;N;$!ba;s/Subpackages\n-----------\n\n.. toctree::\n/Subpackages\n-----------\n\n.. toctree::\n :maxdepth: 1\n /g' {} +
$ sphinx-build -b html doc/documentation/source/ doc/documentation/build/
An up-to-date version of the documentation is hosted on the `Read The Docs platform <https://netzob.readthedocs.org>`_.
Contributing
============
There are multiple ways to help-us.
Defects and Features Requests
------------------------------
Help-us by reporting bugs and requesting features using the `Bug Tracker <https://github.com/netzob/netzob/issues>`_.
Join the Development Team
-------------------------
To participate in the development, you need to get the latest version,
modify it and submit your changes.
These operations are detailed on Netzob's wiki through the following
pages:
* `Accessing and using Git Repositories for Netzob development <https://dev.netzob.org/projects/netzob/wiki/Accessing_and_using_Git_Repositories_for_Netzob_development>`_
* `First steps for a new developer <https://dev.netzob.org/projects/netzob/wiki/First_steps_for_a_new_developer>`_
You're interested in joining, please contact-us !
Authors, Contributors and Sponsors
==================================
See the top distribution file ``AUTHORS.txt`` for the detailed and updated list
of authors, contributors and sponsors.
Licenses
========
This software is provided under the GPLv3 License. See the ``COPYING.txt`` file
in the top distribution directory for the full license text.
The documentation is under the CC-BY-SA licence.
Extra
=====
.. figure:: https://raw.githubusercontent.com/netzob/netzob/next/netzob/doc/documentation/source/zoby.png
:width: 200 px
:alt: Zoby, the official mascot of Netzob
:align: center
Zoby, the official mascot of Netzob.
NEWS
====
v1.0.2 -- 2016-04-30
--------------------
:Version name: StompingFrilledShark
* major improvement
* global improvement of code architecture and organization
* enable custom operations on fields values
* a list of valid bytes can be specified to specify a subset of values a domain accepts
* add SendReceived() method in all channels to simplify trafic generation
* add traffic rate and duration constraints in channels and abstraction layer
* new channel 'RawEthernetClient' to send raw Ethernet frames
* new channel 'IPClient' to send raw IP frames
* the Protocol class can load ZDL format and automata files
* introduces Pseudo Field: a field used in the computation of another field but dont produce real content
* minor improvement
* pcap importer can merge consecutives messages with same source and destination to mimic a flow
* improve unit-tests on relationship finder
* an AbstractMessage stores the type of message
* remove typechecking in channel write() methods to allow better performances
* presets can be used when using AbstractionLayer
* increase default maximum data size to 65535 * 8 (i.e. 65535 bytes)
* major bug fix
* fix symbol display when messages includes exotic encoded bytes
* fix InternetChecksum computation
* fix size field computation to allow payload of size > 23535 bytes
* fix size field computation that depends on Repeat variable
* fix InternetChecksum CRC computation to support bitarrays structures
* minor bug fix
* remove nbUnits from Integer class
* remove the layer attribute of Fields
* fix the identification of data-type relationships
* convert snippets in the documentation to python3
v1.0.1 -- 2017-03-05
--------------------
:Version name: StompingFrilledShark
* minor improvement
* Improves code style (mostly PEP8 constraints)
* File Importer
* On-demand performance tests for parallel alignment (de)activation
* Ensures determinism of ClusterBySize clustering method
* Base64 and Zlib Encoding functions
* minor bug fix
* Updates Copyright dates in source headers
* Removes obsolete Rep module
v1.0 -- 2017-02-03
------------------
:Version name: StompingFrilledShark
* major improvement
* "FlowParser" to parse a succession of message made of different symbols
* large test campaign created to ensure netzob's quality
* migration to python3
* add SSL client and server channels
* improve package hierarchy
* minor improvement
* enable travis supervision
* measure test coverage
* Entropy measurement methods
* support for timestamps detection
* improve size field detection algorithm
* enable RAWIP channels
* network pcap importer supports ICMP message
* major bug fix
* various bug fixes
* multiple bug fixes in C alignment methods
* minor bug fix
* fix Symbol comparison methods
* fix logging verbosity
* remove useless codes and resources
v0.4.1 -- 2013-02-02
--------------------
:Version name: WaddlingPeccary
* Export plugins
* Automatic generation of Wireshark dissectors
* Automatic generation of Peach fuzzers
* Workspaces and projects
* Workspace manager
* Project manager
* Trace manager
* Pretty print of XML files
* Simplify the default Variable
* Provide extra compile arguments to the build process
v0.4 -- 2012-11-15
------------------
:Version name: JumpingRhino
* User interface
* New user-friendly graphical interface
* Port Netzob to GTK+3
* Allow specification of logging level in the UI
* New plugin architecture
* Internationalization of Netzob
* Vocabulary inference
* Support of layers
* Support customized transformation functions
* Provide the edition of a variable
* Support IPv4, MAC and random binary variables
* Support filters for displayed messages
* Allow export of a selection of fields as a new symbol
* Import
* Importer for OSpy projects
* Allow user to specify the import layer (2,3 or 4) while importing network messages
* Allow to keep delimiter while file importing. Indicate the position of the delimiter
* Automatic Bug Reporter
v0.3.3 -- 2012-06-06
--------------------
:Version name: FlyingRazorback
* Graphical interface
* Visualization and encoding filters
* Mathematical filters (Base64, GZIP, BZ2)
* Dedicated Search View
* Preview of data rendering in contextual menu
* Support format visualization at the symbol level
* Partitioning
* Alignment and sequencing by field
* Execute alignment on specified symbols
* Split field by the right
* Allow the partitioning of messages with specified boundaries
* Allow partitioning at the project and symbol level
* Similarity score based on number of common dynamic elements
* Optimization of Needleman : don't repeat the same computation twice
* Implement native UPGMA algorithm
* Grammar inference
* Infer the grammar of a network client
* Project/trace management
* Export / Import projects
* Importer for XML formated traces
v0.3.2 -- 2012-02-23
--------------------
* Upgrade Vocabulary Inference
* Add Octal visualization
* Feature #57: Resize columns
* Feature #59: Allows to copy message/field to clipboard
* Feature #60: Support simple alignment
* Feature #62: Allow the deletion of multiple messages at a time
* Feature #20: Show the current status of an alignment
* Manual modification of the Regex of a field
* Upgrade Grammar Inference
* Feature #55: Dedicated GUI for the automatic inferring process
* Upgrade Simulator
* Feature #87: Specify source port for network simulator
* Upgrade Import/Export and Traces Management
* Feature #22: Activate the management of traces
* Feature #61: Traces must be compressed when stored in the trace manager
* Feature #92: Handle cooked socket (SLL) packet format
* Feature #83: Support of human readable format export
* Support Unicode for filenames
* Extra
* Workspace can be specified through a command line argument
* Feature #73: Add manpage for Netzob
* Feature #74: Add ".desktop" file in the official version
* Apply pep8 quality repository on source code
v0.3.1 -- 2012-01-12
--------------------
* Small fixes
v0.3 -- 2012-01-12
------------------
* Upgraded GUI and user experience
* Add a menu
* Simplify the Vocabulary inference panel
* Add Workspaces and Projects definitions
* Upgrade Vocabulary Inference
* Add alignment based on an arbitrary delimitor
* Identification of the definition domain of a field
* Add support for environmental dependencies
* Add new visualization of data encoding
* Format: hex, string and binary
* Unit size: bit, 8-bits, 16-bits, 32-bits and 64-bits
* Sign: signed and unsigned
* Endianess: big and little endian
* Add concept of variable :
* Include Binary Value
* Include Word Value
* Include Aggregate Value
* Include Alternate Value
* Add grammar inference module
* Add the definition of the MMSTD model
* Implementation of the Angluin L* algorithm
* Implementation of the W-Method Algorithm
* Add an alpha version of the automatic inferring process
* Add simulation module
* Supports Network Server and Client simulations
* Add import modules : files and library calls
* Add multiple files import
* Extra
* SVN to GIT migration
* Dedicated website (http://www.netzob.org)
v0.2 -- 2011-09-01
------------------
* Add import modules : IPC, PCAP and Live network flows
* Add export module : raw XML format
* Improvement of Needleman and Wunsh performance with OpenMP
v0.1 -- 2011-08-16
------------------
* Initial release