==============================================
Tendril Frame-based Network Connection Tracker
==============================================
Tendril is a network communication library based on two main features:
it is based on sending and receiving frames, and it tracks the state
of an abstract connection as defined by the application. Tendril is
designed to be easy to use: creating an application requires
subclassing the ``Application`` class and providing an implementation
for the recv_frame() method; then get a ``TendrilManager`` class
instance and start it, and Tendril manages the rest.
Tendril Concepts
================
Frames
------
Tendril is based on the concept of passing around frames or packets of
data. The fact is, most network protocols are based on sending and
receiving frames; for instance, in the SMTP protocol used for sending
email, the sender will start off sending the frame "MAIL FROM
email@example.com" followed by a line termination sequence (a carriage
return followed by a newline). The SMTP server will then respond with
another frame acknowledging the "MAIL FROM" frame, and that frame will
also end with a line termination sequence. Thus, even though SMTP is
defined on top of the TCP protocol, which provides an undivided stream
of data between the client and server, a framing boundary is imposed
upon it--in this case, the carriage return followed by a newline that
terminates each frame.
Tendril includes the concept of *framers*. A framer is nothing more
than a subclass of ``Framer`` which has one method which extracts a
single frame from the stream of undifferentiated data, and another
method which converts a frame into an appropriate representation. In
the case of the SMTP protocol exchange above, the ``frameify()``
method finds each line terminated by the carriage return-newline pair,
strips off those characters, and returns just the frame. In the same
way, the corresponding ``streamify()`` method takes the frame and
appends a carriage return-newline pair.
For text-based protocols such as SMTP, this may seem like overkill.
However, for binary-based protocols, a lot of code is dedicated to
determining the boundaries between frames, and in some cases even
decoding the frame. Tendril's concept of a framer for a connection
enables the framing logic to be isolated from the rest of the
application, and even reused: Tendril comes with several pre-built
framers, including framers designed to work with a text-based protocol
such as SMTP.
Another important advantage of the framer concept is the ability to
switch between framers as needed. Taking again the example of the
SMTP protocol--the actual email data is transferred to the server by
the client first sending a "DATA" frame; the server responds
indicating that it is ready to begin receiving the message data, and
then the client simply sends the message data, ending it with a line
containing only a single period ("."). In this case, an SMTP server
application based on Tendril may wish to receive the message data as a
single frame; it can do this by creating a framer which buffers stream
data until it sees that ending sentinel (the period on a line by
itself), then returns the whole message as a single frame. Once the
server receives the "DATA" frame from the client, all it has to do is
temporarily switch out the framer in use for the receiving side of the
connection, then switch it back to the standard line-based framer once
it has received the message frame.
Tendril allows for different framers to be used on the receiving side
and sending side of the connection. This could be used in a case like
the SMTP server example cited above, where the server still wishes to
send line-oriented frames to the client, even while buffering a
message data frame. In addition, although the provided framers deal
with byte data, Tendril itself treats the frames as opaque;
applications can use this to build a framer that additionally parses a
given frame into a class object that the rest of the application then
processes as necessary.
Connection Tracking
-------------------
Tendril is also based on the concept of tracking connection state.
For connection-oriented protocols such as TCP, obviously, this is not
a big problem; however, Tendril is also designed to support
connectionless protocols such as UDP, where some applications need to
manage state information relevant to a given exchange. As an
admittedly contrived example, consider DNS, which is based on UDP. A
client of the DNS system will send a request to a DNS server over UDP;
when a response is received from that DNS server, the connection state
information tracked by Tendril can help connect that response with the
appropriate request, ensuring that the response goes to the right
place.
This connection state tracking is primarily intended to assist
applications which desire to be available over both
connection-oriented protocols such as TCP and over connectionless
protocols such as UDP. Although Tendril does not address reliability
or frame ordering, its connection state tracking eases the
implementation of an application which utilizes both types of
protocols.
Extensibility
-------------
Careful readers may have noticed the use of the terms, "such as TCP"
and "such as UDP." Although Tendril only has built-in support for TCP
and UDP connections, it is possible to extend Tendril to support other
protocols. All that is required is to create subclasses of
``Tendril`` (representing an individual connection) and of
``TendrilManager`` (which accepts and creates connections and manages
any necessary socket data flows), and to register the
``TendrilManager`` subclasses as ``pkg_resources`` entry points under
the ``tendril.manager`` namespace. See the ``setup.py`` for Tendril
for an example of how this may be done.
In addition to allowing Tendril to support protocols other than TCP
and UDP, it is also possible to implement new framers by subclassing
the ``Framer`` class. (Note: as Tendril deals with ``Framer``
objects, it is not necessary to register these framers using
``pkg_resources`` entry points.) Objects of these classes may then
simply be assigned to the appropriate ``framers`` attribute on the
``Tendril`` instance representing the connection.
Advanced Interfaces
-------------------
Tendril also provides an advanced interface that allows a given raw
socket to be "wrapped." Using this feature, an ordinary TCP socket
could be converted into an SSL socket. Other uses for this interface
are possible, such as setting socket options for the socket. Tendril
also provides an interface to allow multiple of these wrapper
functions to be called in a given order.
Standard Usage
==============
The first step in using Tendril is to define an application by
subclassing the ``Application`` class. (Subclassing is not strictly
necessary--Tendril uses Python's standard ``abc`` package for defining
abstract base classes--but using subclassing will pull in a few
helpful and/or required methods.) The subclass need merely implement
the recv_frame() method, which will be called when a frame is
received. The ``Application`` subclass constructor itself can be the
*acceptor* to be used by Tendril (more on acceptors in a moment).
Once the ``Application`` subclass has been created, the developer then
needs to get a ``TendrilManager`` instance, using the
``get_manager()`` factory function. The exact call to
``get_manager()`` depends on the needs; for making outgoing
connections, simply calling ``get_manager("tcp")`` is sufficient. If
listening on a port or making an outgoing connection from a specific
address and/or port is desired, the second argument to
``get_manager()`` may be a tuple of the desired local IP address and
the port number (i.e., ``("127.0.0.1", 80)``).
All managers must be started, and ``get_manager()`` does not start the
manager by itself. Check the manager's ``running`` attribute to see
if the manager is already running, and if it is not, call its
``start()`` method. To accept connections, pass ``start()`` the
*acceptor* (usually the ``Application`` subclass). The ``start()``
method also accepts a *wrapper*, which will be called with the
listening socket when it is created.
If, instead of accepting connections (as a server would do), the
desire is to make outgoing connections, simply call ``start()`` with
no arguments, then call the ``connect()`` method of the manager. This
method takes the *target* of the connection (i.e., the IP address and
port number, as a tuple) and the *acceptor*. (It also has an optional
*wrapper*, which will be called with the outgoing socket just prior to
initiating the connection.)
Acceptors
---------
An *acceptor* is simply a callable taking a single argument--the
``Tendril`` instance representing the connection--and returning an
instance of a subclass of ``Application``, which will be assigned to
the ``application`` attribute of the ``Tendril`` instance. The
acceptor initializes the application; it also has the opportunity to
manipulate that ``Tendril``, such as setting framers, calling the
``Tendril`` instance's ``wrap()`` method, or simply closing the
connection.
Although the ``TendrilManager`` does not provide the opportunity to
pass arguments to the acceptor, it is certainly possible to do so.
The standard Python ``functools.partial()`` is one obvious interface,
but Tendril additionally provides its own ``TendrilPartial`` utility;
the advantage of ``TendrilPartial`` is that the positional argument
passed to the acceptor--the ``Tendril`` instance--will be the first
positional argument, rather than the last one, as would be the case
with ``functools.partial()``.
Wrappers
--------
A *wrapper* is simply a callable again taking a single argument--in
this case, the socket object--and returning a wrapped version of that
argument; that wrapped version of the socket will then be used in
subsequent network calls. A wrapper which manipulates socket options
can simply return the socket object which was passed in, while one
which performs SSL encapsulation can return the SSL wrapper. Again,
although there is no opportunity to pass arguments to the wrapper in a
manager ``start()`` or ``connect()`` call (or a ``Tendril`` object's
``wrap()`` call), ``functools.partial()`` or Tendril's
``TendrilPartial`` utility can be used. In particular, in conjunction
with ``TendrilPartial``, the ``ssl.wrap_socket()`` call can be used as
a socket wrapper directly, enabling an SSL connection to be set up
easily.
Of course, it may be necessary to perform multiple "wrapping"
activities on a connection, such as setting socket options followed by
wrapping the socket in an SSL connection. For this case, Tendril
provides the ``WrapperChain``; it can be initialized in the same way
that ``TendrilPartial`` is, but additional wrappers can be added by
calling the ``chain()`` method; when called, the ``WrapperChain``
object will call each wrapper in the order defined, returning the
final wrapped socket in the end.