=========
configgen
=========
description
===========
``configgen`` is a tool and library for generating JSON files given a script
and set of options. [#]_ it's useful when you have multiple configurations
that share some information, but change in different ways in different
contexts. define your config info in one place and generate a version specific
to your app, cron jobs, or developer utilities, in development, staging, or
production, for your dev virtual machine or server environments,
in any combination.
this is not intended to replace a full template language.
install it with ``pip``::
pip install configgen
the PyPI page is: https://pypi.python.org/pypi/configgen
the source code lives at bitbucket: https://bitbucket.org/dystedium/configgen
.. [#] this totally wasn't written just to make the simpsons reference in the
short description.
command line usage
==================
::
usage: configgen [-h] [--config CONFIG] [--module] [--option OPTION]
[--printconfig] [--squishee] [--no-replace] [--verbose]
[--version]
input [output]
generate a config file given a python module and set of options
positional arguments:
input where to get input class structure, by default this
expects a path to a file, also see help text for the
--module option
output optional, file to which output is written, no output
by default (see --printconfig)
optional arguments:
-h, --help show this help message and exit
--config CONFIG, -c CONFIG
if there are multiple Config instances in the class
structure, the value of this parameter specifies which
one to output
--module, -m interpret the input parameter as a module import with
dot notation, e.g. my.config.module, must be in python
path
--option OPTION, -o OPTION
an option string that will be used to choose which
value to output for a given field name (may be passed
multiple times)
--printconfig, -p print the converted config object to the console
--squishee, -s make the converted config output less human-readable
--no-replace, -n do not do internal string replacement in the output
--verbose, -v increase console output
--version show program's version number and exit
command line examples
=====================
**disclaimer:** all of the below examples are for illustrative purposes only.
this is not a recommendation to do things like store production usernames and
passwords in plaintext files.
basic usage
-----------
if ``hello.py`` contained the following::
from configgen import Config
cfg = Config(
_object='world', # keys starting with "_" are not output
_object__frog='ma baby', # options are separated from the key by "__"
statement='hello, {_object}', # string replacement
statement__frog__crowd='ribbit' # if multiple options are present, all must match
)
``configgen --printconfig hello.py`` would output::
{
"statement": "hello, world"
}
``configgen --printconfig --option frog hello.py`` would output::
{
"statement": "hello, ma baby"
}
put the frog in front of a crowd with ``configgen -p -o frog -o crowd hello.py``::
{
"statement": "ribbit"
}
inheritance and string replacement
----------------------------------
contents of ``inherit.py``::
from configgen import Config, KeyValue, make_multi_key as MK
# option strings
production = 'production'
# define these field name strings in one place
projectName = 'projectName'
_db = '_db'
dbType = 'dbType'
user = 'user'
password = 'password'
host = 'host'
# KeyValue objects have most of the same abilities as Config
site1=KeyValue(
projectName='site1',
_db={ # KeyValue values aren't restricted to base types
dbType:'mysql',
user:'testuser',
password:'testpass', # use your best judgment
host:'localserver'
},
braces='{{}}' # actual { or } character escaping
)
# since this is a normal python script, the usual syntax rules and patterns
# apply - parameters to KeyValue can be passed in a keyword argument dictionary
# using the ** syntax. make_multi_key(), abbreviated here as MK(),
# is a convenience function for combining keys and options - because it just
# returns the concatenated string, it can't be used when using the
# KeyValue(parameter=value) style declaration.
site2 = KeyValue(**{
projectName:'site2',
_db:KeyValue(**{ # it is possible to nest KeyValue instances
dbType:'postgresql',
user:'testuser',
password:'testpass',
MK(user, production):'realuser',
MK(password, production):'realpass', # use your best judgment
host:'localserver',
MK(host, production):'cloudserver'
})
})
# note that the string replacement here references fields inherited by
# the parent Config object. also note the use of attribute-style access
# (via the "." operator) to reference fields in dictionaries/KeyValues
siteCfg = KeyValue(
rootPath='~/www/{projectName}',
databaseUrl='{_db.dbType}://{_db.user}:{_db.password}@{_db.host}'
)
cfgSite1 = Config(inherits=site1, site=siteCfg)
cfgSite2 = Config(inherits=site2, site=siteCfg)
if multiple Config objects are defined in a file, the one to output must be
specified with the --config or -c option.
``configgen --printconfig --config cfgSite1 inherit.py`` would output::
{
"projectName": "site1",
"site": {
"databaseUrl": "mysql://testuser:testpass@localserver",
"rootPath": "~/www/site1"
},
"braces": "{}"
}
``configgen -p -c cfgSite2 inherit.py`` would output::
{
"projectName": "site2",
"site": {
"databaseUrl": "postgresql://testuser:testpass@localserver",
"rootPath": "~/www/site2"
}
}
``configgen -p -c cfgSite2 -o production inherit.py`` would output::
{
"projectName": "site2",
"site": {
"databaseUrl": "postgresql://realuser:realpass@cloudserver",
"rootPath": "~/www/site2"
}
}
notes
-----
there is currently no simple way to emit a key/value pair only for
a specific option set. there is a clunky way to do this::
common = KeyValue(key1=value1, key2=value2)
extraField = KeyValue(inherits=common, key3=value3)
output = KeyValue(fields=common, fields__addkey3=extraField)
the JSON generated by output will include key3 only when the option string
'addkey3' is present.
library examples
================
the ``configgen`` package can also be used as a library in a larger program.
the source of ``configgen.main`` is an example of basic usage.
operation
=========
``KeyValue`` string replacement
-------------------------------
when resolving named references to other parts of the KeyValue structure
during string replacement, the following steps are taken:
1. define the ``KeyValue`` instance that contains the string being resolved
as "nearest"
2. start searching at the following ``KeyValue`` instances for the entire
reference, continuing to the next one if the reference cannot be resolved:
a. the nearest instance
b. the nearest instance's inherited fields, if present
c. the outermost ``KeyValue`` instance (usually the ``Config`` instance)
d. the outermost instance's inherited fields, if present
3. while searching, if a named field is found in a ``KeyValue`` instance,
define that instance as "nearest" (overwriting any previous value)
4. if not found, emit an error, otherwise, if the replaced value is a string,
use the new nearest instance and begin a new string replacement operation
(allowing replaced strings to contain string replacement directives
themselves)
TODO: create examples, for now, `the test.py script
<https://bitbucket.org/dystedium/configgen/src/tip/scripts/test.py>`_ has
some barely-commented examples
building ``MultiValue`` sets
----------------------------
when adding multiple keys with the same base name and options separated by
"__", those keys are grouped and become a ``MultiValue``. only one value
will be emitted into the JSON, selected based on the set of options provided
to the conversion function. for now, a ``MultiValue`` set must contain a
default value, which is output when no options are provided or there is no
matching option set. any combination of the default value and keys for
specific option sets may be split across a ``KeyValue`` instance and the
instance it inherits from. some examples::
from configgen import Config, KeyValue, make_multi_key as MK
# keys
multiValue0 = 'multiValue0'
multiValue1 = 'multiValue1'
multiValue2 = 'multiValue2'
# options
one = 'one'
two = 'two'
three = 'three'
# the make_multi_key() convenience function (imported here as "MK") helps
# define MultiValues in a KeyValue by joining all the string parameters with
# the option separator.
defaults = KeyValue(**{
multiValue0:'multiValue0 inherited, default',
MK(multiValue0, one):'multiValue0 inherited, options: one',
})
# MultiValue keys can come from the inherited object. when this happens,
# all of the relevant keys are copied to the inheriting KeyValue instance
# during construction and combined into a local MultiValue instance, so
# overrides from one KeyValue instance will not affect another instance
# that inherits from the same object. if a key is defined with the same
# option set in both the inheriting and inherited KeyValue instances, the
# one in the inheriting instance is used.
# note that section0 does not define a default for multiValue0 - it's
# inherited from the defaults object.
section0 = KeyValue(inherits=defaults, **{
MK(multiValue0, one):'multiValue0 section0, options: one',
MK(multiValue0, two, three):'multiValue0 section0, options: two, three',
multiValue1:'multiValue1 section0, default',
MK(multiValue1, three):'multiValue1 section0, options: three',
#MK(multiValue2, two):'defining this by itself is an error (no default value)'
})
# section1 overrides only the default value for multiValue0. section1.multiValue2
# shows that the value type need not be the same across all of a MultiValue.
# section1.multiValue2 also demonstrates a case where multiple values have an
# option set of the same cardinality. as long as the options "one" and "two"
# are mutually exclusive, this will not cause an error.
section1 = KeyValue(inherits=defaults, **{
multiValue0:'multiValue0 section1, default',
multiValue1:'multiValue1 section1, simple value',
multiValue2:None,
MK(multiValue2, one):{'info':'options: one'},
MK(multiValue2, two):{'info':'options: two'},
})
cfg = Config(section0=section0, section1=section1)
selecting a specific value in a ``MultiValue``
----------------------------------------------
from the command line, specify the ``--option``/``-o`` option one or more times
to build a set of options to use when generating JSON output (the *generation
set*). when using the ``configgen`` library directly, a python ``frozenset``
of options should be passed to the ``Config.convertToJson()`` function as the
generation set. ``configgen`` uses the following criteria when selecting a
value, in order:
1. a value whose option set matches the generation set exactly
2. the value whose option set is the largest common subset of the generation
set (ignoring any option sets that are disjoint)
3. the default value is used if no value's option set is a subset of the
generation set
note that because the values are selected based on set intersections, neither
the order of the options specified in the MultiValue key nor the command line
affect the selection process. if there is a tie for the largest common option
subset between values, a ``configgen.LookupError`` *may* be raised. this
error can be avoided in at least two ways:
* the generation set matches one of the values' option sets exactly
* one or more of the options in the tied values' option sets are
mutually exclusive
for example, given a MultiValue containing values with these option sets
(excluding the default value):
A. ("server", "logToFile", "production")
B. ("server", "verbose", "staging")
C. ("server", "logToFile", "staging")
here are example cases:
* ("server", "logToFile", "production"): exact match for value A, use that one
* ("server", "verbose", "logToFile", "staging"): error, values B and C both
match 3 items in the generation set
* as long as "production" and "staging" never appear in the same generation
set alongside "server" and "logToFile", values A and C will not conflict