An interface for using plain SQL, in files.
=============================================
Does writing complex queries in an ORM feel like driving with the handbrake on?
Embrace SQL! Put your SQL queries in regular ``.sql`` files, and embrace will
load them.
Installation::
pip install embrace
Usage::
import embrace
# Connect to your database, using any db-api connector.
# If python supports it, so does embrace.
conn = psycopg2.connect("postgresql:///mydb")
# Create a module populated with queries from a collection of *.sql files:
queries = embrace.module("resources/sql")
# Run a query
users = queries.list_users(conn, order_by='created_at')
Add ``resources/sql/list_users.sql`` containing an SQL query::
-- :name list_users :many
select * from users where active = :active order by :identifier:order_by
What is the format of a query SQL file?
----------------------------------------
Embrace-SQL tries to stick close to the format used by HugSQL and PugSQL.
SQL files contain special comments to specify the query name and result type.
::
-- :name get_user_count
-- :result :scalar
SELECT count(1) FROM users
If a result type is omitted, it will default to ``cursor``. The result type
can be included in the same line as the name:
::
-- :name get_user_count :scalar
If ``:name`` is omitted, it will default to the filename without extension.
A single file may contain multiple SQL queries, separated by a structured SQL
comment. For example to create two query objects accessible as
``queries.list_users()`` and ``queries.get_user_by_id()``:
::
-- :name list_users :many
select * from users
-- :name get_user_by_id :one
select * from users where id=:id
But if you *don't* have the separating comment, embrace-sql can run
multiple statements in a single query call, returning the result from just the last one.
Why? Because it makes this possible in MySQL:
::
-- :result :column
insert into users (name, email) values (:name, :email);
select last_insert_id();
Including queries
`````````````````
You may include one query inside another using the ``:include:`` directive. For
example:
::
-- :name select_valid_users
SELECT * FROM users WHERE deleted_at IS NULL;
-- :name select_user_by_email
SELECT * FROM (:include:select_valid_users) WHERE email = :email;
What can queries return?
------------------------------
The following result types are supported:
========================= ======================================================
``:affected``, ``:n`` The number of rows affected
``:first`` The first row, as returned by ``cursor.fetchone()``,
or ``None`` if no row is found.
``:one``, ``:1`` A single row, as returned by ``cursor.fetchone()``,
usually as a tuple (but most db-api modules have
extensions allowing you to access rows as dicts or
named tuples.
If no row is generated by the query,
``embrace.exceptions.NoResultFound`` will be raised.
If more than one row is generated by the query,
``embrace.exceptions.MultipleResultsFound`` will be
raised.
``exactly-one`, ``:=1`` Synonyms for ``:one``, retained for compatibility
``:one-or-none`` As ``one``, but returns None if no row is returned by
the query.
``:many``, ``:*`` An iterator over a number of rows. Each row will be
the value returned by ``cursor.fetchone()``, usually
a tuple.
``:cursor``, ``:raw`` The cursor object.
``:scalar`` The value of the first column of the first row
returned by the query.
If no row is generated by the query, a
``NoResultFound`` will be raised.
``:column`` An iterator over the values in the first column
returned.
``:resultset`` An object supporting access to query results as any of
the above result types.
**This is the default result type if no result type is
specified**
========================= ======================================================
You can override the return type specified by the query from Python code by
using one of the following methods on the ``Query`` object:
- ``affected``
- ``one``
- ``exactlyone``
- ``many``
- ``cursor``
- ``scalar``
- ``column``
- ``resultset``
Use ``resultset`` to get access to both the results and cursor metadata, for example::
result = Query("SELECT * from mytable").resultset(conn)
print(result.many())
print(result.cursor.description)
How do I return rows as dicts or named tuples?
----------------------------------------------
Queries return rows directly from the underlying db-api driver.
Many drivers have options to return data structures other than tuples (for
example ``sqlite3.Row`` or ``psycopg2.extras.DictCursor``). You will need to
configure these at the connection level.
See the next section for how to use ``embrace.query.mapobject`` to map rows
on to namedtuples, dicts or your own ORM-style model classes.
How do I map rows onto objects?
-------------------------------
Embrace supports simple ORM style mapping.
Example::
import embrace
from dataclasses import dataclass
@dataclass
class User:
id: int
name: str
query = queries.query("SELECT * from users").returning(User)
users = query.many(conn)
Map multiple classes in a single query::
query = queries.query(
"SELECT * FROM posts JOIN users ON posts.user_id = users.id"
).returning((Post, User))
for post, user in query.many(conn):
…
By default embrace looks for fields named ``id`` (case insensitive) to
split up the row.
If you need to split on different columns, use ``mapobject`` to specify how to
map the returned columns onto objects::
from embrace import mapobject
query = queries.query(
"""
SELECT posts.*, users.*
FROM posts JOIN users ON posts.user_id = users.id
"""
).returning(
(
mapobject(Post, split="post_id"),
mapobject(User, split="user_id")
)
)
for post, user in query.many(conn):
…
``mapobject`` can also load columns into dicts and namedtuples::
from embrace import mapobject
query = queries.query(
"""
SELECT posts.*, users.*
FROM posts JOIN users ON posts.user_id = users.id
"""
).returning(
(
mapobject.dict(split="post_id"),
mapobject.namedtuple(split="user_id")
)
)
for post, user in query.many(conn):
…
and pass individual columns through unchanged::
query = queries.query(
"""
SELECT posts.*, count(*) as reply_count
FROM posts JOIN replies ON posts.id = replies.post_id
"""
).returning(
(
mapobject(Post, split="post_id"),
mapobject.passthrough(split="reply_count"),
)
)
for post, reply_count in query.many(conn):
…
You can also tell embrace to populate join relationships::
from embrace import one_to_many
from embrace import one_to_one
query = queries.query(
"""
SELECT users.*, orders.*, products.*
FROM users
JOIN orders ON orders.user_id = users.id
JOIN products ON orders.product_id = products.id
ORDER BY users.id, orders.id
"""
).returning(
# Each row of this query returns data for a User, Order and Product
# object. The `key` parameter tells embrace to map items with identical
# key values to the same python object.
(
mapobject(User, key="id"),
mapobject(Order, key="id"),
mapobject(Product, key="id"),
),
joins=[
# Populate User.orders with the list of Order objects
one_to_many(User, 'orders', Order),
# Populate Order.product with the product object
one_to_one(Order, 'product', Product),
],
)
for user in query.many(conn):
for order in user.order:
product = order.product
…
Note that methods like ``query.one`` operate at the level of the database
cursor.
If you use ``joins`` to consolidate multiple database rows into a single
object,
you will still need to call ``query.many`` even if you only require a
single object to be returned.
How do parameters work?
------------------------
Placeholders inserted using the ``:name`` syntax are escaped by the db-api
driver:
::
-- Outputs `select * from user where name = 'o''brien'`;
select * from users where name = :name
You can interpolate lists and tuples too:
``:tuple:`` creates a placeholder like this ``(?, ?, ?)``
``:value*:`` creates a placeholder like this ``?, ?, ?``
``:tuple*:`` creates a placeholder like this ``(?, ?, ?), (?, ?, ?), …``
(useful for multiple insert queries)
::
-- Call this with `queries.insert_foo(data=(1, 2, 3))`
INSERT INTO foo (a, b, c) VALUES :tuple:data
-- Call this with `queries.get_matching_users(names=("carolyn", "douglas"))`
SELECT * from users WHERE name in (:value*:names)
You can escape identifiers with ``:identifier:``, like this:
::
-- Outputs `select * from "some random table"`
select * from :identifier:table_name
You can pass through raw sql too. This leaves you open to SQL injection attacks if you allow user input into such parameters:
::
-- Outputs `select * from users order by name desc`
select * from users order by :raw:order_clause
How do I handle connections? Transactions?
------------------------------------------
You must pass a db-api connection object every time you call a query.
You can manage these connections yourself, but Embrace also offers a connection
pooling module.
::
from embrace import pool
# Create a connection pool
connection_pool = pool.ConnectionPool(
partial(psycopg2.connect, database='mydb'),
limit=10
)
# Example 1 - explicit calls to getconn/release
conn = connection_pool.getconn()
try:
queries.execute_some_query(conn)
finally:
connection_pool.release(conn)
# Example 2 - context manager
with connection_pool.connect() as conn:
queries.execute_some_query(conn)
Transaction handling may be handled manually by calling ``commit()`` or
``rollback()`` on the connection object, or you can also use the
``transaction`` context run to queries in a transaction:
::
with queries.transaction(conn) as q:
q.increment_counter()
The transaction will be commited when the ``with`` block exits, or rolled back
if an exception occurred.
How do I reload queries when the underlying files change?
---------------------------------------------------------
Pass auto_reload=True when constructing a module:
::
m = module('resources/sql', auto_reload=True)
Exceptions
----------
Exceptions raised from the underlying db-api connection are wrapped in
exception classes from ``embrace.exceptions``, with PEP-249 compliant names.
You can use this like so:
::
try:
queries.execute("SELECT 1.0 / 0.0")
except embrace.exceptions.DataError:
pass
The original exception is available in the ``__cause__`` attribute of the
embrace exception object.