Techniques¶
Validate with Voluptuous¶
This example shows how to use voluptuous to validate and parse a NestedText
file and it demonstrates how to use the keymap argument from loads() or
load() to add location information to Voluptuous error messages.
The input file is the same as in the previous example, i.e. deployment settings for a web server:
debug: false
secret_key: t=)40**y&883y9gdpuw%aiig+wtc033(ui@^1ur72w#zhw3_ch
allowed_hosts:
- www.example.com
database:
engine: django.db.backends.mysql
host: db.example.com
port: 3306
user: www
webmaster_email: admin@example.com
Below is the code to parse this file. Note how the structure of the data is
specified using basic Python objects. The Coerce() function is
necessary to have Voluptuous convert string input to the given type; otherwise
it would simply check that the input matches the given type:
#!/usr/bin/env python3
import nestedtext as nt
from voluptuous import Schema, Coerce, MultipleInvalid
from inform import error, full_stop, terminate
from pprint import pprint
schema = Schema({
'debug': Coerce(bool),
'secret_key': str,
'allowed_hosts': [str],
'database': {
'engine': str,
'host': str,
'port': Coerce(int),
'user': str,
},
'webmaster_email': str,
})
try:
keymap = {}
raw = nt.load('deploy.nt', keymap=keymap)
config = schema(raw)
except nt.NestedTextError as e:
e.terminate()
except MultipleInvalid as exception:
voluptuous_error_messages = { # provide user-friendly error messages
"extra keys not allowed": ("unknown key", "key"),
"expected a dictionary": ("expected key-value pair", "value"),
}
for e in exception.errors:
msg, flag = voluptuous_error_messages.get(
e.msg, (e.msg, 'value')
)
loc = keymap[tuple(e.path)]
error(full_stop(msg), culprit=e.path, codicil=loc.as_line(flag))
terminate()
pprint(config)
This produces the same result as in the previous example.
See the PostMortem example for a more flexible approach to validating with Voluptuous.
Validate with Pydantic¶
This example shows how to use pydantic to validate and parse a NestedText file. The file in this case specifies deployment settings for a web server:
debug: false
secret_key: t=)40**y&883y9gdpuw%aiig+wtc033(ui@^1ur72w#zhw3_ch
allowed_hosts:
- www.example.com
database:
engine: django.db.backends.mysql
host: db.example.com
port: 3306
user: www
webmaster_email: admin@example.com
Below is the code to parse this file. Note that basic types like integers, strings, Booleans, and lists are specified using standard type annotations. Dictionaries with specific keys are represented by model classes, and it is possible to reference one model from within another. Pydantic also has built-in support for validating email addresses, which we can take advantage of here:
#!/usr/bin/env python3
import nestedtext as nt
from pydantic import BaseModel, EmailStr
from typing import List
from pprint import pprint
class Database(BaseModel):
engine: str
host: str
port: int
user: str
class Config(BaseModel):
debug: bool
secret_key: str
allowed_hosts: List[str]
database: Database
webmaster_email: EmailStr
obj = nt.load('deploy.nt')
config = Config.parse_obj(obj)
pprint(config.dict())
This produces the following data structure:
{'allowed_hosts': ['www.example.com'],
'database': {'engine': 'django.db.backends.mysql',
'host': 'db.example.com',
'port': 3306,
'user': 'www'},
'debug': False,
'secret_key': 't=)40**y&883y9gdpuw%aiig+wtc033(ui@^1ur72w#zhw3_ch',
'webmaster_email': 'admin@example.com'}
Normalizing Keys¶
With data files created by non-programmers it is often desirable to allow
a certain amount of flexibility in the keys. For example, you may wish to
ignore case and if you allow multi-word keys you may want to be tolerant of
extra spaces between the words. However, the end applications often needs the
keys to be specific values. It is possible to normalize the keys using
a schema, but this can interfere with error reporting. Imagine there is an
error in the value associated with a set of keys, if the keys have been changed
by the schema the keymap can no longer be used to convert the keys into a line
number for an error message. NestedText provides the normalize_key argument
to load() and loads() to address this issue. It allows you to pass
in a function that normalizes the keys before the keymap is created, releasing
the schema from that task.
The following contact look-up program demonstrates both the normalization of keys and the associated error reporting. In this case, the first level of keys contains the names of the contacts and should not be normalized. Keys at all other levels are considered keywords and so should be normalized.
#!/usr/bin/env python3
"""
Display Contact Information
Usage:
contact <name>...
"""
from docopt import docopt
from inform import codicil, display, fatal, full_stop, os_error
import nestedtext as nt
from voluptuous import Schema, Required, Any, MultipleInvalid
import re
contacts_file = "address.nt"
def normalize_key(key, parent_keys):
if len(parent_keys) == 0:
return key
return '_'.join(key.lower().split())
def render_contact(data, keymap=None):
text = nt.dumps(data, map_keys=keymap)
return re.sub(r'^(\s*)[>:][ ]?(.*)$', r'\1\2', text, flags=re.M)
cmdline = docopt(__doc__)
names = cmdline['<name>']
try:
# define structure of contacts database
contacts_schema = Schema({
str: {
'position': str,
'address': str,
'phone': Required(Any({str:str},str)),
'email': Required(Any({str:str},str)),
'additional_roles': Any(list,str),
}
})
# read contacts database
contacts = contacts_schema(
nt.load(
contacts_file,
top = 'dict',
normalize_key = normalize_key,
keymap = (keymap:={})
)
)
# display requested contact information, excluding additional_roles
filtered = {}
for fullname, contact_info in contacts.items():
for name in names:
if name in fullname.lower():
filtered[fullname] = contact_info
if 'additional_roles' in contact_info:
del contact_info['additional_roles']
# display contact using normalized keys
# display(render_contact(filtered))
# display contact using original keys
display(render_contact(filtered, keymap))
except nt.NestedTextError as e:
e.report()
except MultipleInvalid as exception:
for e in exception.errors:
kind = 'key' if 'key' in e.msg else 'value'
keys = tuple(e.path)
codicil = keymap[keys].as_line(kind) if keys in keymap else None
fatal(
full_stop(e.msg),
culprit = (contacts_file, nt.join_keys(keys, keymap=keymap)),
codicil = codicil
)
except OSError as e:
fatal(os_error(e))
This program takes a name as a command line argument and prints out the corresponding address. It uses the pretty print idea from the previous section to render the contact information. Voluptuous checks the validity of the contacts database, which is shown next. Notice the variability in the keys given in Fumiko’s entry:
# Contact information for our officers
Katheryn McDaniel:
position: president
address:
> 138 Almond Street
> Topeka, Kansas 20697
phone:
cell: 1-210-555-5297
# Katheryn prefers that we call her on her cell phone
work: 1-210-555-8470
email: KateMcD@aol.com
additional roles:
- board member
Margaret Hodge:
position: vice president
address:
> 2586 Marigold Lane
> Topeka, Kansas 20682
phone: 1-470-555-0398
email: margaret.hodge@ku.edu
additional roles:
- new membership task force
- accounting task force
Fumiko Purvis:
Position: Treasurer
# Fumiko's term is ending at the end of the year.
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Additional Roles:
- accounting task force
There are two display statements near the end of the program, the first of which is commented out. The first outputs the contact information using normalized keys, and the second outputs the information using the original keys.
Now, requesting Fumiko’s contact information gives:
Fumiko Purvis:
Position: treasurer
Address:
3636 Buffalo Ave
Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Notice that any processing of the information (error checking, deleting additional_roles) is performed using the normalized keys, but by choice, the information is output using the original keys.
Duplicate Keys¶
There are occasions where it is useful to be able to read dictionaries from
NestedText that contain duplicate keys. For example, imagine that you have two
contacts with the same name, and the name is used as a key. Normally
load() and loads() throw an exception if duplicate keys are detected
because the underlying Python dictionaries cannot hold items with duplicate
keys. However, you can pass a function to the on_dup argument that
de-duplicates the keys, making them safe for Python dictionaries. For example
the following NestedText document that contains duplicate keys:
Michael Jordan:
occupation: basketball player
Michael Jordan:
occupation: actor
Michael Jordan:
occupation: football player
In the following, the de_dup function adds “#*N*” to the end of the key where N starts at 2 and increases as more duplicates are found.
#!/usr/bin/env python3
from inform import codicil, display, fatal, full_stop, os_error
import nestedtext as nt
filename = "michael_jordan.nt"
def de_dup(key, state):
if key not in state:
state[key] = 1
state[key] += 1
return f"{key}#{state[key]}"
try:
# read contacts database
data = nt.load(filename, 'dict', on_dup=de_dup, keymap=(keymap:={}))
# display contact using deduplicated keys
display("DE-DUPLICATED KEYS:")
display(nt.dumps(data))
# display contact using original keys
display()
display("ORIGINAL KEYS:")
display(nt.dumps(data, map_keys=keymap))
except nt.NestedTextError as e:
e.terminate()
except OSError as e:
fatal(os_error(e))
As shown below, this code outputs the data twice, the first time with the de-duplicated keys and the second time using the original keys. Notice that the first contains the duplication markers whereas the second does not.
With de-duplicated keys:
Michael Jordan:
occupation: basketball player
Michael Jordan#2:
occupation: actor
Michael Jordan#3:
occupation: football player
With original keys:
Michael Jordan:
occupation: basketball player
Michael Jordan:
occupation: actor
Michael Jordan:
occupation: football player
Sorting Keys¶
The default order of dictionary items in the NestedText output of dump()
and dumps() is the natural order of the underlying dictionary, but you can
use sort_keys argument to change the order. For example, here are two
different ways of sorting the address list. The first is a simple alphabetic
sort of the keys at each level, which you get by simply specifying
sort_keys=True.
>>> addresses = nt.load( 'examples/address.nt')
>>> print(nt.dumps(addresses, sort_keys=True))
Fumiko Purvis:
Additional Roles:
- accounting task force
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
EMail: fumiko.purvis@hotmail.com
Phone: 1-268-555-0280
Position: Treasurer
Katheryn McDaniel:
additional roles:
- board member
address:
> 138 Almond Street
> Topeka, Kansas 20697
email: KateMcD@aol.com
phone:
cell: 1-210-555-5297
work: 1-210-555-8470
position: president
Margaret Hodge:
additional roles:
- new membership task force
- accounting task force
address:
> 2586 Marigold Lane
> Topeka, Kansas 20682
email: margaret.hodge@ku.edu
phone: 1-470-555-0398
position: vice president
The second sorts only the first level, by last name then remaining names. It passes a function to sort_keys. That function takes two arguments, the key to be sorted and the tuple of parent keys. The key to be sorted is also a tuple that contains the key and the rendered item. The key is the key as specified in the object being dumped, and rendered item is a string that takes the form “mapped_key: value”.
The sort_keys function is expected to return a string that contains the sort key, the key used by the sort. For example, in this case a first level key “Fumiko Purvis” is mapped to “Purvis Fumiko” for the purposes of determining the sort order. At all other levels any key is mapped to “”. In this way the sort keys are all identical, and so the original order is retained.
>>> def sort_key(key, parent_keys):
... if len(parent_keys) == 0:
... # rearrange names so that last name is given first
... names = key[0].split()
... return ' '.join([names[-1]] + names[:-1])
... return '' # do not reorder lower levels
>>> print(nt.dumps(addresses, sort_keys=sort_key))
Margaret Hodge:
position: vice president
address:
> 2586 Marigold Lane
> Topeka, Kansas 20682
phone: 1-470-555-0398
email: margaret.hodge@ku.edu
additional roles:
- new membership task force
- accounting task force
Katheryn McDaniel:
position: president
address:
> 138 Almond Street
> Topeka, Kansas 20697
phone:
cell: 1-210-555-5297
work: 1-210-555-8470
email: KateMcD@aol.com
additional roles:
- board member
Fumiko Purvis:
Position: Treasurer
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Additional Roles:
- accounting task force
Key Presentation¶
When generating a NestedText document, it is sometimes desirable to transform the keys upon output. Generally one transforms the keys in order to change the presentation of the key, not the meaning. For example, you may want change its case, rearrange it (ex: swap first and last names), translate it, etc. These are done by passing a function to the map_keys argument. This function takes two arguments: the key after it has been rendered to a string and the tuple of parent keys. It is expected to return the transformed string. For example, lets print the address book again, this time with names printed with the last name first.
>>> def last_name_first(key, parent_keys):
... if len(parent_keys) == 0:
... # rearrange names so that last name is given first
... names = key.split()
... return f"{names[-1]}, {' '.join(names[:-1])}"
>>> def sort_key(key, parent_keys):
... return key if len(parent_keys) == 0 else '' # only sort first level keys
>>> print(nt.dumps(addresses, map_keys=last_name_first, sort_keys=sort_key))
Hodge, Margaret:
position: vice president
address:
> 2586 Marigold Lane
> Topeka, Kansas 20682
phone: 1-470-555-0398
email: margaret.hodge@ku.edu
additional roles:
- new membership task force
- accounting task force
McDaniel, Katheryn:
position: president
address:
> 138 Almond Street
> Topeka, Kansas 20697
phone:
cell: 1-210-555-5297
work: 1-210-555-8470
email: KateMcD@aol.com
additional roles:
- board member
Purvis, Fumiko:
Position: Treasurer
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Additional Roles:
- accounting task force
When round-tripping a NestedText document (reading the document and then later writing it back out), one often wants to undo any changes that were made to the keys when reading the documents. These modifications would be due to key normalization or key de-duplication. This is easily accomplished by simply retaining the keymap from the original load and passing it to the dumper by way of the map_keys argument.
>>> def normalize_key(key, parent_keys):
... if len(parent_keys) == 0:
... return key
... return '_'.join(key.lower().split())
>>> keymap = {}
>>> addresses = nt.load('examples/address.nt', normalize_key=normalize_key, keymap=keymap)
>>> filtered = {k:v for k,v in addresses.items() if 'fumiko' in k.lower()}
>>> print(nt.dumps(filtered))
Fumiko Purvis:
position: Treasurer
address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
phone: 1-268-555-0280
email: fumiko.purvis@hotmail.com
additional_roles:
- accounting task force
>>> print(nt.dumps(filtered, map_keys=keymap))
Fumiko Purvis:
Position: Treasurer
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Additional Roles:
- accounting task force
Notice that the keys differ between the two. The normalized key are output in the former and original keys in the latter.
Finally consider the case where you want to do both things; you want to return to the original keys but you also want to change the presentation. For example, imagine wanting to display the original keys in blue. That can be done as follows:
>>> from inform import Color
>>> blue = Color('blue', enable=Color.isTTY())
>>> def format_key(key, parent_keys):
... orig_keys = nt.get_original_keys(parent_keys + (key,), keymap)
... return blue(orig_keys[-1])
>>> print(nt.dumps(filtered, map_keys=format_key))
Fumiko Purvis:
Position: Treasurer
Address:
> 3636 Buffalo Ave
> Topeka, Kansas 20692
Phone: 1-268-555-0280
EMail: fumiko.purvis@hotmail.com
Additional Roles:
- accounting task force
The result looks identical in the documentation, but if you ran this program in a terminal you would see the keys in blue.
References¶
A reference allows you to define some content once and insert that content multiple places in the document. A reference is also referred to as a macro. Both simple and parametrized references can be easily implemented. For parametrized references, the arguments list is treated as an embedded NestedText document.
The technique is demonstrated with an example. This example is a fragment of a diet program. It reads two NestedText documents, one containing the known foods, and the other that documents the actual meals as consumed. The foods may be single ingredient, like steel cut oats, or it may contain multiple ingredients, like oatmeal. The use of parametrized references allows one to override individual ingredients in a composite ingredient. In this example, the user simply specifies the composite ingredient oatmeal on 21 March. On 22 March, they specify it as a simple reference, meaning that they end up with the same ingredients, but this time they are listed separately in the final summary. Finally, on 23 March they specify oatmeal using a parametrized reference so as to override the number of tangerines consumed and add some almonds.
#!/usr/bin/env python3
from inform import Error, display, dedent
import nestedtext as nt
import re
foods = nt.loads(dedent("""
oatmeal:
steel cut oats: 1/4 cup
tangerines: 1 each
whole milk: 1/4 cup
steel cut oats:
calories by weight: 150/40 cals/gram
tangerines:
calories each: 40 cals
calories by weight: 53/100 cals/gram
whole milk:
calories by weight: 149/255 cals/gram
calories by volume: 149 cals/cup
almonds:
calories each: 40 cals
calories by weight: 822/143 cals/gram
calories by volume: 822 cals/cup
"""), dict)
meals = nt.loads(dedent("""
21 March 2023:
breakfast: oatmeal
22 March 2023:
breakfast: @oatmeal
23 March 2023:
breakfast: @oatmeal(tangerines: 0 each, almonds: 10 each)
"""), dict)
def expand_foods(value):
# allows macro values to be defined as a top-level food.
# allows macro reference to be found anywhere.
if isinstance(value, str):
value = value.strip()
if value[:1] == '@':
value = parse_macro(value[1:].strip())
return value
if isinstance(value, dict):
return {k:expand_foods(v) for k, v in value.items()}
if isinstance(value, list):
return [expand_foods(v) for v in value]
raise NotImplementedError(value)
def parse_macro(macro):
match = re.match(r'(\w+)(?:\((.*)\))?', macro)
if match:
name, args = match.groups()
try:
food = foods[name].copy()
except KeyError:
raise Error("unknown food.", culprit=name)
if args:
args = nt.loads('{' + args + '}', dict)
food.update(args)
return food
raise Error("unknown macro.", culprit=macro)
try:
meals = expand_foods(meals)
display(nt.dumps(meals))
except Error as e:
e.terminate()
It produces the following output:
21 March 2023:
breakfast: oatmeal
22 March 2023:
breakfast:
steel cut oats: 1/4 cup
tangerines: 1 each
whole milk: 1/4 cup
23 March 2023:
breakfast:
steel cut oats: 1/4 cup
tangerines: 0 each
whole milk: 1/4 cup
almonds: 10 each
In this example the content for the references was pulled from a different NestedText document. See the PostMortem as an example that pulls the referenced content from the same document.
Pretty Printing¶
Besides being a readable file format, NestedText makes a reasonable display format for structured data. This example further simplifies the output by stripping leading multiline string tags.
>>> import nestedtext as nt
>>> import re
>>>
>>> def pp(data):
... try:
... text = nt.dumps(data, default=repr)
... print(re.sub(r'^(\s*)[>:][ ]?(.*)$', r'\1\2', text, flags=re.M))
... except nt.NestedTextError as e:
... e.report()
>>> addresses = nt.load('examples/address.nt')
>>> pp(addresses['Katheryn McDaniel'])
position: president
address:
138 Almond Street
Topeka, Kansas 20697
phone:
cell: 1-210-555-5297
work: 1-210-555-8470
email: KateMcD@aol.com
additional roles:
- board member
Long Lines¶
One of the benefits of NestedText is that no escaping of special characters is ever needed. However, you might find it helpful to add your own support for removing escaped newlines in multi-line strings. Doing so allows you to keep your lines short in the source document so as to make them easier to interpret in windows of limited width.
This example uses the pretty-print function from the previous example.
>>> import nestedtext as nt
>>> from textwrap import dedent
>>> from voluptuous import Schema
>>> document = dedent(r"""
... lorum ipsum:
... > Lorem ipsum dolor sit amet, \
... > consectetur adipiscing elit.
... > Sed do eiusmod tempor incididunt \
... > ut labore et dolore magna aliqua.
... """)
>>> def reverse_escaping(text):
... return text.replace("\\\n", "")
>>> schema = Schema({str: reverse_escaping})
>>> data = schema(nt.loads(document))
>>> pp(data)
lorum ipsum:
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.