Part 1: Parse
Parse our sample SF crime data.
Module Setup
Open up parse.py
, found: new-coder/dataviz/tutorial_source/parse.py
The beginning of the module is an introduction as well as any copyright and/or license information.
Quick note: Licensing Open Source Software
When you start using third-party libraries, pay attention to the license and/or copyright information that is written out. Generally, if the library/code has no license, it means all rights reserved for the author (do not use the code). If it’s GPL, your application/script must also be licensed under the GPL. Although, technically any license that is GPL-compatible is fine to use too: GPL Compatible Licenses or GPL compatibility.
If it’s mit/2-clause bsd you can do whatever you want (no need to use the same license, or even have a license), if it’s 3-clause bsd you can do whatever you want but have to credit the original author.
For the curious
Code that is up on GitHub does not mean that it is free to use. If you want to use a library, ask the developer if s/he has plans to include a LICENSE
file or in the headers of the files if it’s not there already.
If you want to open source your code (yay, go you!), include your desired license either as a separate file or within the preamble/beginning of your code. Licensing your code is simply copying & pasting the required language of a license of your choice into your codebase.
For reference, this tutorial is licensed under the Creative Commons license, specifically, Creative Commons Attribution 3.0 Unported license, with the code under zlib/libpng simply because it’s short.
Back to the tutorial
In order to read a CSV/Excel file, we have to import the csv
module from Python's standard library.
1 | import csv
|
MY_FILE
is defining a global - notice how it‘s all caps, a convention for variables we won't be changing. Included in this repo is a sample file to which this variable is assigned.
1 | MY_FILE = "../data/sample_sfpd_incident_all.csv"
|
The Parse Function
In defining the function, we know that we want to give it the CSV file, as well as the delimiter in which the CSV file uses to delimit each element/column.
1 | def parse(raw_file, delimiter):
|
We also know that we want to return a JSON-like object. A JSON file/object is just a collection of dictionaries, much like Python’s dictionary.
1 2 3 | def parse(raw_file, delimiter):
return parsed_data
|
Let’s be good coders and write a documentation-string (doc-string) for future folks that may read our code. Notice the triple-quotes:
1 2 3 4 | def parse(raw_file, delimiter):
"""Parses a raw CSV file to a JSON-line object."""
return parsed_data
|
For the curious
If you are interested in understanding how docstrings work, Python’s PEP (Python Enhancement Proposals) documents spell out how one should craft his/her docstrings: PEP8 and PEP257. This also gives you a peek at what is considered “Pythonic”.
The difference between """docstrings"""
and # comments
have to do with who the reader will be. Within the a Python shell, if you call help
on a particular function or class, it will return the """docstring"""
that the developer has written.
There are also documentation programs that look specifically for """docstrings"""
to help the developer automatically produce documentation separated out of the code. Within docstrings, it’s helpful to say imperatively what the function/method or class is supposed to do. Examples of how the documented code should work can also be written in the docstrings (and, subsequently, tested). # comments
, on the otherhand, are for those reading through the code — the comments are to simply say what a specific piece/line of code is meant to do. Inline # comments
are always appreciated by those reading through your code. Many developers also litter # TODO
or # FIXME
statements for combing through later.
Back to the tutorial
What we have now is a pretty good skeleton - we know what parameters the function will take (raw_file
and delimiter
), what it is supposed to do (our """doc-string"""
), and what it will return, parsed_data
. Notice how the parameters and the return value is descriptive in itself.
Let’s sketch out, with comments, how we want this function to take a raw file and give us the format that we want. First, let’s open the file, and the read the file, then build the parsed_data element.
1 2 3 4 5 6 7 8 9 10 11 12 | def parse(raw_file, delimiter):
"""Parses a raw CSV file to a JSON-line object"""
# Open CSV file
# Read CSV file
# Close CSV file
# Build a data structure to return parsed_data
return parsed_data
|
Thankfully, there are a lot of built-in methods that Python has that we can use to do all the steps that we’ve outlined with our comments. The first one we’ll use is open
and pass raw_file
to it, which we got from defining our own parameters in the parse
function:
1 2 3 | with open(raw_file) as opened_file:
...
|
So we’ve told Python to open the file, now we have to read the file. We have to use the CSV module that we imported earlier:
1 | csv_data = csv.reader(opened_file, delimiter=delimiter)
|
Here, csv.reader
is a function of the CSV module. We gave it two parameters: opened_file, and delimiter. It’s easy to get confused when parameters and variables share names. In delimiter=delimiter
, the first 'delimiter' is referring to the name of the parameter that csv.reader
needs; the second 'delimiter' refers to the argument that our parse
function takes in.
Just to quickly put these two lines in our parse
function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | def parse(raw_file, delimiter):
"""Parses a raw CSV file to a JSON-line object"""
# Open CSV file
open_file = open(raw_file)
# Read the CSV data
csv_data = csv.reader(opened_file, delimiter=delimiter)
# Build a data structure to return parsed_data
# Close the CSV file
return parsed_data
|
For the curious
The csv_data object
, in Python terms, is now an iterator. In very simple terms, this means we can get each element in csv_data
one at a time.
Back to the tutorial
Alright — the building of the data structure might seem tricky. The best way to start off is to set up an empty Python list to our parsed_data
variable so we can add every row of data that we will parse through.
1 | parsed_data = []
|
Good — we have a good data structure to add to. Now let’s first address our column headers that came with the CSV file. They will be the first row, and we’ll asign them to the variable fields
:
1 | fields = csv_data.next()
|
For the curious
We were able to call the .next
method on csv_data
because it is a generator. We just call .next
once, since headers are in the 1st and only row of our CSV file.
Back to the tutorial
Let’s loop over each row now that we have the headers properly taken care of. With each loop, we will add a dictionary that maps a field (those column headers) to the value in the CSV cell.
1 2 | for row in csv_data:
parsed_data.append(dict(zip(fields, row)))
|
Here, we iterated over each row in the csv_data
item. With each loop, we appended a dictionary (dict()
) to our list,parsed_data
. We use Python’s built-in zip()
function to zip together header → value to make our dictionary of every row.
Now let’s put the function together:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | def parse(raw_file, delimiter):
"""Parses a raw CSV file to a JSON-like object"""
# Open CSV file
opened_file = open(raw_file)
# Read the CSV data
csv_data = csv.reader(opened_file, delimiter=delimiter)
# Setup an empty list
parsed_data = []
# Skip over the first line of the file for the headers
fields = csv_data.next()
# Iterate over each row of the csv file, zip together field -> value
for row in csv_data:
parsed_data.append(dict(zip(fields, row)))
# Close the CSV file
opened_file.close()
return parsed_data
|
Using the new Parse function
Let’s define a main()
function to act as the starting point for our script,
and use our new parse()
function:
1 2 3 4 5 6 | def main():
# Call our parse function and give it the needed parameters
new_data = parse(MY_FILE, ",")
# Let's see what the data looks like!
print new_data
|
We called our function parse()
and gave it the MY_FILE
global variable that we defined at the beginning, as well as the delimiter ","
.
We assign the function to the variable new_data
since the parse()
function will return a parsed_data
object. Last, we print new_data
to see our list of dictionaries!
One final bit — when running a Python file from the command line, Python will execute all of the code found on it. Since the following bit is True
,
1 2 | if __name__ == "__main__":
main()
|
it will call the main()
function. By doing the name == __main__
check, you can have that code only execute when you want to run the module as a program (via the command line) and not have it execute when someone just wants to import the parse()
function itself into another Python file. This is referred to as “boilerplate code” — code doesn’t really do anything and yet is necessary.
Putting it to action
So you’ve written the parse function and your parse.py
file looks like mine. Now what? Let’s run it and parse some d*mn files!
Be sure to have your virtualenv activated that you created earlier in setup. Your terminal prompt should look something like this:
1 | (DataVizProj) $
|
Within the new-coder/dataviz/
directory, let’s make a directory for the python files you are writing with the bash command mkdir [Directory_Name]
:
1 2 3 4 5 6 | (DataVizProj) $ mkdir MySourceFiles
(DataVizProj) $ ls # list available files and directories where we are
README.me requirements.txt data full_source MySourceFiles tutorial_source
(DataVizProj) $ pwd # current location in our directory structure
Users/lynnroot/MyProjects/new-coder/dataviz/
(DataVizProj) $ cd MySourceFiles # change directories into our new directory
|
Go ahead and save your copy of parse.py
into MySourceFiles
(through “Save As” within your text editor). You should see the file in the directory if you return to your terminal and type ls
.
To run the python code, you have to tell the terminal to execute the parse.py file with python:
1 | (DataVizProj) $ python parse.py
|
If you got a traceback, or an error message, compare your parse.py
file with new-coder/dataviz/tutorial_source/parse.py
. Perhaps a typo, or you don’t have your virtualenv setup properly.
 For reference, the last bit of output you should see in your terminal should look like (doesn’t have to be exact data, but the structure of {"key": "value"} should look familiar):
1 2 3 4 | 'ARRESTED, BOOKED'},{'Category': 'OTHER OFFENSES', 'IncidntNum': '030204238',
'DayOfWeek': 'Tuesday', 'Descript': 'OBSCENE PHONE CALLS(S)', 'PdDistrict':
'PARK', 'Y': '37.7773636900243', 'Location': '800 Block of CENTRAL AV', 'Time':
'18:59', 'Date': '02/18/2003', 'X': '-122.445006858202', 'Resolution': 'NONE'}]
|
You see this output because in the def main()
function, and you explicitly say print new_data
which feeds to the output of the terminal. You could, for instance, not print the new_data
variable, and just pass the new_data
variable to another function. Coincidently, that’s what Part II and Part III are about!
Explore further
Play around with parse.py
within the Python interpreter itself. Make sure you’re in your MySourceFiles
directory, then start the Python interpreter from there:
1 2 3 4 5 | (DataVizProj) $ python
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
|
To exit out of the Python shell, press CTRL-D
.
Next, import your parse.py
file into the interpreter. Notice there is no need to include the .py
portion when importing:
1 2 | >>> import parse
>>>
|
If all things go well with import parse
you should just see the >>>
prompt. If there’s an error, perhaps you are not in the correct directory from two steps ago.
Play with the following commands. Notice to access any object defined in parse.py (object meaning a variable, function, etc), you must preface it with parse
:
1 2 3 4 5 6 7 8 9 | >>> parse.MY_FILE
'../data/sample_sfpd_incident_all.csv'
>>> type(parse.MY_FILE)
<type: 'str'>
>>> copy_my_file = parse.MY_FILE
>>> copy_my_file
'../data/sample_sfpd_incident_all.csv'
>>> type(copy_my_file)
<type: 'str'>
|
So we made what seems like a copy. Not so! check it out:
1 2 3 4 5 | >>> id(copy_my_file)
4404350288
>>> id(parse.MY_FILE)
4404350288
>>>
|
Those numbers from calling the id
function reflect where the variable is saved in the computer’s memory. Since they are the same number, Python has set up a reference from copy_my_file
to the same location that parse.MY_FILE
was saved. No need to allocate new space in memory for what is essentially the same variable with a different name.
Let’s play with the parser function a bit:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | >>> new_data = parse.parse(copy_my_file, ",")
>>> type(new_data)
<type: 'list'>
>>> type(new_data[0])
<type: 'dict'>
>>> type(new_data[0]["DayOfWeek"])
<type: 'str'>
>>> new_data[0].keys()
['Category', 'IncidntNum', 'DayOfWeek', 'Descript', 'PdDistrict', 'Y', 'Location', 'Time', 'Date', 'X', 'Resolution']
>>> new_data[0].values()
['FRAUD', '030203898', 'Tuesday', 'FORGERY, CREDIT CARD', 'NORTHERN', '37.8014488257836', '2800 Block of VAN NESS AV', '16:30', '02/18/2003', '-122.424612993055', 'NONE']
>>> for dict_item in new_data:
... print dict_item["Descript"]
...
DRIVERS LICENSE, SUSPENDED OR REVOKED
LOST PROPERTY
POSS OF LOADED FIREARM
<--snip-->
BATTERY
OBSCENE PHONE CALLS(S)
>>>
|
Here we checked ot the type of data that gets returned back to use from the parse function, as well as ways to simply check out what is the contents of the parsed data.
You can continue to play around; try >>> help(parse.parse)
to see our docstring, see what happens if you feed the parse function a different file, delimiter, or just a different variable. Challenge yourself to see if you can create a new file to save the parsed data, rather than just a variable. The example in the python docs may help.


Preparation time
less than 30 mins
Cooking time
10 to 30 mins
Serves
Serves 4
Mix your own curry paste for a flavour sensation. Try this Thai-style curry with steamed sticky rice.
To make the curry paste, in a small, dry frying pan toast the cardamom, cumin seeds, cloves and star anise. Tip into a small blender or spice grinder, or pestle and mortar, and work to a powder. Add all the other ingredients along with 2 tablespoons water and blend to a paste. Set aside.
For the curry, heat the oil in a large frying pan and cook the onions for a few minutes until soft and golden-brown. Add half the curry paste and cook for a minute or so. Add the potatoes, slashed bird’s-eye chillies, stock, coconut milk, chopped tomatoes, lime leaves, palm sugar, soy sauce and fish sauce. Simmer for about 15 minutes until the potatoes are tender and the wonderful flavours have developed.
Add the prawns and cook for a couple of minutes until cooked just through.
Check the seasoning, add a little more fish sauce if necessary, and stir in the lime juice and basil, keeping a few leaves back for garnish. Add half the chopped peanuts.
To serve, spoon into bowls and garnish with the reserved basil, peanuts and top it off with the red chilli. Serve with some steamed sticky rice.


Provides a methodology to solve most of multicriteria ranking problems using partial and total pre-order from Promethee methods. Albuquerque & Montenegro (2015) <doi:10.1080/03610926.2014.942432>.
| Version: | 0.1.1 |
| Depends: | R (≥ 3.1.0) |
| Imports: | Rcpp, RcppNumerical, linprog, lpSolve, ggplot2, gridExtra, ggnetwork, network, pastecs, methods, graphics, stats, utils, dplyr |
| LinkingTo: | Rcpp, RcppEigen, RcppNumerical |
| Published: | 2018-10-11 |
| Author: | Pedro Albuquerque and Gustavo Monteiro |
| Maintainer: | Pedro Albuquerque <pedroa at unb.br> |
| BugReports: | https://github.com/lamfo-unb/RMCriteria |
| License: | GPL-2 |
| NeedsCompilation: | yes |
| SystemRequirements: | GNU make |
| Materials: | README |
| CRAN checks: | RMCriteria results |
| Reference manual: | RMCriteria.pdf |
| Package source: | RMCriteria_0.1.1.tar.gz |
| Windows binaries: | r-devel: RMCriteria_0.1.1.zip, r-release: RMCriteria_0.1.1.zip, r-oldrel: RMCriteria_0.1.1.zip |
| OS X binaries: | r-release: RMCriteria_0.1.1.tgz, r-oldrel: RMCriteria_0.1.1.tgz |
| Old sources: | RMCriteria archive |
Please use the canonical form https://CRAN.R-project.org/package=RMCriteria to link to this page.


Developing a “recipe” extension¶
The objective of this tutorial is to illustrate roles, directives and domains. Once complete, we will be able to use this extension to describe a recipe and reference that recipe from elsewhere in our documentation.
Note
This tutorial is based on a guide first published on opensource.com and is provided here with the original author’s permission.
Overview¶
We want the extension to add the following to Sphinx:
A
recipe
directive, containing some content describing the recipe steps, along with a:contains:
option highlighting the main ingredients of the recipe.A
ref
role, which provides a cross-reference to the recipe itself.A
recipe
domain, which allows us to tie together the above role and domain, along with things like indices.
For that, we will need to add the following elements to Sphinx:
A new directive called
recipe
New indexes to allow us to reference ingredient and recipes
A new domain called
recipe
, which will contain therecipe
directive andref
role
Prerequisites¶
We need the same setup as in the previous extensions. This time,
we will be putting out extension in a file called recipe.py
.
Here is an example of the folder structure you might obtain:
└── source
├── _ext
│ └── recipe.py
├── conf.py
└── index.rst
Writing the extension¶
Open recipe.py
and paste the following code in it, all of which we will
explain in detail shortly:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 | from collections import defaultdict
from docutils.parsers.rst import directives
from sphinx import addnodes
from sphinx.directives import ObjectDescription
from sphinx.domains import Domain, Index
from sphinx.roles import XRefRole
from sphinx.util.nodes import make_refnode
class RecipeDirective(ObjectDescription):
"""A custom directive that describes a recipe."""
has_content = True
required_arguments = 1
option_spec = {
'contains': directives.unchanged_required,
}
def handle_signature(self, sig, signode):
signode += addnodes.desc_name(text=sig)
return sig
def add_target_and_index(self, name_cls, sig, signode):
signode['ids'].append('recipe' + '-' + sig)
if 'noindex' not in self.options:
ingredients = [
x.strip() for x in self.options.get('contains').split(',')]
recipes = self.env.get_domain('recipe')
recipes.add_recipe(sig, ingredients)
class IngredientIndex(Index):
"""A custom index that creates an ingredient matrix."""
name = 'ingredient'
localname = 'Ingredient Index'
shortname = 'Ingredient'
def generate(self, docnames=None):
content = defaultdict(list)
recipes = {name: (dispname, typ, docname, anchor)
for name, dispname, typ, docname, anchor, _
in self.domain.get_objects()}
recipe_ingredients = self.domain.data['recipe_ingredients']
ingredient_recipes = defaultdict(list)
# flip from recipe_ingredients to ingredient_recipes
for recipe_name, ingredients in recipe_ingredients.items():
for ingredient in ingredients:
ingredient_recipes[ingredient].append(recipe_name)
# convert the mapping of ingredient to recipes to produce the expected
# output, shown below, using the ingredient name as a key to group
#
# name, subtype, docname, anchor, extra, qualifier, description
for ingredient, recipe_names in ingredient_recipes.items():
for recipe_name in recipe_names:
dispname, typ, docname, anchor = recipes[recipe_name]
content[ingredient].append(
(dispname, 0, docname, anchor, docname, '', typ))
# convert the dict to the sorted list of tuples expected
content = sorted(content.items())
return content, True
class RecipeIndex(Index):
"""A custom index that creates an recipe matrix."""
name = 'recipe'
localname = 'Recipe Index'
shortname = 'Recipe'
def generate(self, docnames=None):
content = defaultdict(list)
# sort the list of recipes in alphabetical order
recipes = self.domain.get_objects()
recipes = sorted(recipes, key=lambda recipe: recipe[0])
# generate the expected output, shown below, from the above using the
# first letter of the recipe as a key to group thing
#
# name, subtype, docname, anchor, extra, qualifier, description
for name, dispname, typ, docname, anchor, _ in recipes:
content[dispname[0].lower()].append(
(dispname, 0, docname, anchor, docname, '', typ))
# convert the dict to the sorted list of tuples expected
content = sorted(content.items())
return content, True
class RecipeDomain(Domain):
name = 'recipe'
label = 'Recipe Sample'
roles = {
'ref': XRefRole()
}
directives = {
'recipe': RecipeDirective,
}
indices = {
RecipeIndex,
IngredientIndex
}
initial_data = {
'recipes': [], # object list
'recipe_ingredients': {}, # name -> object
}
def get_full_qualified_name(self, node):
return '{}.{}'.format('recipe', node.arguments[0])
def get_objects(self):
for obj in self.data['recipes']:
yield(obj)
def resolve_xref(self, env, fromdocname, builder, typ, target, node,
contnode):
match = [(docname, anchor)
for name, sig, typ, docname, anchor, prio
in self.get_objects() if sig == target]
if len(match) > 0:
todocname = match[0][0]
targ = match[0][1]
return make_refnode(builder, fromdocname, todocname, targ,
contnode, targ)
else:
print('Awww, found nothing')
return None
def add_recipe(self, signature, ingredients):
"""Add a new recipe to the domain."""
name = '{}.{}'.format('recipe', signature)
anchor = 'recipe-{}'.format(signature)
self.data['recipe_ingredients'][name] = ingredients
# name, dispname, type, docname, anchor, priority
self.data['recipes'].append(
(name, signature, 'Recipe', self.env.docname, anchor, 0))
def setup(app):
app.add_domain(RecipeDomain)
return {
'version': '0.1',
'parallel_read_safe': True,
'parallel_write_safe': True,
}
|
Let’s look at each piece of this extension step-by-step to explain what’s going on.
The directive class
The first thing to examine is the RecipeDirective
directive:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | class RecipeDirective(ObjectDescription):
"""A custom directive that describes a recipe."""
has_content = True
required_arguments = 1
option_spec = {
'contains': directives.unchanged_required,
}
def handle_signature(self, sig, signode):
signode += addnodes.desc_name(text=sig)
return sig
def add_target_and_index(self, name_cls, sig, signode):
signode['ids'].append('recipe' + '-' + sig)
if 'noindex' not in self.options:
ingredients = [
x.strip() for x in self.options.get('contains').split(',')]
recipes = self.env.get_domain('recipe')
recipes.add_recipe(sig, ingredients)
|
Unlike Developing a “Hello world” extension and Developing a “TODO” extension, this directive doesn’t derive from
docutils.parsers.rst.Directive
and doesn’t define a run
method.
Instead, it derives from sphinx.directives.ObjectDescription
and
defines handle_signature
and add_target_and_index
methods. This is
because ObjectDescription
is a special-purpose directive that’s intended
for describing things like classes, functions, or, in our case, recipes. More
specifically, handle_signature
implements parsing the signature of the
directive and passes on the object’s name and type to its superclass, while
add_taget_and_index
adds a target (to link to) and an entry to the index
for this node.
We also see that this directive defines has_content
, required_arguments
and option_spec
. Unlike the TodoDirective
directive added in the
previous tutorial, this directive takes a single argument, the
recipe name, and an option, contains
, in addition to the nested
reStructuredText in the body.
The index classes
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | class IngredientIndex(Index):
"""A custom index that creates an ingredient matrix."""
name = 'ingredient'
localname = 'Ingredient Index'
shortname = 'Ingredient'
def generate(self, docnames=None):
content = defaultdict(list)
recipes = {name: (dispname, typ, docname, anchor)
for name, dispname, typ, docname, anchor, _
in self.domain.get_objects()}
recipe_ingredients = self.domain.data['recipe_ingredients']
ingredient_recipes = defaultdict(list)
# flip from recipe_ingredients to ingredient_recipes
for recipe_name, ingredients in recipe_ingredients.items():
for ingredient in ingredients:
ingredient_recipes[ingredient].append(recipe_name)
# convert the mapping of ingredient to recipes to produce the expected
# output, shown below, using the ingredient name as a key to group
#
# name, subtype, docname, anchor, extra, qualifier, description
for ingredient, recipe_names in ingredient_recipes.items():
for recipe_name in recipe_names:
dispname, typ, docname, anchor = recipes[recipe_name]
content[ingredient].append(
(dispname, 0, docname, anchor, docname, '', typ))
# convert the dict to the sorted list of tuples expected
content = sorted(content.items())
return content, True
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | class RecipeIndex(Index):
"""A custom index that creates an recipe matrix."""
name = 'recipe'
localname = 'Recipe Index'
shortname = 'Recipe'
def generate(self, docnames=None):
content = defaultdict(list)
# sort the list of recipes in alphabetical order
recipes = self.domain.get_objects()
recipes = sorted(recipes, key=lambda recipe: recipe[0])
# generate the expected output, shown below, from the above using the
# first letter of the recipe as a key to group thing
#
# name, subtype, docname, anchor, extra, qualifier, description
for name, dispname, typ, docname, anchor, _ in recipes:
content[dispname[0].lower()].append(
(dispname, 0, docname, anchor, docname, '', typ))
# convert the dict to the sorted list of tuples expected
content = sorted(content.items())
return content, True
|
Both IngredientIndex
and RecipeIndex
are derived from Index
.
They implement custom logic to generate a tuple of values that define the
index. Note that RecipeIndex
is a simple index that has only one entry.
Extending it to cover more object types is not yet part of the code.
Both indices use the method Index.generate()
to do their work. This
method combines the information from our domain, sorts it, and returns it in a
list structure that will be accepted by Sphinx. This might look complicated but
all it really is is a list of tuples like ('tomato', 'TomatoSoup', 'test',
'rec-TomatoSoup',...)
. Refer to the domain API guide for more information on this API.
These index pages can be referred by combination of domain name and its
name
using ref
role. For example, RecipeIndex
can be
referred by :ref:`recipe-recipe`
.
The domain
A Sphinx domain is a specialized container that ties together roles, directives, and indices, among other things. Let’s look at the domain we’re creating here.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | class RecipeDomain(Domain):
name = 'recipe'
label = 'Recipe Sample'
roles = {
'ref': XRefRole()
}
directives = {
'recipe': RecipeDirective,
}
indices = {
RecipeIndex,
IngredientIndex
}
initial_data = {
'recipes': [], # object list
'recipe_ingredients': {}, # name -> object
}
def get_full_qualified_name(self, node):
return '{}.{}'.format('recipe', node.arguments[0])
def get_objects(self):
for obj in self.data['recipes']:
yield(obj)
def resolve_xref(self, env, fromdocname, builder, typ, target, node,
contnode):
match = [(docname, anchor)
for name, sig, typ, docname, anchor, prio
in self.get_objects() if sig == target]
if len(match) > 0:
todocname = match[0][0]
targ = match[0][1]
return make_refnode(builder, fromdocname, todocname, targ,
contnode, targ)
else:
print('Awww, found nothing')
return None
def add_recipe(self, signature, ingredients):
"""Add a new recipe to the domain."""
name = '{}.{}'.format('recipe', signature)
anchor = 'recipe-{}'.format(signature)
self.data['recipe_ingredients'][name] = ingredients
# name, dispname, type, docname, anchor, priority
self.data['recipes'].append(
(name, signature, 'Recipe', self.env.docname, anchor, 0))
|
There are some interesting things to note about this recipe
domain and domains
in general. Firstly, we actually register our directives, roles and indices
here, via the directives
, roles
and indices
attributes, rather than
via calls later on in setup
. We can also note that we aren’t actually
defining a custom role and are instead reusing the
sphinx.roles.XRefRole
role and defining the
sphinx.domains.Domain.resolve_xref
method. This method takes two
arguments, typ
and target
, which refer to the cross-reference type and
its target name. We’ll use target
to resolve our destination from our
domain’s recipes
because we currently have only one type of node.
Moving on, we can see that we’ve defined initial_data
. The values defined in
initial_data
will be copied to env.domaindata[domain_name]
as the
initial data of the domain, and domain instances can access it via
self.data
. We see that we have defined two items in initial_data
:
recipes
and recipe2ingredient
. These contain a list of all objects
defined (i.e. all recipes) and a hash that maps a canonical ingredient name to
the list of objects. The way we name objects is common across our extension and
is defined in the get_full_qualified_name
method. For each object created,
the canonical name is recipe.<recipename>
, where <recipename>
is the
name the documentation writer gives the object (a recipe). This enables the
extension to use different object types that share the same name. Having a
canonical name and central place for our objects is a huge advantage. Both our
indices and our cross-referencing code use this feature.
The setup
function
As always, the setup
function is a requirement and is used to
hook the various parts of our extension into Sphinx. Let’s look at the
setup
function for this extension.
1 2 3 4 5 6 7 8 | def setup(app):
app.add_domain(RecipeDomain)
return {
'version': '0.1',
'parallel_read_safe': True,
'parallel_write_safe': True,
}
|
This looks a little different to what we’re used to seeing. There are no calls
to add_directive()
or even add_role()
. Instead, we
have a single call to add_domain()
followed by some
initialization of the standard domain. This is because we
had already registered our directives, roles and indexes as part of the
directive itself.
Using the extension¶
You can now use the extension throughout your project.  I highly recommend the
:recipe:ref:`TomatoSoup` recipe in particular!
.. toctree::
tomato-soup
The recipe contains `tomato` and `cilantro`.
.. recipe:recipe:: TomatoSoup
:contains: tomato cilantro salt pepper
This recipe is a tasty tomato soup, combine all ingredients
and cook.
The important things to note are the use of the :recipe:ref:
role to
cross-reference the recipe actually defined elsewhere (using the
:recipe:recipe:
directive.


This is a complete graph. Though there's an overwhelming number of edges, we know how many edges there are by just counting the vertices, of which there are far fewer. There's 32 vertices, so we know that there's 496 edges. How do we know that? Let's see.
To explain this, we need to define some terms.
First, here is a vertex:
Vertices (plural of vertex) are represented as dots.
Here is an edge:
Edges are represented as lines that connect two vertices.
A graph is a collection of vertices and edges. Here are two examples:
There is no requirement that vertices must be connected by edges, though, they can be connected by edges. There isn't even a requirement that there are any edges at all. A single vertex can be a graph.
A complete graph is a graph in which every vertex is connected to every other vertex in the graph by an edge.
This is a complete graph for 3 vertices:
Move the slider to change in the number of vertices (which in turn will really change the number of edges).
Generally, when you increase the number of vertices in a complete graph by 1, the number of edges increases by v + e, where v is the number of vertices you had before the increase, and e is the number edges you had before the increase.
For example, increasing v, the number of vertices from 2 to 3 vertices increases e, the number of edges, from 1 to 3 (v = 2, e = 1, 2 + 1 = 3). So, v increased by 1, but e increased by 2.
Here is a table with the edge counts for complete graphs with given numbers of vertices.
| Graph | Vertices | Previous v+e | Edges | |
| 1 | ⇒ | 0 | ||
| 2 | 1 + 0 | ⇒ | 1 | |
| 3 | 2 + 1 | ⇒ | 3 | |
| 4 | 3 + 3 | ⇒ | 6 | |
| 5 | 4 + 6 | ⇒ | 10 | |
| 6 | 5 + 10 | ⇒ | 15 | |
| 7 | 6 + 15 | ⇒ | 21 | |
| 8 | 7 + 21 | ⇒ | 28 | |
| 9 | 8 + 28 | ⇒ | 36 | |
| 10 | 9 + 36 | ⇒ | 45 |
When you add a vertex, the number of edges also increases. But (except when going from 0 vertices to 1 or from one vertex to two) the increase in edges is always greater than the increase in vertices.


It all began, as usual, with a simple yes or no question on how to loop in C++.
Which led to size_type
vs size_t
discussion and ended with a joke and a bit of assembly.
M: Assuming type of x is known and the code is not inside a template function or class, 1 or 2?
// 1
const size_t n = x.size();
for (size_t i = n + 1; ....
// OR
// 2
const auto n = x.size();
for (std::decay<decltype(n)>::type i = n + 1; ....
AL: Is it a joke?
M: No. Why?
AL: Then the 1st. How can anyone pick the 2nd? No templates, more difficult to read.
M: More generic?
AL: Yeah… Potentially generic, but readability is much worse. But let’s ask AR, he loves generic code.
AR: First of all, you don’t care about x
. You only care about type deduction on x.size()
. Further in the second option
you don’t need decay
, auto
will simply work. So I vote for the third option (if we agree that ranges/iterators are out of the scope):
const auto n = x.size();
for (auto i = n + 1; ....
But the first option is also fine for every-day code. But not for STL or library code.
AL: Type of x
itself is not necessary to know, but it is good to know that it is not a template.
AR: But what does it change?
AL: Anyway, does not matter. Your 3rd option is the best. But I was also fine with the 1st one.
AR: Not for STL.
AL: Why?
AR: Type of size()
is not size_t
, but typename decltype(x)::size_type
.
AL: That’s what I’m saying, if type of x
is known then it should not matter.
AR: But it does not mean that size()
returns size_t
. In standard library it returns size_type
even if a container is known.
size_type
is implementation defined. Anyway, these are details.
AL: If it’s known that x
is vector<int>
then vector<int>::size_type
is equivalent to size_t
.
AR: Nope. size_type
is implementation defined.
AL: Are you saying that vector<int>::size_type
does not always resolve to size_t
? The elements of a vector are stored contiguously.
And it is guaranteed that size_t
can address anything in memory.
AR: Ha!!!111:
template <.....> class vector {
typedef /*implementation defined*/ size_type;
size_type size() const;
//...
AL: Can it be smaller than size_t
? It cannot be wider. It must be equivalent to size_t
. But technically you are right.
AR: We have to deal with what standard says. It’s a law!
AL: I can prove that vector<int>::size_type
is equivalent to size_t
. Otherwise it cannot be. But in general it is not known, indeed.
Can you provide a single example where vector<int>::size_type
is not equivalent to size_t
.
AR: You can probably find a platform where sizeof(size_type) != sizeof(size_t)
.
AL: Can you provide an example of such platform? I think I could show that it is impossible. It’s guaranteed that
vector<X>::size_type
cannot be smaller than a pointer. And I think it could not be larger than a pointer. Let me think…
AR: Standard explicitly states that it is implementation defined. Why would I break it anyway? Just because I cannot find an example?
AL: I think I found a mistake in my “proof”. &x[n]
does not have to be a pointer. It’s just vector<T>::reference_type
.
AR: It’s more safer just to listen to what the standard says to avoid potential UB.
AL: But if you prove that size_type
is equivalent to size_t
then you could just start using it.
AR: Until the next standard.
M: Google says that it’s quite easy to make an allocator with sizeof(size_type) > sizeof(size_t)
. It will map container’s memory to bigger than system’s memory storage.
AR: Yes, you could make such allocator. But what would be vector<T, A>::reference_type
then?
AL: Any type with operator.
? Which we could not overload yet. Anyway, we all agree that AR’s option auto i = n + 1
was the best.
M: What if 0
is used instead of n + 1
? Should it be like this auto i = decltype(x.size())0;
then?
AR: Better would be auto i = decltype(x.size())();
.
AL: auto i = (n - n);
That’s C++, you cannot just write auto i = 0
. Instead you should do either auto i = decltype(x.size())();
or auto i = x.size() - x.size();
.
Although the latter looks funny, it is identical to the former (based on optimized assembly from gcc-4.9).


{% extends 'framework/base.html' %} {% block content %} {# This is a local page link to the top of the page in case of navigation switch ie. click on All Posts #}


channel.36a4c3 __
channel.36313e __
channel.afbb32 __
channel.f9fbfc __
octree __
channel.6b9aa6 __
channel.54afc9 __
channel.7b9b37 __
channel.f91b15 __
binary __
channel.75c7db __
channel.7a9648 __
channel.2e9ebb __
channel.8165a2 __
channel.785e9a __


The web audio API context is loading. Loading
Player One
Default options
Peak Clear duration
Player Two
Vertical set to true


This script delivers a mail message taken from stdin to a mail queue, and
then starts the mail filter program if it isn’t already running.
Usage
deliver.rb is invoked from the command line using:
% deliver.rb <options>
Options are:
| —load-path: | Prepend the given directory to ruby’s load path.
|
| —log filename: | Log to the given filename. If no log is specified, no logging
occurs.
|
| —home directory: | Specify the home directory. rdeliver will change to this directory
before reading and writing any files. The home directory defaults to the
value of the HOME or LOGDIR environment variable.
|
Installation
Assuming you have the RubyMail mail classes installed, you typically have
to put something like this in your .forward file:
|"/usr/local/bin/deliver.rb --log /home/you/.rlog"
This will call rdeliver for each new message you get, and log to
/home/you/.rlog.
Catastrophic Errors
The deliver.rb script is very careful with errors. If there is any problem,
it logs the error to the log file you specify. But if you do not specify a
log file, or the error occurs before the log file is opened, a record of
the error is placed in a file called CATASTROPHIC_DELIVERY_FAILURE in the
home directory. If that fails, the error information is printed to the
standard output in the hopes that it will be part of a bounce message. In
all cases, the exit code 75 is returned, which tells the MTA to re-try the
delivery again.