Getting Started

Example

Let’s start by way of an example. Note that several concepts are introduced here such as snippets, scope and the gcgen configuration file format - we will cover all of these in the coming sections, this example will just serve to illustrate the concept.

Consider the following toy example, where the modulo (mod) and exponent (exp) functions are hand-written, but we wish to generate the functions for addition (add), subtraction (sub), division (div) and multiplication (mul):

my_mathlib.py - before code generation
 1"""This library contains some arithmetic functions"""
 2
 3def mod(x, y):
 4    return x % y
 5
 6
 7# <<? common_math_funcs [["add", "+"], ["sub", "-"], ["div", "/"], ["mul", "*"]] ?>>
 8# <<? /common_math_funcs ?>>
 9
10
11def exp(number, exponent):
12    return number ** exponent

Note the highlighted lines - these identify the start- and end of a snippet. Snippets are regions of auto-generated code in an otherwise hand-written source file. In this case, it is referencing some snippet called common_math_funcs, when gcgen is run, it will resolve this name to a function (an implementation) of the snippet, which defines, through code, what the output should be.

The following configuration file implements the snippet, starting at line 5, and tells gcgen to process the my_mathlib.py, expanding any snippets, by defining the gcgen_parse_files hook in line 27-30:

gcgen_conf.py – gcgen configuration file
 1from gcgen.api import Section, Scope, Json, snippet
 2
 3
 4# Snippets are annotated `@snippet("<name>")`
 5@snippet("common_math_funcs")
 6def gen_arithmetic_ops(sec: Section, s: Scope, val: Json):
 7    first = True
 8    for lbl, op in val:
 9        if not first:
10            sec.newline()
11            sec.newline()
12        else:
13            first = False
14        sec.emitln(f"def {lbl}(x, y):")
15        sec.indent()
16        sec.emitln(f"return x {op} y")
17        sec.dedent()
18
19
20def gcgen_parse_files():
21    # tell gcgen which files to process for snippets
22    # inside of this directory.
23    return ["my_mathlib.py"]

Given this configuration file, running gcgen changes my_mathlib.py to have the following contents:

my_mathlib.py - after code generation
 1"""This library contains some arithmetic functions"""
 2
 3
 4def mod(x, y):
 5    return x % y
 6
 7
 8# <<? common_math_funcs [["add", "+"], ["sub", "-"], ["div", "/"], ["mul", "*"]] ?>>
 9def add(x, y):
10    return x + y
11
12
13def sub(x, y):
14    return x - y
15
16
17def div(x, y):
18    return x / y
19
20
21def mul(x, y):
22    return x * y
23
24
25# <<? /common_math_funcs ?>>
26
27
28def exp(number, exponent):
29    return number**exponent

This example is a toy example, it demonstrates the basic concept, but cannot really illustrate the potential of code generation. As you come to understand scope and how subdirectories in your projects can contain their own gcgen_conf.py files which extend the scope, define new snippets or override the implementation of existing ones,

A high-level introduction to gcg

Snippets & Generators

gcg supports two forms of code generation, snippets and generators, both generate code using a section, which provides a convenient API for writing output, lines and handling indentation.

Snippets, as shown above, mark regions within a file, whose contents are auto-generated and will be rewritten each time gcgen is run.

By contrast generators are functions in a gcgen file (see next paragraph), which are mostly useful when generating files, where the number and naming of the files is dynamic and depends on some input data or model.

Each snippet and generator are passed two objects, a section as mentioned before, and a scope.

Scope

The scope functions as a lexical scope or a layered dictionary. When looking up a variable in a scope, the key is first checked against the inner-most scope, the instance you are holding. If the key is not found, the key is then searched for in the parent scope, and so on, until reaching the top scope.

In essence this allows a derived/child scope to set values in its own scope without impacting the parent scope(s), while still being able to read values from the parent scopes.

Each subdirectory containing a gcgen file creates a new scope, and the configuration file may modify the scope’s definitions by implementing the gcgen_scope_extend function.

Each generator and each file being parsed for snippets gets their own scope.

The gcgen file(s)

Both snippets and generators are defined in a gcgen file. These files are always named gcgen_conf.py. When running gcgen inside a project, these files are imported and their definitions are used.

These files define snippets, generators, mark which files in a directory should be parsed (meaning, scanned for snippets to expand) and they manage the scope, which is passed along to each snippet and generator when called.

The compile process

The gcgen compile process traverses down the project hierarchy of directories, importing each directory’s gcgen file as they encounter them.

After importing a gcgen file, they create a new child scope and run the gcgen_scope_extend hook, if defined, which can add or remove variables from the scope.

Furthermore, the compiler also maintains a mapping of snippet definitions. Any snippets defined in the gcgen file extend the mapping, possibly overriding similarly named snippets from parent directories. These snippet definitions, in turn, are available for use in any file contained in the current directory, or any of its subdirectories.

Snippet and generator execution is done in a depth-first manner, meaning the snippets and generators in the deepest subdirectories are executed first, and any snippets and generators in the project root directory are executed last.