Introduction
snakedown is a command line tool to extract API documentation from a python package. The most popular solutions like Sphinx and Quarto are fairly slow and require a lot of configuration, as well as having poor DevX.
Snakedown is designed to let you use your favourite static site generator to host your Python documentation.
Features include:
- Easily extract both type signatures and docstrings
- lightweight syntax for linking to other objects in docstrings, inspired by Obsidian
- Can link to other Sphinx sites like the
numpydocs - Themes in supported static site generators give you precie control over the look of your docs
It currently supports the following static site generators and themes:
Installation
cargo
Currently the only place where snakedown is distributed to is crates.io. Other places might be considered later, but for how the easiest way to install it is by using cargo install like so:
cargo install snakedown
This will install the latest released version.
If you want to install the latest dev version you can also do that:
cargo install --git https://github.com/savente93/snakedown --branch main --locked
Getting Started
As an example we will go through how to setup a docs site with snakedown using zola and the snakedown theme. In addition to snakedown being installed (see installing)
to follow along with this example you’ll also need zola and git installed. These are not strictly required to use snakedown as you can use it without them
but it is the easiest way.
Finally you’ll also need a python package. For the purposes of this tutorial we will use the following file:
"""A python module to demonstrate the functionality of snakedown."""
def fun():
"""this is a free function called fun!"""
print("I'm having fun!")
Class Foo:
"""this is the docstring for the Foo class."""
def bar(self) -> int:
"""this is a method called bar on the [[ test_pkg.Foo ]] class"""
return 42
Setup
First make a new folder for your python project. We’ll call both the project and the package pkg as this is common for python projects:
mkdir pkg
now first create the folders for the python module and the documentation:
mkdir pkg/{pkg,docs}
After that create a file called __init__.py in the pkg/pkg folder to mark it as a package, and paste
the above python code into it.
the folder structure of your new directory should look like this:
❯ eza --tree -L 2 --git-ignore -A pkg
pkg
├── docs
└── pkg
└── __init__.py
Note
ezais just anlsalternative we use here to print the directory structure, you don’t need it for this tutorial.
Zola site
We’ll now first setup the site that the output will be added to. For this tutorial we will use Zola, but you can use any compatible SSG here, just make sure to setup the site however your ssg expects it.
Move into the directly and initialise the site:
cd pkg/docs
zola init
zola will now ask you a few questions about how you’d like to set up your site. For the purposes of this tutorial you can just accept all of the defaults.
after this the structure of your project should look like this:
❯ eza --tree -L 2 --git-ignore -A pkg
pkg
├── docs
│ ├── config.toml
│ ├── content
│ ├── sass
│ ├── static
│ ├── templates
│ └── themes
└── pkg
└── __init__.py
From the root of your zola site (the docs folder) add the snakedown theme to your site:
git clone https://github.com/savente93/zola-snakedown-theme.git themes/snakedown
Also make sure to enable the theme in zola’s config.toml by adding the following line:
theme = "snakedown"
Snakedown
Now that the zola site is setup, we just need to setup snakedown itself. In this case all you have to do is create a snakedown.toml file in the root of the project (the top most pkg folder). You can also create a pyproject.toml file there if you prefer.
the snakedown.toml file should have the following contents:
api_content_path = "api/"
site_root = "docs"
pkg_path = "pkg"
ssg = "Zola"
Because we now have setup snakedown with all the information you don’t even have to provide snakedown with any arguments, since it will take the information from the config file. You can just run it like this:
snakedown
Now you should see files being created:
❯ eza --tree -L 4 --git-ignore -A
.
├── docs
│ ├── config.toml
│ ├── content
│ │ └── api
│ │ ├── pkg.foo.Foo._private.md
│ │ ├── pkg.foo.Foo.bar.md
│ │ ├── pkg.foo.Foo.md
│ │ ├── pkg.foo.fun.md
│ │ └── pkg.foo.md
│ ├── sass
│ ├── static
│ ├── templates
│ └── themes
├── pkg
│ ├── __init__.py
│ └── foo.py
└── snakedown.toml
If you want you can still override configurations in your config files by providing cli arguments:
snakedown pkg docs tmp
after which it should look like this:
❯ eza --tree -L 4 --git-ignore -A
.
├── docs
│ ├── config.toml
│ ├── content
│ │ ├── api
│ │ │ ├── pkg.foo.Foo.bar.md
│ │ │ ├── pkg.foo.Foo.md
│ │ │ ├── pkg.foo.fun.md
│ │ │ └── pkg.foo.md
│ │ └── tmp
│ │ ├── pkg.foo.Foo.bar.md
│ │ ├── pkg.foo.Foo.md
│ │ ├── pkg.foo.fun.md
│ │ └── pkg.foo.md
│ ├── sass
│ ├── static
│ ├── templates
│ └── themes
├── pkg
│ ├── __init__.py
│ └── foo.py
└── snakedown.toml
Writing Content
Snakedown can extract a variety of information from you python package, including type signatures and docstrings. The output is structured similarly to that of Sphinx, meaning that all “objects” (e.g. functions, classes and modules) get their own page in a flat directory structure.
Linking
Snakedown introduces a lightweight syntax for linking to other objects that is inspired by the one that Obsidian uses, namelly [[ fully.qualified.name ]] this will then get turned into the correct link in whatever format your supported static site generator expects. For the moment only fully qualified references are supported, meaning that you can only reference them via their full import path. You can use this syntax anywhere in your docstrings.
You can also provide some optional display text by using the | character like so: [[ foo.bar.baz | The baz module]] which will be used as the text for the generated link. If you do not provide any, the reference target name will be used (i.e. [[ foo.bar ]] will be changed to [foo.bar](foo.bar.md) but [[ foo.bar | the bar module]] would be changed to [the bar module](foo.bar.md))
Jupyter Notebooks
Snakedown now supports including the output of jupyter notebooks in your documentation. Currently only python notebooks are supported. This is more out of consistency because python is the only language we currently parse, so it doesn’t make much sense to allow for notebooks in other languages, however, this could change in the future.
The output of a notebook being processed by snakedown will be a markdown file with the same file stem as the notebook. This will contain all the markdown cells included as is. Code cells will be included wrapped in a python code block. Output of supported formats will be included as well. If there is image data, this will be included as a colocated file which will be referenced in the output.
We plan to add support for expanding references like in the section above, though likely only markdown cells will be expanded.
Why is my Jupyter notebook output not showing up?
The Jupyter format can actually output a surprisng amount of different kinds of output, many of those we didn’t have an example case for, and as a general rule we don’t implement things we can’t verify the use of. However, if you have something that produces output we don’t support and are willing to share (a simplified version) so we can make sure it works properly, please open a feature request. The media types that our dependency support are listed here.
Using supported Static Site Generators
Snakedown knows how to produce output that some static site generators can work with, and provides at least one compatible theme to make customisation easier. Bellow are the static sites we can work with at the moment and some information on how to customise them.
Plain Markdown
Not technically a site generator, but in case you just want the output of snakedown for whatever reason without having to scaffold a static site around it, you can use this option. As Markdown doesn’t support themes, or other types of customisation, this option is pretty basic and doesn’t come with a theme.
Zola
Zola is a static site generator written in rust and snakedown was originally written with Zola in mind. Snakedown will put the output in the content section of the site you provide, so once the output is generated you can develop and distribute your site like any other zola site.
Many themes allow customisation though the config.toml file of the zola site, or by modifying or extending the templates they provide. Please see the zola documentation or the theme documentation for more detail.
Currently snakedown only dumps the content and doesn’t do much pre-processing, however in future we may provide output containing shortcodes so that the site can more easily customise them.
Supported themes
Because the output of snakedown is just content that get’s added to an existing site, and we don’t yet produce shortcodes, you can currently use snakedown with any theme. However, the following themes are made to work with the output of snakedown with minimal configuration and should be kept up to date with new snakedown features.
- Snakedown The default snakedown theme for Zola, developed by the Snakedown devs. Based on the PyData Sphinx Theme.
Other compatible themes may be added here. If you know of one, please submit a PR to add it to this list!
Configuration
There are a few was that you can configure the behaviour of snakedown. The main way you’ll probably want to do this is via one of the configuration file formats detailed below to document what arguments need to be passed and save you a lot of typing. However, most things that can be configured via a file, can also be configured (and overridden) by the cli.
In general options can be provided and get resolved at run time according to the following precedence from highest to lowest:
- cli arguments
snakedown.toml[tools.snakedown]table in apyproject.tomlfile- program defaults
For a complete list of all options and their explanation see the options page.
Formats
snakedown.toml
This is a toml file with configuration options that snakedown will use. It can be located either in the current working directory or in $HOME/.config/snakedown/snakedown.toml
And example file filled with all the defaults can be found here
The possible values are documented below.
pyproject.toml
This one will only be found if it is in the current working directory. It has the same options and behavior as snakedown.toml but the tables should be prefixed with tool.snakedown like so:
[tool.snakedown.externals]
builtins = {name = "Python", url = "https://docs.python.org/3/"}
Options
pkg_path
The relative path from the current working directory to where the python package you want to document is located. This is the folder that snakedown will try to crawl and extract all documentation from.
Default value: .
skip_undoc
Whether to skip undocumented objects (meaning ones that don’t have a docstring). If you set this to true, no page will be generated for these objects. If it is set to false, an empty page will be generated with just the signature.
Default Value: true
api_content_path
The relative path from the site root (see site_root) to where the api docs are located. This is tracked separately because this needs to be reflected in the generated links (they need to be relative to the site root, not the current working directory for example).
The full place where the docs will get placed is determined by joining api_content_path to site_root along with whatever intermediate path is required for the SSG (e.g. for zola, this is content). Meaning that by default the docs will be placed in ./docs/api/ for makrdown and ./docs/content/api/ for zola.
Default value: api
notebook_path
A path from the current working directory to a folder where notebooks are located that you want to be included.
Default value: examples
notebook_content_path
The relative path from the site root (see site_root) to where the output of the jupyter notebooks should be located, similar to (api_content_path)[#apicontentpath]
Default value: user-guide
site_root
The relative path from the current working directory where root of the docs site is located. For example when using a
Zola site, this should be the path to the folder containing your contents folder and config.toml file.
Default value: docs
skip_private
Whether to skip private objects (meaning ones whose name starts with an _ e.g. _foo) If you set this to true, no page will be generated for these objects. If it is set to false, an empty page will be generated with just the signature.
Default Value: true
ssg
Which static site the output should be compatible with.
Default value:Markdown
Possible values:
MarkdownZola
exclude
A list of paths that should be explicitly not documented by snakedown. Paths in this list will be skipped regardless of the values of skip_undoc and skip_private and can be either relative or absolute.
externals
Similar to Sphinx, snakedown can parse references to external documentation by parsing a file called objects.inv which sphinx produces. External references mentioned in this table will be retriefec, cached and parsed so that you can refer to them in your docstrings the same way you can to internal objects. The key (in the example below that would be builtins) is not used for anything other than defining the table. The url should point to the location on the internet where the objects.inv file is located.
Default value:
builtins = {name = "Python", url = "https://docs.python.org/3/"}
render
Not all though some renderers take parameters to modify their behavior. You can set those parameters in this table like so:
[render.zola]
use_shortcodes = true
The markdown renderer currently does not have any options.
Customisation
You likely already know how to customise things in your favourite static site generator. Therefore our aim has been that the customisation should happen there. If you set up your config correctly then you should need to do anything with snakedown for customisation.
Here we will however give an overview of the things we generate so that you can customise them in your ssg so that you know where to look. You may additionally want to consult the documentation of the theme itself. For now this is quite sparse, but we do tend to add to this as we develop features!
Contributing to snakedown
Thanks for wanting to contribute! There are many ways to contribute and we appreciate any level you’re willing to do. This repository is just for snakedown itself if the issue, request or contribution is to do with the theme, please report that in the specific repository.
Feature Requests
Need some new functionality? You can let us know by opening an issue. Please include a good description of what you would like and what you are hoping to accomplish with it. If you have considered alternatives or current work arounds it is often also helpful to describe those.
In general if your feature request is well written and complete it vastly increaces the chances it will be picked up soon. In general we don’t ask for information for no reason, so if you omit things that may slow things down.
Bug Reports
Please let us know about what problems you run into, whether in behavior or ergonomics of API. You can do this by opening an issue.
When you do so please include the version you were using as well good descriptions of what you were doing that may have triggered the error and what you expect to happen instead (unless it is trivial for example when you are reporting a crash)
In general if your bug report is well written and complete it vastly increaces the chances it will be picked up soon. In general we don’t ask for information for no reason, so if you omit things that may slow things down.
Pull Requests
Looking for an idea? Check our issues. If the issue looks open ended, it is probably best to post on the issue how you are thinking of resolving the issue so you can get feedback early in the process. We want you to be successful and it can be discouraging to find out a lot of re-work is needed. We encourage you to reach out and ask questions early in the process if you are uncertain.
Don’t know where to start? check out the issues labeled good first issue and mentoring available. In general these are beginner friendly with more detailed instructions.
Already have an idea? It might be good to first create an issue to propose it so we can make sure we are aligned and lower the risk of having to re-work some of it and the discouragement that goes along with that.
Process
As a heads up, we’ll be running your PR through the following gauntlet:
- warnings turned to compile errors
cargo testrustfmtclippyrustdoctaplo(toml formatter)codecovcommittedas we use Conventional commit style. Ideally the commit message shouldn’t just say what was done but also why.typosto check spelling
In generally you can make sure these are okay by installing the pre-commit hooks in this repository. Not everything can be checked automatically though.
We also don’t allow “TODO” comments in the code unless they also link to an issue, since TODO comments usually get forgotten and overlooked.
We request that the commit history gets cleaned up so that that commits are atomic, meaning they are complete and have a single responsibility. A complete commit should build, pass tests, update documentation and tests, and not have dead or commented out code.
PRs should tell a cohesive story, with refactor and test commits that keep the fix or feature commits simple and clear.
We understand having a clean history requires more advanced git skills; feel free to ask us for help! We might even suggest where it would work to be lax.
We also understand that editing some early commits may cause a lot of churn with merge conflicts which can make it not worth editing all of the history. One way to do this is to just keep one big temporary commit (or a bunch of temporary commits) while you prototype until things are the way you want them to be, soft reset all the commits (or move all changes to a new branch) and then re-commit things in atomic commits one by one. This also gives you a good opportunity to do a self review!
Coverage
Coverage in Rust can be a bit fineky at times, and additionally coverage doesn’t always tell the whole story, so we usually don’t enforce hard limits on coverage. For example llvm-cov marking a file that only contains a )? when returning a Result that will (almost) never error is not uncommon. We don’t think there’s much value in enforcing that these lines be covered, so if you miss those for example, that’s okay. That being said, we do like to keep our coverage high, so if you don’t cover something, please have good explanation as to why!
Organisation
For code organization, we recommend
- Grouping
implblocks next to their type (or trait) - Grouping private items after the
pubitem that uses them.- The intent is to help people quickly find the “relevant” details, allowing them to “dig deeper” as needed. Or put another way, the
pubitems serve as a table-of-contents. - The exact order is fuzzy; do what makes sense
- The intent is to help people quickly find the “relevant” details, allowing them to “dig deeper” as needed. Or put another way, the
Benchmarking
Benchmarking/profilling to improve performance is tricky, because so many smart and complicated tricks are already done for you by compilers/operating systems/hardware etc. meaning that you have to make sure you work together with them instead of against them. Performance measuring and improving is part science, part art. To accommodate for this we have designed our benchmarks with the following considerations:
- Results of the benchmarks are intentionally not saved because there is an inherent amount of irreproducibility to them
- While the results are not saved, the setup is. All benchmarks should use exactly the config used in
benchmark-config.tomlfor the snakedown configuration and thebenchprofile for compilation (unless you are benchmarking different profiles). If you modify any of these you’ll have to run new baselines! - Remember that benchmarks are inherently hardware bound, so do not pay too much attention to the individual numbers, and if you are bench marking yourself, remember to always run the baselines yourself.
- There are currently two benchmark commands in the
pixi.tomlwe use:benchthis runs the citereon benchmark. Use this if you want to see if your changes had an impact on performanceflamegraphRuns the benchmark config undercargo flamegraphwhich will produce a graph that can help you gain insight into which parts of the code need optimising. Knowing how to read this graph can be tricky at first. See the flamegrpah repo for more information on this
Dev tips
We use pixi to manage installation of external tools, and as a cross platform task runner. Though originally developed for Python, it has access to conda-forge where many tools are already available, as well as making sure we use the correct versions through it’s lockfile.
Though not necessary, as you can install the tools yourself and run the commands listed in the pixi.toml file yourself, we recommend accessing our workflows through pixi. You can install all the dependencies with the command pixi install. The default environment contains everything you might need and as a dev this is almost certainly what you want.
You can see which tasks you can run with the command pixi task run. This should give you an overview of all common workflows. Do note that if you install everything through pixi, then you will need to either activate the environment with pixi shell or run tasks (including things like cargo run) through pixi like so: pixi run test or pixi run cargo run.
In case you would like to install the tools yourself, below is a list of tools we use:
pre-commitThis will run lints when you try to commit so you don’t fail CI tasks unnecessarily. Make sure to activate the hooks by runningpre-commit installafter you clone the repo.baconA runner that will watch your files and run checks, tests, linting etc. when they change. Very useful while developing for fast feedback cycles.ghCan be used this to quickly open PRs when done working locally and make sure they aren’t duplicated. Quite convenient, but not necessary.- While our CI will test the outputs with all generators we support, and therefore you don’t technically have to install them, it makes local development a lot easier. Currently we support the following formats:
Also keep in mind not all our rules have to be met at every single stage. It is totally allowed to iterate/prototype until you are happy with things, and then clean up after!
Troubleshooting
My unit tests are passing but the integration test is failing, what’s going on?
Make sure that you’ve initialized the git submodules. We use submodules to include the supported themes for the SSGs in our repo. If you’ve already cloned the repo, you can do so by executing the command git submodule update --recursive --init. If you haven’t you can make sure this is done correctly by cloning with the --recursive option like so: git clone --recursive https://github.com/savente93/snakedown
Publishing
For this project we have a release-plz action setup. This gets updated automatically, and to release to all the places we distribute too all you have to do is edit and merge that release PR.