proposal: define module comment convention for module documentation

[dmitshur]

Note: This was a draft of a proposal that I decided against proposing at the time. I'm posting it at this time to share information and considerations relevant to this topic. See comment below for more information about why the proposal was ultimately not started.

Abstract

There is no standard way to document Go modules. I propose defining the convention of a “module comment”, a comment immediately preceding the module statement in the go.mod file with semantics similar to that of a package comment, and using it for this purpose.

Background

There is well-established convention for how to document a single Go package. It is done with package comments. A package comment is a comment preceding the package clause, with no blank line separating it. For example:

// Package bytes implements functions for the manipulation of byte slices.
// It is analogous to the facilities of the strings package.
package bytes

This convention is described in detail in Effective Go and Go Code Review Comments.

A package comment can be very detailed (for example, see encoding/gob or cmd/go), but a cleaned version of the first sentence is the synopsis (computed with doc.Synopsis), which can be used in contexts where a short description is preferred, such as when many packages are listed on a page (e.g., https://golang.org/pkg/#stdlib).

Go 1.11 has added preliminary support for Go modules. They are described as:

A module is a collection of related Go packages. Modules are the unit of source code interchange and versioning.

At this time, there is no standardized approach for documenting Go modules.

Proposal

The go.mod file contains information at the module scope. It has a module statement. It allows line comments.

I propose following a pattern similar to package comments, and defining a “module comment” to be a comment preceding the module statement, with no blank line separating it. Here are a few simple examples of go.mod files with a module comment to illustrate the idea:

// Module sync provides Go concurrency primitives in addition to the ones
// provided by the language and "sync" and "sync/atomic" packages.
module golang.org/x/sync

go 1.12

// Module text holds supplementary Go libraries for text processing, many involving Unicode.
module golang.org/x/text

require golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e

// Module tools holds various packages and tools
// that support the Go programming language.
//
// Some of the tools, godoc and vet for example,
// are included in binary Go distributions.
//
// Others, including the Go guru and the test
// coverage tool, can be fetched with go get.
//
// Highlights of functionality provided:
//
// • a type-checker for Go
//
// • an implementation of the Static Single Assignment form (SSA)
// representation for Go programs
//
module golang.org/x/tools

go 1.11

require (
	golang.org/x/net v0.0.0-20190620200207-3b0461eec859
	golang.org/x/sync v0.0.0-20190423024810-112230192c58
	golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7
)

The cleaned version of the first sentence of the module comment is the synopsis, which can be used in contexts where a short description is preferred. The synopsis of the tools module comment above is “Module tools holds various packages and tools that support the Go programming language.”

The entire module comment is the complete module documentation. It can be used in contexts where a detailed description is preferred, for example when viewing the module on a module discovery website (such as #33654). It would use the same semantics as Go package documentation, and the same rules could be used to render it into HTML form. This is documented and implemented in doc.ToHTML.

See the following as an example of what very comprehensive module documentation can look like when rendered to HTML:

Integration with go list

The go list command can be used to list packages or modules.

The -f flag specifies an alternate format for the list, using the syntax of package template. The default output is equivalent to -f '{{.ImportPath}}'. The struct being passed to the template is:

type Package struct {
	ImportPath    string   // import path of package in dir
	Doc           string   // package documentation string
	...
}

The Doc field contains the package synopsis. It makes it possible to see the description of packages. For example:

$ go list -f '{{.ImportPath}} - {{.Doc}}' math/...
math - Package math provides basic constants and mathematical functions.
math/big - Package big implements arbitrary-precision arithmetic (big numbers).
math/bits - Package bits implements bit counting and manipulation functions for the predeclared unsigned integer types.
math/cmplx - Package cmplx provides basic constants and mathematical functions for complex numbers.
math/rand - Package rand implements pseudo-random number generators.

The -m flag causes go list to list modules instead of packages. The -f flag still specifies a format template applied to a Go struct, but now a Module struct. A Doc field can be added to it if the module comment convention is established:

type Module struct {
	Path     string       // module path
	Version  string       // module version
	Doc      string       // module documentation synopsis
	...
}

It becomes possible, for example, to get a better sense of the modules involved in a given project by running:

$ go list -m -f '{{.Path}} - {{.Doc}}’ all
cloud.google.com/go - Module go holds packages used to access Google Cloud Services.
go4.org - Module go4.org is a collection of packages for Go programmers.
golang.org/x/build - Module build holds the source for various packages and tools that support Go's build system and the development of the Go programming language.
golang.org/x/crypto - Module crypto holds supplementary Go cryptography libraries.
golang.org/x/text - Module text holds supplementary Go libraries for text processing, many involving Unicode.

Rationale

Relationship with package documentation and README files

It's important to think about what is good information, if any, to include and not to include in a module comment before creating any convention.

Multi-package modules

If a module has n packages, and they are all well documented (via package comments), what is left for the module documentation?

In thinking about this, I’ve found that a short descriptive sentence works well for most modules that contain n > 1 packages. This is sufficient for the synopsis to be meaningful. Additional module documentation details are best left to the discretion of the module authors.

A repository maps to at least one module, and these repository descriptions lead to good module synopses. Multi-module repositories will want to use more narrow module descriptions.

On the golang.org website, there is a page that lists prominent golang.org/x sub-repositories. Each one comes with a short description of the sub-repository:

With minor modifications, these descriptions are good module synopses.

On GitHub, GitLab, Bitbucket, and many other VCS hosting services, repository metadata often includes a “description” field. Repository authors use it to describe a repository as a whole.

(A Go repository on GitHub with a description.)

(A Go repository on GitLab with a description.)

These descriptions lead to good module synopses.

Single-package modules

When a module contains a single Go package at the root, the value of documenting the module is diminished. Such modules may choose to leave out module documentation, and users may look into package documentation instead.

README files

If a repository has a good README, what is left for module documentation?

The README file is a very popular language-agnostic convention for providing information about projects, commonly placed at the root of a repository. Go repository READMEs commonly include a section that describes the repository, and additional optional sections such as:

• Project-specific warnings, disclaimers, or important notes
• Images, badges, videos, links
• How to install Go or set up a working environment
• How to download/install
• How to test
• How to run locally
• How to deploy to production
• How to regenerate, install needed developer-only tools
• How to contribute, report issues, send patches
• License used

READMEs can be written in one of many different formats (plain text, various flavours of Markdown, reStructuredText, many more). They do not follow an exact template, so the description of the repository may follow after various headings, images, badges, other sections or notes. In general, READMEs are meant primarily for humans, and not for parsing by tools.

In contrast, module comment is a Go-specific convention for documenting a Go module, and can assume general knowledge of the Go ecosystem. As a result, they can be brief, targeted on describing just the Go module, with an easy to extract high-quality synopsis.

Authors who wish to document the Go module specifically should prefer using a module comment. README files can be used in parallel to mirror the module description and provide additional language-agnostic project information.

Some tools have been created that help generate (in full or in part) Go repository READMEs by accessing Go package documentation. Such tools can be modified to make additional use of Go module documentation.

Line comments

The go.mod file syntax allows only for // line comments, not /* */ block comments.

It’s possible to use multiple // line comments in sequence to write a module comment of arbitrary length. Line comments may be more tedious to write by hand, but many text editors make this process easier. Line comments do not impose any restrictions on the content of the comment, in contrast to block comments which preclude the inclusion of the character sequence “*/”.

Compatibility

The introduction of module comments does not have backwards compatibility concerns, because it uses existing support for line comments in go.mod files. If some existing go.mod files contain line comments preceding the module statement that are not meant to be interpreted as a module comment, a blank line separating them from the module statement can be used to resolve that.

Implementation

If the proposal is accepted, Dmitri Shuralyov plans to do the implementation in the cmd/go command.

Prototype

In order to understand the problem space better and evaluate the proposal more concretely, a simple prototype of the cmd/go integration was implemented and played around with. It has not gone through code review and isn’t expected to before this proposal is accepted.

The prototype patch is available here. You can apply it to a local checkout of the Go tree with:

$ cd ~/gotip && git checkout -b cmdgo-moddoc master
$ curl https://gist.githubusercontent.com/dmitshur/7e055acb298b42549f9a7d53784545c9/raw/3f9cd5d87ca19dbfa7beb0a658a3650865b11611/cmdgo-moddoc.patch | git am
$ cd src && ./make.bash

Afterwards, you can try adding a module comment to the go.mod file of the main module, and using ~/gotip/bin/go list -m -f ‘{{.Doc}}’ to see its synopsis.

Open issues

For non-main packages, the package comment conventionally begins with “Package name …”. This is documented in Go Code Review Comments and checked in golint via its lintPackageComment check.

It may be a good idea to establish a similar pattern for module comments, and recommend they begin with “Module name …”. Modules do not have a well-established concept of a name, only path. A module name may be derived from the module path by first trimming any major version components (v2, v3, etc.), then taking the base of the module path. This is the process used when determining the binary name for commands (see load.DefaultExecName). For example:

example.com/mycmd/v2 -> mycmd
github.com/russross/blackfriday/v2 -> blackfriday
github.com/google/go-github/v24 -> go-github

This leads to module comments such as:

// Module blackfriday provides a Markdown parser and renderer.
module github.com/russross/blackfriday/v2

// Module go-github provides a client library for accessing the GitHub API v3.
module github.com/google/go-github/v24

An alternative is to not strip the vN component and include it in the module name:

// Module blackfriday/v2 provides a Markdown parser and renderer.
module github.com/russross/blackfriday/v2

// Module go-github/v24 provides a client library for accessing the GitHub API v3.
module github.com/google/go-github/v24

An alternative is to use the entire module path:

// Module github.com/russross/blackfriday/v2 provides a Markdown parser and renderer.
module github.com/russross/blackfriday/v2

// Module github.com/google/go-github/v24 provides a client library for accessing the GitHub API v3.
module github.com/google/go-github/v24

Another alternative is to not recommend starting module comments with “Module name …” pattern, but rather follow the additional freedom granted to package comments for commands (package main):

// Blackfriday provides a Markdown parser and renderer.
module github.com/russross/blackfriday/v2

// Go-github is a client library for accessing the GitHub API v3.
module github.com/google/go-github/v24

In this proposal, I have so far chosen to use the first option of requiring a “Module name …” prefix, which is the most restrictive. The rationale for that is that it’s easier to start with the most restrictive form, and after gathering data from real world usage, decide to loosen the format if there’s evidence supporting that change. On the other hand, starting with the least restrictive format may make it harder to eventually make the format more strict, if that ends up being preferable.

[dmitshur]

I am closing this issue because this is a proposal draft that I am withdrawing at the time; I don't actually propose we make this change now. It was posted to make information and relevant considerations available.

The main issue with this proposal turned out to be the problem it set out to solve, namely:

There is no standard way to document Go modules.

It became less clear what it meant to "document a module". A module can be thought of as a versioned delivery mechanism for Go packages. Go packages are documented, so it's not clear what is left to document at the scope of a module.

In order to revive this proposal, there needs to be an answer to the question of "why document a Go module."