I agree completely — the current situation that mixes docstrings and literate comments is unfortunate. This is a copy of an email I sent recently to a few people working on documentation on this topic:
As I’ve said to some of you in the past, I think it helps to distinguish two fairly separate use cases when talking about documentation tools.
One is attaching metadata to bits of code, in order to assemble documentation for APIs, provide code completion, in-buffer hints, etc; this is commonly done by documenting individual functions using special code comments (“docstrings”) and putting these comments together into a larger document.
Another one is writing manuals, tutorials, and books, possibly in literate programming style, where the focus is on interleaving code and prose.
There can be interdependencies between these systems. For example, Python folks commonly use small examples of input and output within a function’s docstring — a small form of literate programming embedded in doc comments, called doctests — and conversely Sphinx has a feature (autodoc) that will pull the docstring of a function from its file and include it in a reStructuredText document.
(In the long run, I’d like to do that with Coq’s tactics: one would put a docstring next to the definition of the tactic, and it would be possible to include that text into the manual just by referring to the tactic’s name. This would open plenty of opportunities: for example, you could display tactic docs and function docstrings in search results, even for user-defined tactics and functions).
I think it’s a feature when docstrings and literate comments are different objects, so I get uneasy when they are handled together, e.g. the way it’s done in Doxygen. OCaml gets away with it because mli files can serve as literate files: you can put signatures in any order and there’s no real code in those files, (in fact, signatures in an mli file are quite similar to autodoc commands in a Sphinx document). On the other end of the spectrum there’s the way GNU projects are documented, with a clear separation between the reference manual and the docstrings (to make this concrete, if you have a copy of Emacs at hand, compare the result of the following two commands:
M-x info-lookup-symbol start-processandM-x describe-function start-process).In Coq, CoqDoc has a hybrid role: basically we write all library documentation as literate files. This makes it hard to retrieve documentation programmatically, which means that I can’t include function docstrings in company-coq, for example. In F* things were done differently (doc comments were part of the AST), which means that F*-mode and other UIs were able to include function documentation reliably (there were issues with the implementation of the comment parsing, however, and eventually it was dropped). In contrast, OCaml’s ecosystem does have a notion of what bit of code docstrings attach to, IIUC, which is very good.