docs: module-level docstrings (@defgroup) #22498

Problem:
gen_vimdoc.py / lua2dox.lua does not support @defgroup or \defgroup
except for "api-foo" modules.

Solution:
Modify `gen_vimdoc.py` to look for section names based on `helptag_fmt`.

TODO:
- Support @module ?
  https://github.com/LuaLS/lua-language-server/wiki/Annotations#module

3 files changed, 96 insertions, 59 deletions

diff --git a/runtime/doc/treesitter.txt b/runtime/doc/treesitter.txt
index 855fc6bdc6..1f78e4d5d9 100644
--- a/runtime/doc/treesitter.txt
+++ b/runtime/doc/treesitter.txt
@@ -29,42 +29,6 @@ A parser can also be loaded manually using a full path: >lua
     vim.treesitter.require_language("python", "/path/to/python.so")
 <
 ==============================================================================
-LANGUAGE TREES                                       *treesitter-languagetree*
-                                                                *LanguageTree*
-
-As buffers can contain multiple languages (e.g., Vimscript commands in a Lua
-file), multiple parsers may be needed to parse the full buffer. These are
-combined in a |LanguageTree| object.
-
-To create a LanguageTree (parser object) for a buffer and a given language,
-use >lua
-
-    tsparser = vim.treesitter.get_parser(bufnr, lang)
-<
-`bufnr=0` can be used for current buffer. `lang` will default to 'filetype'.
-Currently, the parser will be retained for the lifetime of a buffer but this
-is subject to change. A plugin should keep a reference to the parser object as
-long as it wants incremental updates.
-
-Whenever you need to access the current syntax tree, parse the buffer: >lua
-
-    tstree = tsparser:parse()
-<
-This will return a table of immutable |treesitter-tree|s that represent the
-current state of the buffer. When the plugin wants to access the state after a
-(possible) edit it should call `parse()` again. If the buffer wasn't edited,
-the same tree will be returned again without extra work. If the buffer was
-parsed before, incremental parsing will be done of the changed parts.
-
-Note: To use the parser directly inside a |nvim_buf_attach()| Lua callback,
-you must call |vim.treesitter.get_parser()| before you register your callback.
-But preferably parsing shouldn't be done directly in the change callback
-anyway as they will be very frequent. Rather a plugin that does any kind of
-analysis on a tree should use a timer to throttle too frequent updates.
-
-See |lua-treesitter-languagetree| for the list of available methods.
-
-==============================================================================
 TREESITTER TREES                                             *treesitter-tree*
                                                                       *TSTree*
 
@@ -221,7 +185,7 @@ Nvim looks for queries as `*.scm` files in a `queries` directory under
 purpose, e.g., `queries/lua/highlights.scm` for highlighting Lua files.
 By default, the first query on `runtimepath` is used (which usually implies
 that user config takes precedence over plugins, which take precedence over
-queries bundled with Neovim). If a query should extend other queries instead
+queries bundled with Nvim). If a query should extend other queries instead
 of replacing them, use |treesitter-query-modeline-extends|.
 
 See |lua-treesitter-query| for the list of available methods for working with
@@ -321,7 +285,7 @@ Use |vim.treesitter.list_directives()| to list all available directives.
 
 TREESITTER QUERY MODELINES                          *treesitter-query-modeline*
 
-Neovim supports to customize the behavior of the queries using a set of
+Nvim supports to customize the behavior of the queries using a set of
 "modelines", that is comments in the queries starting with `;`. Here are the
 currently supported modeline alternatives:
 
@@ -938,6 +902,44 @@ TSHighlighter:destroy({self})                        *TSHighlighter:destroy()*
 ==============================================================================
 Lua module: vim.treesitter.languagetree          *lua-treesitter-languagetree*
 
+
+A *LanguageTree* contains a tree of parsers: the root treesitter parser
+for {lang} and any "injected" language parsers, which themselves may
+inject other languages, recursively. For example a Lua buffer containing
+some Vimscript commands needs multiple parsers to fully understand its
+contents.
+
+To create a LanguageTree (parser object) for a given buffer and language, use:
+
+>lua
+    local parser = vim.treesitter.get_parser(bufnr, lang)
+<
+
+(where `bufnr=0` means current buffer). `lang` defaults to 'filetype'.
+Note: currently the parser is retained for the lifetime of a buffer but
+this may change; a plugin should keep a reference to the parser object if
+it wants incremental updates.
+
+Whenever you need to access the current syntax tree, parse the buffer:
+
+>lua
+    local tree = parser:parse()
+<
+
+This returns a table of immutable |treesitter-tree| objects representing
+the current state of the buffer. When the plugin wants to access the state
+after a (possible) edit it must call `parse()` again. If the buffer wasn't
+edited, the same tree will be returned again without extra work. If the
+buffer was parsed before, incremental parsing will be done of the changed
+parts.
+
+Note: To use the parser directly inside a |nvim_buf_attach()| Lua
+callback, you must call |vim.treesitter.get_parser()| before you register
+your callback. But preferably parsing shouldn't be done directly in the
+change callback anyway as they will be very frequent. Rather a plugin that
+does any kind of analysis on a tree should use a timer to throttle too
+frequent updates.
+
 LanguageTree:children({self})                        *LanguageTree:children()*
     Returns a map of language to child tree.
 
diff --git a/runtime/lua/vim/treesitter/languagetree.lua b/runtime/lua/vim/treesitter/languagetree.lua
index 1bc7971eba..b8b0dd867e 100644
--- a/runtime/lua/vim/treesitter/languagetree.lua
+++ b/runtime/lua/vim/treesitter/languagetree.lua
@@ -1,3 +1,37 @@
+--- @defgroup lua-treesitter-languagetree
+---
+--- @brief A \*LanguageTree\* contains a tree of parsers: the root treesitter parser for {lang} and
+--- any "injected" language parsers, which themselves may inject other languages, recursively.
+--- For example a Lua buffer containing some Vimscript commands needs multiple parsers to fully
+--- understand its contents.
+---
+--- To create a LanguageTree (parser object) for a given buffer and language, use:
+---
+--- <pre>lua
+---     local parser = vim.treesitter.get_parser(bufnr, lang)
+--- </pre>
+---
+--- (where `bufnr=0` means current buffer). `lang` defaults to 'filetype'.
+--- Note: currently the parser is retained for the lifetime of a buffer but this may change;
+--- a plugin should keep a reference to the parser object if it wants incremental updates.
+---
+--- Whenever you need to access the current syntax tree, parse the buffer:
+---
+--- <pre>lua
+---     local tree = parser:parse()
+--- </pre>
+---
+--- This returns a table of immutable |treesitter-tree| objects representing the current state of
+--- the buffer. When the plugin wants to access the state after a (possible) edit it must call
+--- `parse()` again. If the buffer wasn't edited, the same tree will be returned again without extra
+--- work. If the buffer was parsed before, incremental parsing will be done of the changed parts.
+---
+--- Note: To use the parser directly inside a |nvim_buf_attach()| Lua callback, you must call
+--- |vim.treesitter.get_parser()| before you register your callback. But preferably parsing
+--- shouldn't be done directly in the change callback anyway as they will be very frequent. Rather
+--- a plugin that does any kind of analysis on a tree should use a timer to throttle too frequent
+--- updates.
+
 local a = vim.api
 local query = require('vim.treesitter.query')
 local language = require('vim.treesitter.language')
diff --git a/scripts/gen_vimdoc.py b/scripts/gen_vimdoc.py
index 9e9e966627..0164278b34 100755
--- a/scripts/gen_vimdoc.py
+++ b/scripts/gen_vimdoc.py
@@ -1054,17 +1054,18 @@ def main(doxygen_config, args):
 
         fn_map_full = {}  # Collects all functions as each module is processed.
         sections = {}
-        intros = {}
+        section_docs = {}
         sep = '=' * text_width
 
         base = os.path.join(output_dir, 'xml')
         dom = minidom.parse(os.path.join(base, 'index.xml'))
 
-        # generate docs for section intros
+        # Generate module-level (section) docs (@defgroup).
         for compound in dom.getElementsByTagName('compound'):
             if compound.getAttribute('kind') != 'group':
                 continue
 
+            # Doxygen "@defgroup" directive.
             groupname = get_text(find_first(compound, 'name'))
             groupxml = os.path.join(base, '%s.xml' %
                                     compound.getAttribute('refid'))
@@ -1083,33 +1084,39 @@ def main(doxygen_config, args):
                 if doc:
                     doc_list.append(doc)
 
-            intros[groupname] = "\n".join(doc_list)
+            section_docs[groupname] = "\n".join(doc_list)
 
+        # Generate docs for all functions in the current module.
         for compound in dom.getElementsByTagName('compound'):
             if compound.getAttribute('kind') != 'file':
                 continue
 
             filename = get_text(find_first(compound, 'name'))
             if filename.endswith('.c') or filename.endswith('.lua'):
-                xmlfile = os.path.join(base,
-                                       '{}.xml'.format(compound.getAttribute('refid')))
+                xmlfile = os.path.join(base, '{}.xml'.format(compound.getAttribute('refid')))
                 # Extract unformatted (*.mpack).
                 fn_map, _ = extract_from_xml(xmlfile, target, 9999, False)
                 # Extract formatted (:help).
                 functions_text, deprecated_text = fmt_doxygen_xml_as_vimhelp(
-                    os.path.join(base, '{}.xml'.format(
-                                 compound.getAttribute('refid'))), target)
+                    os.path.join(base, '{}.xml'.format(compound.getAttribute('refid'))), target)
 
                 if not functions_text and not deprecated_text:
                     continue
                 else:
-                    name = os.path.splitext(
-                            os.path.basename(filename))[0].lower()
+                    filename = os.path.basename(filename)
+                    name = os.path.splitext(filename)[0].lower()
                     sectname = name.upper() if name == 'ui' else name.title()
+                    sectname = CONFIG[target]['section_name'].get(filename, sectname)
+                    title = CONFIG[target]['section_fmt'](sectname)
+                    section_tag = CONFIG[target]['helptag_fmt'](sectname)
+                    # Module/Section id matched against @defgroup.
+                    #   "*api-buffer*" => "api-buffer"
+                    section_id = section_tag.strip('*')
+
                     doc = ''
-                    intro = intros.get(f'api-{name}')
-                    if intro:
-                        doc += '\n\n' + intro
+                    section_doc = section_docs.get(section_id)
+                    if section_doc:
+                        doc += '\n\n' + section_doc
 
                     if functions_text:
                         doc += '\n\n' + functions_text
@@ -1119,12 +1126,7 @@ def main(doxygen_config, args):
                         doc += deprecated_text
 
                     if doc:
-                        filename = os.path.basename(filename)
-                        sectname = CONFIG[target]['section_name'].get(
-                                filename, sectname)
-                        title = CONFIG[target]['section_fmt'](sectname)
-                        helptag = CONFIG[target]['helptag_fmt'](sectname)
-                        sections[filename] = (title, helptag, doc)
+                        sections[filename] = (title, section_tag, doc)
                         fn_map_full.update(fn_map)
 
         if len(sections) == 0:
@@ -1139,15 +1141,14 @@ def main(doxygen_config, args):
 
         for filename in CONFIG[target]['section_order']:
             try:
-                title, helptag, section_doc = sections.pop(filename)
+                title, section_tag, section_doc = sections.pop(filename)
             except KeyError:
                 msg(f'warning: empty docs, skipping (target={target}): {filename}')
                 msg(f'    existing docs: {sections.keys()}')
                 continue
             if filename not in CONFIG[target]['append_only']:
                 docs += sep
-                docs += '\n%s%s' % (title,
-                                    helptag.rjust(text_width - len(title)))
+                docs += '\n{}{}'.format(title, section_tag.rjust(text_width - len(title)))
             docs += section_doc
             docs += '\n\n\n'