feat(docs): replace lua2dox.lua

Problem:

The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.

Solution:

Re-implement the flow using Lua, LPEG and treesitter.

- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
  - LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
  - LPEG for C parsing (see `scripts/cdoc_parser.lua`)
  - Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
  - Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
   - `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).

1 files changed, 39 insertions, 0 deletions

diff --git a/test/functional/script/text_utils_spec.lua b/test/functional/script/text_utils_spec.lua
new file mode 100644
index 0000000000..c429d306d5
--- /dev/null
+++ b/test/functional/script/text_utils_spec.lua
@@ -0,0 +1,39 @@
+local helpers = require('test.functional.helpers')(after_each)
+local exec_lua = helpers.exec_lua
+local eq = helpers.eq
+
+local function md_to_vimdoc(text)
+  return exec_lua(
+    [[
+    local text_utils = require('scripts/text_utils')
+    return text_utils.md_to_vimdoc(table.concat(..., '\n'), 0, 0, 70)
+  ]],
+    text
+  )
+end
+
+local function test(act, exp)
+  eq(table.concat(exp, '\n'), md_to_vimdoc(act))
+end
+
+describe('md_to_vimdoc', function()
+  before_each(function()
+    helpers.clear()
+  end)
+
+  it('can render para after fenced code', function()
+    test({
+      '- Para1',
+      '  ```',
+      '  code',
+      '  ```',
+      '  Para2',
+    }, {
+      '• Para1 >',
+      '    code',
+      '<',
+      '  Para2',
+      '',
+    })
+  end)
+end)