Merge #15585 refactor: move vim.lsp.diagnostic to vim.diagnostic

## Overview

- Move vim.lsp.diagnostic to vim.diagnostic
- Refactor client ids to diagnostic namespaces
- Update tests
- Write/update documentation and function signatures

Currently, non-LSP diagnostics in Neovim must hook into the LSP subsystem. This
is what e.g. null-ls and nvim-lint do. This is necessary because none of the
diagnostic API is exposed separately from the LSP subsystem.

This commit addresses this by generalizing the diagnostic subsystem beyond the
scope of LSP. The `vim.lsp.diagnostic` module is now simply a specific
diagnostic producer and primarily maintains the interface between LSP clients
and the broader diagnostic API.

The current diagnostic API uses "client ids" which only makes sense in the
context of LSP. We replace "client ids" with standard API namespaces generated
from `nvim_create_namespace`.

This PR is *mostly* backward compatible (so long as plugins are only using the
publicly documented API): LSP diagnostics will continue to work as usual, as
will pseudo-LSP clients like null-ls and nvim-lint. However, the latter can now
use the new interface, which looks something like this:

```lua
-- The namespace *must* be given a name. Anonymous namespaces will not work with diagnostics
local ns = vim.api.nvim_create_namespace("foo")

-- Generate diagnostics
local diagnostics = generate_diagnostics()

-- Set diagnostics for the current buffer
vim.diagnostic.set(ns, diagnostics, bufnr)
```

Some public facing API utility methods were removed and internalized directly in `vim.diagnostic`:

* `vim.lsp.util.diagnostics_to_items`

## API Design

`vim.diagnostic` contains most of the same API as `vim.lsp.diagnostic` with
`client_id` simply replaced with `namespace`, with some differences:

* Generally speaking, functions that modify or add diagnostics require a namespace as their first argument, e.g.

  ```lua
  vim.diagnostic.set({namespace}, {bufnr}, {diagnostics}[, {opts}])
  ```

   while functions that read or query diagnostics do not (although in many cases one may be supplied optionally):

   ```lua
   vim.diagnostic.get({bufnr}[, {namespace}])
   ```

* We use our own severity levels to decouple `vim.diagnostic` from LSP. These
  are designed similarly to `vim.log.levels` and currently include:

  ```lua
  vim.diagnostic.severity.ERROR
  vim.diagnostic.severity.WARN
  vim.diagnostic.severity.INFO
  vim.diagnostic.severity.HINT
  ```

  In practice, these match the LSP diagnostic severity levels exactly, but we
  should treat this as an interface and not assume that they are the same. The
  "translation" between the two severity types is handled transparently in
  `vim.lsp.diagnostic`.

* The actual "diagnostic" data structure is: (**EDIT:** Updated 2021-09-09):

  ```lua
  {
    lnum = <number>,
    col = <number>,
    end_lnum = <number>,
    end_col = <number>,
    severity = <vim.diagnostic.severity>,
    message = <string>
  }
  ```

This differs from the LSP definition of a diagnostic, so we transform them in
the handler functions in vim.lsp.diagnostic.

## Configuration

The `vim.lsp.with` paradigm still works for configuring how LSP diagnostics are
displayed, but this is a specific use-case for the `publishDiagnostics` handler.
Configuration with `vim.diagnostic` is instead done with the
`vim.diagnostic.config` function:

```lua
vim.diagnostic.config({
    virtual_text = true,
    signs = false,
    underline = true,
    update_in_insert = true,
    severity_sort = false,
}[, namespace])
```

(or alternatively passed directly to `set()` or `show()`.)

When the `namespace` argument is `nil`, settings are set globally (i.e. for
*all* diagnostic namespaces). This is what user's will typically use for their
local configuration. Diagnostic producers can also set configuration options for
their specific namespace, although this is generally discouraged in order to
respect the user's global settings. All of the values in the table passed to
`vim.diagnostic.config()` are resolved in the same way that they are in
`on_publish_diagnostics`; that is, the value can be a boolean, a table, or
a function:

```lua
vim.diagnostic.config({
    virtual_text = function(namespace, bufnr)
        -- Only enable virtual text in buffer 3
        return bufnr == 3
    end,
})
```

## Misc Notes

* `vim.diagnostic` currently depends on `vim.lsp.util` for floating window
  previews. I think this is okay for now, although ideally we'd want to decouple
  these completely.

1 files changed, 17 insertions, 0 deletions

diff --git a/scripts/gen_vimdoc.py b/scripts/gen_vimdoc.py
index c8eb71985c..64ed8d61f6 100755
--- a/scripts/gen_vimdoc.py
+++ b/scripts/gen_vimdoc.py
@@ -187,6 +187,23 @@ CONFIG = {
         'module_override': {},
         'append_only': [],
     },
+    'diagnostic': {
+        'mode': 'lua',
+        'filename': 'diagnostic.txt',
+        'section_start_token': '*diagnostic-api*',
+        'section_order': [
+            'diagnostic.lua',
+        ],
+        'files': os.path.join(base_dir, 'runtime/lua/vim/diagnostic.lua'),
+        'file_patterns': '*.lua',
+        'fn_name_prefix': '',
+        'section_name': {'diagnostic.lua': 'diagnostic'},
+        'section_fmt': lambda _: 'Lua module: vim.diagnostic',
+        'helptag_fmt': lambda _: '*diagnostic-api*',
+        'fn_helptag_fmt': lambda fstem, name: f'*vim.{fstem}.{name}()*',
+        'module_override': {},
+        'append_only': [],
+    },
     'treesitter': {
         'mode': 'lua',
         'filename': 'treesitter.txt',