feat(lsp)!: add rule-based sem token highlighting (#22022)

feat(lsp)!: change semantic token highlighting

Change the default highlights used, and add more highlights per token.

Add an LspTokenUpdate event and a highlight_token function.

:Inspect now shows any highlights applied by token highlighting rules,
default or user-defined.

BREAKING CHANGE: change the default highlight groups used by semantic
token highlighting.

7 files changed, 440 insertions, 201 deletions

diff --git a/runtime/doc/lsp.txt b/runtime/doc/lsp.txt
index 7e46698614..0d7e8e7ab4 100644
--- a/runtime/doc/lsp.txt
+++ b/runtime/doc/lsp.txt
@@ -482,6 +482,71 @@ LspSignatureActiveParameter
     Used to highlight the active parameter in the signature help. See
     |vim.lsp.handlers.signature_help()|.
 
+------------------------------------------------------------------------------
+LSP SEMANTIC HIGHLIGHTS                               *lsp-semantic-highlight*
+
+When available, the LSP client highlights code using |lsp-semantic_tokens|,
+which are another way that LSP servers can provide information about source
+code.  Note that this is in addition to treesitter syntax highlighting;
+semantic highlighting does not replace syntax highlighting.
+
+The server will typically provide one token per identifier in the source code.
+The token will have a `type` such as "function" or "variable", and 0 or more
+`modifier`s such as "readonly" or "deprecated." The standard types and
+modifiers are described here:
+https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens
+LSP servers may also use off-spec types and modifiers.
+
+The LSP client adds one or more highlights for each token. The highlight
+groups are derived from the token's type and modifiers:
+  • `@lsp.type.<type>.<ft>` for the type
+  • `@lsp.mod.<mod>.<ft>` for each modifier
+  • `@lsp.typemod.<type>.<mod>.<ft>` for each modifier
+Use |:Inspect| to view the higlights for a specific token. Use |:hi| or
+|nvim_set_hl()| to change the appearance of semantic highlights: >vim
+
+    hi @lsp.type.function guifg=Yellow        " function names are yellow
+    hi @lsp.type.variable.lua guifg=Green     " variables in lua are green
+    hi @lsp.mod.deprecated gui=strikethrough  " deprecated is crossed out
+    hi @lsp.typemod.function.async guifg=Blue " async functions are blue
+<
+The value |vim.highlight.priorities|`.semantic_tokens` is the priority of the
+`@lsp.type.*` highlights. The `@lsp.mod.*` and `@lsp.typemod.*` highlights
+have priorities one and two higher, respectively.
+
+You can disable semantic highlights by clearing the highlight groups: >lua
+
+    -- Hide semantic highlights for functions
+    vim.api.nvim_set_hl(0, '@lsp.type.function', {})
+
+    -- Hide all semantic highlights
+    for _, group in ipairs(vim.fn.getcompletion("@lsp", "highlight")) do
+      vim.api.nvim_set_hl(0, group, {})
+    end
+<
+You probably want these inside a |ColorScheme| autocommand.
+
+Use |LspTokenUpdate| and |vim.lsp.semantic_tokens.highlight_token()| for more
+complex highlighting.
+
+The following groups are linked by default to standard |group-name|s:
+>
+    @lsp.type.class         Structure
+    @lsp.type.decorator     Function
+    @lsp.type.enum          Structure
+    @lsp.type.enumMember    Constant
+    @lsp.type.function      Function
+    @lsp.type.interface     Structure
+    @lsp.type.macro         Macro
+    @lsp.type.method        Function
+    @lsp.type.namespace     Structure
+    @lsp.type.parameter     Identifier
+    @lsp.type.property      Identifier
+    @lsp.type.struct        Structure
+    @lsp.type.type          Type
+    @lsp.type.typeParameter TypeDef
+    @lsp.type.variable      Identifier
+<
 ==============================================================================
 EVENTS                                                            *lsp-events*
 
@@ -516,6 +581,29 @@ callback in the "data" table. Example: >lua
       end,
     })
 <
+
+LspTokenUpdate                                                *LspTokenUpdate*
+
+When a visible semantic token is sent or updated by the LSP server, or when an
+existing token becomes visible for the first time. The |autocmd-pattern| is
+the name of the buffer. When used from Lua, the token and client ID are passed
+to the callback in the "data" table. The token fields are documented in
+|vim.lsp.semantic_tokens.get_at_pos()|. Example: >lua
+
+    vim.api.nvim_create_autocmd('LspTokenUpdate', {
+      callback = function(args)
+        local token = args.data.token
+        if token.type == 'variable' and not token.modifiers.readonly then
+          vim.lsp.semantic_tokens.highlight_token(
+            token, args.buf, args.data.client_id, 'MyMutableVariableHighlight'
+          )
+        end
+      end,
+    })
+<
+Note: doing anything other than calling
+|vim.lsp.semantic_tokens.highlight_token()| is considered experimental.
+
 Also the following |User| |autocommand|s are provided:
 
 LspProgressUpdate                                          *LspProgressUpdate*
@@ -1332,7 +1420,8 @@ force_refresh({bufnr})               *vim.lsp.semantic_tokens.force_refresh()*
     highlighting (|vim.lsp.semantic_tokens.start()| has been called for it)
 
     Parameters: ~
-      • {bufnr}  (nil|number) default: current buffer
+      • {bufnr}  (number|nil) filter by buffer. All buffers if nil, current
+                 buffer if 0
 
                                         *vim.lsp.semantic_tokens.get_at_pos()*
 get_at_pos({bufnr}, {row}, {col})
@@ -1345,7 +1434,34 @@ get_at_pos({bufnr}, {row}, {col})
       • {col}    (number|nil) Position column (default cursor position)
 
     Return: ~
-        (table|nil) List of tokens at position
+        (table|nil) List of tokens at position. Each token has the following
+        fields:
+        • line (number) line number, 0-based
+        • start_col (number) start column, 0-based
+        • end_col (number) end column, 0-based
+        • type (string) token type as string, e.g. "variable"
+        • modifiers (table) token modifiers as a set. E.g., { static = true,
+          readonly = true }
+
+                                   *vim.lsp.semantic_tokens.highlight_token()*
+highlight_token({token}, {bufnr}, {client_id}, {hl_group}, {opts})
+    Highlight a semantic token.
+
+    Apply an extmark with a given highlight group for a semantic token. The
+    mark will be deleted by the semantic token engine when appropriate; for
+    example, when the LSP sends updated tokens. This function is intended for
+    use inside |LspTokenUpdate| callbacks.
+
+    Parameters: ~
+      • {token}      (table) a semantic token, found as `args.data.token` in
+                     |LspTokenUpdate|.
+      • {bufnr}      (number) the buffer to highlight
+      • {client_id}  (number) The ID of the |vim.lsp.client|
+      • {hl_group}   (string) Highlight group name
+      • {opts}       (table|nil) Optional parameters.
+                     • priority: (number|nil) Priority for the applied
+                       extmark. Defaults to
+                       `vim.highlight.priorities.semantic_tokens + 3`
 
 start({bufnr}, {client_id}, {opts})          *vim.lsp.semantic_tokens.start()*
     Start the semantic token highlighting engine for the given buffer with the
diff --git a/runtime/doc/news.txt b/runtime/doc/news.txt
index 28fdaa770d..825496cff5 100644
--- a/runtime/doc/news.txt
+++ b/runtime/doc/news.txt
@@ -90,7 +90,7 @@ The following new APIs or features were added.
   `semanticTokensProvider` from the LSP client's {server_capabilities} in the
   `LspAttach` callback.
 
-  See |lsp-semantic_tokens| for more information.
+  See |lsp-semantic-highlight| for more information.
 
 • |vim.treesitter.inspect_tree()| and |:InspectTree| opens a split window
   showing a text representation of the nodes in a language tree for the current
diff --git a/runtime/lua/vim/_inspector.lua b/runtime/lua/vim/_inspector.lua
index 9e91597192..92d380b08c 100644
--- a/runtime/lua/vim/_inspector.lua
+++ b/runtime/lua/vim/_inspector.lua
@@ -2,7 +2,7 @@
 ---@field syntax boolean include syntax based highlight groups (defaults to true)
 ---@field treesitter boolean include treesitter based highlight groups (defaults to true)
 ---@field extmarks boolean|"all" include extmarks. When `all`, then extmarks without a `hl_group` will also be included (defaults to true)
----@field semantic_tokens boolean include semantic tokens (defaults to true)
+---@field semantic_tokens boolean include semantic token highlights (defaults to true)
 local defaults = {
   syntax = true,
   treesitter = true,
@@ -81,47 +81,54 @@ function vim.inspect_pos(bufnr, row, col, filter)
     end
   end
 
-  -- semantic tokens
-  if filter.semantic_tokens then
-    for _, token in ipairs(vim.lsp.semantic_tokens.get_at_pos(bufnr, row, col) or {}) do
-      token.hl_groups = {
-        type = resolve_hl({ hl_group = '@' .. token.type }),
-        modifiers = vim.tbl_map(function(modifier)
-          return resolve_hl({ hl_group = '@' .. modifier })
-        end, token.modifiers or {}),
-      }
-      table.insert(results.semantic_tokens, token)
+  --- Convert an extmark tuple into a map-like table
+  --- @private
+  local function to_map(extmark)
+    extmark = {
+      id = extmark[1],
+      row = extmark[2],
+      col = extmark[3],
+      opts = resolve_hl(extmark[4]),
+    }
+    extmark.end_row = extmark.opts.end_row or extmark.row -- inclusive
+    extmark.end_col = extmark.opts.end_col or (extmark.col + 1) -- exclusive
+    return extmark
+  end
+
+  --- Check if an extmark overlaps this position
+  --- @private
+  local function is_here(extmark)
+    return (row >= extmark.row and row <= extmark.end_row) -- within the rows of the extmark
+      and (row > extmark.row or col >= extmark.col) -- either not the first row, or in range of the col
+      and (row < extmark.end_row or col < extmark.end_col) -- either not in the last row or in range of the col
+  end
+
+  -- all extmarks at this position
+  local extmarks = {}
+  for ns, nsid in pairs(vim.api.nvim_get_namespaces()) do
+    local ns_marks = vim.api.nvim_buf_get_extmarks(bufnr, nsid, 0, -1, { details = true })
+    ns_marks = vim.tbl_map(to_map, ns_marks)
+    ns_marks = vim.tbl_filter(is_here, ns_marks)
+    for _, mark in ipairs(ns_marks) do
+      mark.ns_id = nsid
+      mark.ns = ns
     end
+    vim.list_extend(extmarks, ns_marks)
+  end
+
+  if filter.semantic_tokens then
+    results.semantic_tokens = vim.tbl_filter(function(extmark)
+      return extmark.ns:find('vim_lsp_semantic_tokens') == 1
+    end, extmarks)
   end
 
-  -- extmarks
   if filter.extmarks then
-    for ns, nsid in pairs(vim.api.nvim_get_namespaces()) do
-      if ns:find('vim_lsp_semantic_tokens') ~= 1 then
-        local extmarks = vim.api.nvim_buf_get_extmarks(bufnr, nsid, 0, -1, { details = true })
-        for _, extmark in ipairs(extmarks) do
-          extmark = {
-            ns_id = nsid,
-            ns = ns,
-            id = extmark[1],
-            row = extmark[2],
-            col = extmark[3],
-            opts = resolve_hl(extmark[4]),
-          }
-          local end_row = extmark.opts.end_row or extmark.row -- inclusive
-          local end_col = extmark.opts.end_col or (extmark.col + 1) -- exclusive
-          if
-            (filter.extmarks == 'all' or extmark.opts.hl_group) -- filter hl_group
-            and (row >= extmark.row and row <= end_row) -- within the rows of the extmark
-            and (row > extmark.row or col >= extmark.col) -- either not the first row, or in range of the col
-            and (row < end_row or col < end_col) -- either not in the last row or in range of the col
-          then
-            table.insert(results.extmarks, extmark)
-          end
-        end
-      end
-    end
+    results.extmarks = vim.tbl_filter(function(extmark)
+      return extmark.ns:find('vim_lsp_semantic_tokens') ~= 1
+        and (filter.extmarks == 'all' or extmark.opts.hl_group)
+    end, extmarks)
   end
+
   return results
 end
 
@@ -174,16 +181,17 @@ function vim.show_pos(bufnr, row, col, filter)
     nl()
   end
 
+  -- semantic tokens
   if #items.semantic_tokens > 0 then
     append('Semantic Tokens', 'Title')
     nl()
-    for _, token in ipairs(items.semantic_tokens) do
-      local client = vim.lsp.get_client_by_id(token.client_id)
-      client = client and (' (' .. client.name .. ')') or ''
-      item(token.hl_groups.type, 'type' .. client)
-      for _, modifier in ipairs(token.hl_groups.modifiers) do
-        item(modifier, 'modifier' .. client)
-      end
+    local sorted_marks = vim.fn.sort(items.semantic_tokens, function(left, right)
+      local left_first = left.opts.priority < right.opts.priority
+        or left.opts.priority == right.opts.priority and left.opts.hl_group < right.opts.hl_group
+      return left_first and -1 or 1
+    end)
+    for _, extmark in ipairs(sorted_marks) do
+      item(extmark.opts, 'priority: ' .. extmark.opts.priority)
     end
     nl()
   end
@@ -197,6 +205,7 @@ function vim.show_pos(bufnr, row, col, filter)
     end
     nl()
   end
+
   -- extmarks
   if #items.extmarks > 0 then
     append('Extmarks', 'Title')
diff --git a/runtime/lua/vim/lsp/semantic_tokens.lua b/runtime/lua/vim/lsp/semantic_tokens.lua
index 24b5c6c24e..7983d066b8 100644
--- a/runtime/lua/vim/lsp/semantic_tokens.lua
+++ b/runtime/lua/vim/lsp/semantic_tokens.lua
@@ -8,8 +8,8 @@ local bit = require('bit')
 --- @field start_col number start column 0-based
 --- @field end_col number end column 0-based
 --- @field type string token type as string
---- @field modifiers string[] token modifiers as strings
---- @field extmark_added boolean whether this extmark has been added to the buffer yet
+--- @field modifiers table token modifiers as a set. E.g., { static = true, readonly = true }
+--- @field marked boolean whether this token has had extmarks applied
 ---
 --- @class STCurrentResult
 --- @field version number document version associated with this result
@@ -36,10 +36,13 @@ local bit = require('bit')
 ---@field client_state table<number, STClientState>
 local STHighlighter = { active = {} }
 
+--- Do a binary search of the tokens in the half-open range [lo, hi).
+---
+--- Return the index i in range such that tokens[j].line < line for all j < i, and
+--- tokens[j].line >= line for all j >= i, or return hi if no such index is found.
+---
 ---@private
-local function binary_search(tokens, line)
-  local lo = 1
-  local hi = #tokens
+local function lower_bound(tokens, line, lo, hi)
   while lo < hi do
     local mid = math.floor((lo + hi) / 2)
     if tokens[mid].line < line then
@@ -51,16 +54,34 @@ local function binary_search(tokens, line)
   return lo
 end
 
+--- Do a binary search of the tokens in the half-open range [lo, hi).
+---
+--- Return the index i in range such that tokens[j].line <= line for all j < i, and
+--- tokens[j].line > line for all j >= i, or return hi if no such index is found.
+---
+---@private
+local function upper_bound(tokens, line, lo, hi)
+  while lo < hi do
+    local mid = math.floor((lo + hi) / 2)
+    if line < tokens[mid].line then
+      hi = mid
+    else
+      lo = mid + 1
+    end
+  end
+  return lo
+end
+
 --- Extracts modifier strings from the encoded number in the token array
 ---
 ---@private
----@return string[]
+---@return table<string, boolean>
 local function modifiers_from_number(x, modifiers_table)
   local modifiers = {}
   local idx = 1
   while x > 0 do
     if bit.band(x, 1) == 1 then
-      modifiers[#modifiers + 1] = modifiers_table[idx]
+      modifiers[modifiers_table[idx]] = true
     end
     x = bit.rshift(x, 1)
     idx = idx + 1
@@ -109,7 +130,7 @@ local function tokens_to_ranges(data, bufnr, client)
         end_col = end_col,
         type = token_type,
         modifiers = modifiers,
-        extmark_added = false,
+        marked = false,
       }
     end
   end
@@ -355,7 +376,7 @@ end
 ---
 ---@private
 function STHighlighter:on_win(topline, botline)
-  for _, state in pairs(self.client_state) do
+  for client_id, state in pairs(self.client_state) do
     local current_result = state.current_result
     if current_result.version and current_result.version == util.buf_versions[self.bufnr] then
       if not current_result.namespace_cleared then
@@ -372,52 +393,55 @@ function STHighlighter:on_win(topline, botline)
       --
       -- Instead, we have to use normal extmarks that can attach to locations
       -- in the buffer and are persisted between redraws.
+      --
+      -- `strict = false` is necessary here for the 1% of cases where the
+      -- current result doesn't actually match the buffer contents. Some
+      -- LSP servers can respond with stale tokens on requests if they are
+      -- still processing changes from a didChange notification.
+      --
+      -- LSP servers that do this _should_ follow up known stale responses
+      -- with a refresh notification once they've finished processing the
+      -- didChange notification, which would re-synchronize the tokens from
+      -- our end.
+      --
+      -- The server I know of that does this is clangd when the preamble of
+      -- a file changes and the token request is processed with a stale
+      -- preamble while the new one is still being built. Once the preamble
+      -- finishes, clangd sends a refresh request which lets the client
+      -- re-synchronize the tokens.
+
+      local set_mark = function(token, hl_group, delta)
+        vim.api.nvim_buf_set_extmark(self.bufnr, state.namespace, token.line, token.start_col, {
+          hl_group = hl_group,
+          end_col = token.end_col,
+          priority = vim.highlight.priorities.semantic_tokens + delta,
+          strict = false,
+        })
+      end
+
+      local ft = vim.bo[self.bufnr].filetype
       local highlights = current_result.highlights
-      local idx = binary_search(highlights, topline)
+      local first = lower_bound(highlights, topline, 1, #highlights + 1)
+      local last = upper_bound(highlights, botline, first, #highlights + 1) - 1
 
-      for i = idx, #highlights do
+      for i = first, last do
         local token = highlights[i]
-
-        if token.line > botline then
-          break
-        end
-
-        if not token.extmark_added then
-          -- `strict = false` is necessary here for the 1% of cases where the
-          -- current result doesn't actually match the buffer contents. Some
-          -- LSP servers can respond with stale tokens on requests if they are
-          -- still processing changes from a didChange notification.
-          --
-          -- LSP servers that do this _should_ follow up known stale responses
-          -- with a refresh notification once they've finished processing the
-          -- didChange notification, which would re-synchronize the tokens from
-          -- our end.
-          --
-          -- The server I know of that does this is clangd when the preamble of
-          -- a file changes and the token request is processed with a stale
-          -- preamble while the new one is still being built. Once the preamble
-          -- finishes, clangd sends a refresh request which lets the client
-          -- re-synchronize the tokens.
-          api.nvim_buf_set_extmark(self.bufnr, state.namespace, token.line, token.start_col, {
-            hl_group = '@' .. token.type,
-            end_col = token.end_col,
-            priority = vim.highlight.priorities.semantic_tokens,
-            strict = false,
-          })
-
-          -- TODO(bfredl) use single extmark when hl_group supports table
-          if #token.modifiers > 0 then
-            for _, modifier in pairs(token.modifiers) do
-              api.nvim_buf_set_extmark(self.bufnr, state.namespace, token.line, token.start_col, {
-                hl_group = '@' .. modifier,
-                end_col = token.end_col,
-                priority = vim.highlight.priorities.semantic_tokens + 1,
-                strict = false,
-              })
-            end
+        if not token.marked then
+          set_mark(token, string.format('@lsp.type.%s.%s', token.type, ft), 0)
+          for modifier, _ in pairs(token.modifiers) do
+            set_mark(token, string.format('@lsp.mod.%s.%s', modifier, ft), 1)
+            set_mark(token, string.format('@lsp.typemod.%s.%s.%s', token.type, modifier, ft), 2)
           end
-
-          token.extmark_added = true
+          token.marked = true
+
+          api.nvim_exec_autocmds('LspTokenUpdate', {
+            pattern = vim.api.nvim_buf_get_name(self.bufnr),
+            modeline = false,
+            data = {
+              token = token,
+              client_id = client_id,
+            },
+          })
         end
       end
     end
@@ -588,7 +612,13 @@ end
 ---@param row number|nil Position row (default cursor position)
 ---@param col number|nil Position column (default cursor position)
 ---
----@return table|nil (table|nil) List of tokens at position
+---@return table|nil (table|nil) List of tokens at position. Each token has
+---        the following fields:
+---        - line (number) line number, 0-based
+---        - start_col (number) start column, 0-based
+---        - end_col (number) end column, 0-based
+---        - type (string) token type as string, e.g. "variable"
+---        - modifiers (table) token modifiers as a set. E.g., { static = true, readonly = true }
 function M.get_at_pos(bufnr, row, col)
   if bufnr == nil or bufnr == 0 then
     bufnr = api.nvim_get_current_buf()
@@ -608,7 +638,7 @@ function M.get_at_pos(bufnr, row, col)
   for client_id, client in pairs(highlighter.client_state) do
     local highlights = client.current_result.highlights
     if highlights then
-      local idx = binary_search(highlights, row)
+      local idx = lower_bound(highlights, row, 1, #highlights + 1)
       for i = idx, #highlights do
         local token = highlights[i]
 
@@ -631,23 +661,60 @@ end
 --- Only has an effect if the buffer is currently active for semantic token
 --- highlighting (|vim.lsp.semantic_tokens.start()| has been called for it)
 ---
----@param bufnr (nil|number) default: current buffer
+---@param bufnr (number|nil) filter by buffer. All buffers if nil, current
+---       buffer if 0
 function M.force_refresh(bufnr)
   vim.validate({
     bufnr = { bufnr, 'n', true },
   })
 
-  if bufnr == nil or bufnr == 0 then
-    bufnr = api.nvim_get_current_buf()
+  local buffers = bufnr == nil and vim.tbl_keys(STHighlighter.active)
+    or bufnr == 0 and { api.nvim_get_current_buf() }
+    or { bufnr }
+
+  for _, buffer in ipairs(buffers) do
+    local highlighter = STHighlighter.active[buffer]
+    if highlighter then
+      highlighter:reset()
+      highlighter:send_request()
+    end
   end
+end
 
+--- Highlight a semantic token.
+---
+--- Apply an extmark with a given highlight group for a semantic token. The
+--- mark will be deleted by the semantic token engine when appropriate; for
+--- example, when the LSP sends updated tokens. This function is intended for
+--- use inside |LspTokenUpdate| callbacks.
+---@param token (table) a semantic token, found as `args.data.token` in
+---       |LspTokenUpdate|.
+---@param bufnr (number) the buffer to highlight
+---@param client_id (number) The ID of the |vim.lsp.client|
+---@param hl_group (string) Highlight group name
+---@param opts (table|nil) Optional parameters.
+---       - priority: (number|nil) Priority for the applied extmark. Defaults
+---         to `vim.highlight.priorities.semantic_tokens + 3`
+function M.highlight_token(token, bufnr, client_id, hl_group, opts)
   local highlighter = STHighlighter.active[bufnr]
   if not highlighter then
     return
   end
 
-  highlighter:reset()
-  highlighter:send_request()
+  local state = highlighter.client_state[client_id]
+  if not state then
+    return
+  end
+
+  opts = opts or {}
+  local priority = opts.priority or vim.highlight.priorities.semantic_tokens + 3
+
+  vim.api.nvim_buf_set_extmark(bufnr, state.namespace, token.line, token.start_col, {
+    hl_group = hl_group,
+    end_col = token.end_col,
+    priority = priority,
+    strict = false,
+  })
 end
 
 --- |lsp-handler| for the method `workspace/semanticTokens/refresh`
diff --git a/src/nvim/auevents.lua b/src/nvim/auevents.lua
index a75ee3bbd5..aef08be820 100644
--- a/src/nvim/auevents.lua
+++ b/src/nvim/auevents.lua
@@ -72,6 +72,7 @@ return {
     'InsertLeavePre',         -- just before leaving Insert mode
     'LspAttach',              -- after an LSP client attaches to a buffer
     'LspDetach',              -- after an LSP client detaches from a buffer
+    'LspTokenUpdate',         -- after a visible LSP token is updated
     'MenuPopup',              -- just before popup menu is displayed
     'ModeChanged',            -- after changing the mode
     'OptionSet',              -- after setting any option
@@ -151,6 +152,7 @@ return {
     DiagnosticChanged=true,
     LspAttach=true,
     LspDetach=true,
+    LspTokenUpdate=true,
     RecordingEnter=true,
     RecordingLeave=true,
     Signal=true,
diff --git a/src/nvim/highlight_group.c b/src/nvim/highlight_group.c
index d2f5b60dc6..70ee6c757c 100644
--- a/src/nvim/highlight_group.c
+++ b/src/nvim/highlight_group.c
@@ -270,16 +270,22 @@ static const char *highlight_init_both[] = {
   "default link @tag Tag",
 
   // LSP semantic tokens
-  "default link @class Structure",
-  "default link @struct Structure",
-  "default link @enum Type",
-  "default link @enumMember Constant",
-  "default link @event Identifier",
-  "default link @interface Identifier",
-  "default link @modifier Identifier",
-  "default link @regexp SpecialChar",
-  "default link @typeParameter Type",
-  "default link @decorator Identifier",
+  "default link @lsp.type.class Structure",
+  "default link @lsp.type.decorator Function",
+  "default link @lsp.type.enum Structure",
+  "default link @lsp.type.enumMember Constant",
+  "default link @lsp.type.function Function",
+  "default link @lsp.type.interface Structure",
+  "default link @lsp.type.macro Macro",
+  "default link @lsp.type.method Function",
+  "default link @lsp.type.namespace Structure",
+  "default link @lsp.type.parameter Identifier",
+  "default link @lsp.type.property Identifier",
+  "default link @lsp.type.struct Structure",
+  "default link @lsp.type.type Type",
+  "default link @lsp.type.typeParameter TypeDef",
+  "default link @lsp.type.variable Identifier",
+
   NULL
 };
 
diff --git a/test/functional/plugin/lsp/semantic_tokens_spec.lua b/test/functional/plugin/lsp/semantic_tokens_spec.lua
index 004fce4983..780d18fce9 100644
--- a/test/functional/plugin/lsp/semantic_tokens_spec.lua
+++ b/test/functional/plugin/lsp/semantic_tokens_spec.lua
@@ -37,10 +37,12 @@ describe('semantic token highlighting', function()
       [6] = { foreground = Screen.colors.Blue1 };
       [7] = { bold = true, foreground = Screen.colors.DarkCyan };
       [8] = { bold = true, foreground = Screen.colors.SlateBlue };
+      [9] = { bold = true, foreground = tonumber('0x6a0dad') };
     }
-    command([[ hi link @namespace Type ]])
-    command([[ hi link @function Special ]])
-    command([[ hi @declaration gui=bold ]])
+    command([[ hi link @lsp.type.namespace Type ]])
+    command([[ hi link @lsp.type.function Special ]])
+    command([[ hi link @lsp.type.comment Comment ]])
+    command([[ hi @lsp.mod.declaration gui=bold ]])
   end)
 
   describe('general', function()
@@ -129,6 +131,46 @@ describe('semantic token highlighting', function()
       ]] }
     end)
 
+    it('use LspTokenUpdate and highlight_token', function()
+      exec_lua([[
+        vim.api.nvim_create_autocmd("LspTokenUpdate", {
+          callback = function(args)
+            local token = args.data.token
+            if token.type == "function" and token.modifiers.declaration then
+              vim.lsp.semantic_tokens.highlight_token(
+                token, args.buf, args.data.client_id, "Macro"
+              )
+            end
+          end,
+        })
+        bufnr = vim.api.nvim_get_current_buf()
+        vim.api.nvim_win_set_buf(0, bufnr)
+        client_id = vim.lsp.start({ name = 'dummy', cmd = server.cmd })
+      ]])
+
+      insert(text)
+
+      screen:expect { grid = [[
+        #include <iostream>                     |
+                                                |
+        int {9:main}()                              |
+        {                                       |
+            int {7:x};                              |
+        #ifdef {5:__cplusplus}                      |
+            {4:std}::{2:cout} << {2:x} << "\n";             |
+        {6:#else}                                   |
+        {6:    printf("%d\n", x);}                  |
+        {6:#endif}                                  |
+        }                                       |
+        ^}                                       |
+        {1:~                                       }|
+        {1:~                                       }|
+        {1:~                                       }|
+                                                |
+      ]] }
+
+    end)
+
     it('buffer is unhighlighted when client is detached', function()
       exec_lua([[
         bufnr = vim.api.nvim_get_current_buf()
@@ -580,14 +622,11 @@ describe('semantic token highlighting', function()
         expected = {
           {
             line = 0,
-            modifiers = {
-              'declaration',
-              'globalScope',
-            },
+            modifiers = { declaration = true, globalScope = true },
             start_col = 6,
             end_col = 9,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
         },
       },
@@ -615,67 +654,67 @@ int main()
         expected = {
           { -- main
             line = 1,
-            modifiers = { 'declaration', 'globalScope' },
+            modifiers = { declaration = true, globalScope = true },
             start_col = 4,
             end_col = 8,
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           { --  __cplusplus
             line = 3,
-            modifiers = { 'globalScope' },
+            modifiers = { globalScope = true },
             start_col = 9,
             end_col = 20,
             type = 'macro',
-            extmark_added = true,
+            marked = true,
           },
           { -- x
             line = 4,
-            modifiers = { 'declaration', 'readonly', 'functionScope' },
+            modifiers = { declaration = true, readonly = true, functionScope = true },
             start_col = 12,
             end_col = 13,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           { -- std
             line = 5,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             start_col = 2,
             end_col = 5,
             type = 'namespace',
-            extmark_added = true,
+            marked = true,
           },
           { -- cout
             line = 5,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             start_col = 7,
             end_col = 11,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           { -- x
             line = 5,
-            modifiers = { 'readonly', 'functionScope' },
+            modifiers = { readonly = true, functionScope = true },
             start_col = 15,
             end_col = 16,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           { -- std
             line = 5,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             start_col = 20,
             end_col = 23,
             type = 'namespace',
-            extmark_added = true,
+            marked = true,
           },
           { -- endl
             line = 5,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             start_col = 25,
             end_col = 29,
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           { -- #else comment #endif
             line = 6,
@@ -683,7 +722,7 @@ int main()
             start_col = 0,
             end_col = 7,
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 7,
@@ -691,7 +730,7 @@ int main()
             start_col = 0,
             end_col = 11,
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 8,
@@ -699,7 +738,7 @@ int main()
             start_col = 0,
             end_col = 8,
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           },
         },
       },
@@ -724,23 +763,23 @@ b = "as"]],
             start_col = 0,
             end_col = 10,
             type = 'comment', -- comment
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 1,
-            modifiers = { 'declaration' }, -- a
+            modifiers = { declaration = true }, -- a
             start_col = 6,
             end_col = 7,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 2,
-            modifiers = { 'static' }, -- b (global)
+            modifiers = { static = true }, -- b (global)
             start_col = 0,
             end_col = 1,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
         },
       },
@@ -770,7 +809,7 @@ b = "as"]],
             start_col = 0,
             end_col = 3, -- pub
             type = 'keyword',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 0,
@@ -778,15 +817,15 @@ b = "as"]],
             start_col = 4,
             end_col = 6, -- fn
             type = 'keyword',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 0,
-            modifiers = { 'declaration', 'public' },
+            modifiers = { declaration = true, public = true },
             start_col = 7,
             end_col = 11, -- main
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 0,
@@ -794,7 +833,7 @@ b = "as"]],
             start_col = 11,
             end_col = 12,
             type = 'parenthesis',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 0,
@@ -802,7 +841,7 @@ b = "as"]],
             start_col = 12,
             end_col = 13,
             type = 'parenthesis',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 0,
@@ -810,15 +849,15 @@ b = "as"]],
             start_col = 14,
             end_col = 15,
             type = 'brace',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 1,
-            modifiers = { 'controlFlow' },
+            modifiers = { controlFlow = true },
             start_col = 4,
             end_col = 9, -- break
             type = 'keyword',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 1,
@@ -826,7 +865,7 @@ b = "as"]],
             start_col = 10,
             end_col = 13, -- rust
             type = 'unresolvedReference',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 1,
@@ -834,15 +873,15 @@ b = "as"]],
             start_col = 13,
             end_col = 13,
             type = 'semicolon',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 2,
-            modifiers = { 'documentation' },
+            modifiers = { documentation = true },
             start_col = 4,
             end_col = 11,
             type = 'comment', -- /// what?
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 3,
@@ -850,7 +889,7 @@ b = "as"]],
             start_col = 0,
             end_col = 1,
             type = 'brace',
-            extmark_added = true,
+            marked = true,
           },
         },
       },
@@ -908,26 +947,26 @@ b = "as"]],
           {
             line = 0,
             modifiers = {
-              'declaration',
-              'globalScope',
+              declaration = true,
+              globalScope = true,
             },
             start_col = 6,
             end_col = 9,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           }
         },
         expected2 = {
           {
             line = 1,
             modifiers = {
-              'declaration',
-              'globalScope',
+              declaration = true,
+              globalScope = true,
             },
             start_col = 6,
             end_col = 9,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           }
         },
         expected_screen1 = function()
@@ -1018,55 +1057,55 @@ int main()
             line = 2,
             start_col = 4,
             end_col = 8,
-            modifiers = { 'declaration', 'globalScope' },
+            modifiers = { declaration = true, globalScope = true },
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 4,
             start_col = 8,
             end_col = 9,
-            modifiers = { 'declaration', 'functionScope' },
+            modifiers = { declaration = true, functionScope = true },
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 5,
             start_col = 7,
             end_col = 18,
-            modifiers = { 'globalScope' },
+            modifiers = { globalScope = true },
             type = 'macro',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 6,
             start_col = 4,
             end_col = 7,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             type = 'namespace',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 6,
             start_col = 9,
             end_col = 13,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 6,
             start_col = 17,
             end_col = 18,
-            extmark_added = true,
-            modifiers = { 'functionScope' },
+            marked = true,
+            modifiers = { functionScope = true },
             type = 'variable',
           },
           {
             line = 7,
             start_col = 0,
             end_col = 5,
-            extmark_added = true,
+            marked = true,
             modifiers = {},
             type = 'comment',
           },
@@ -1076,7 +1115,7 @@ int main()
             modifiers = {},
             start_col = 0,
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 9,
@@ -1084,7 +1123,7 @@ int main()
             end_col = 6,
             modifiers = {},
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           }
         },
         expected2 = {
@@ -1092,63 +1131,63 @@ int main()
             line = 2,
             start_col = 4,
             end_col = 8,
-            modifiers = { 'declaration', 'globalScope' },
+            modifiers = { declaration = true, globalScope = true },
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 4,
             start_col = 8,
             end_col = 9,
-            modifiers = { 'declaration', 'globalScope' },
+            modifiers = { declaration = true, globalScope = true },
             type = 'function',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 5,
             end_col = 12,
             start_col = 11,
-            modifiers = { 'declaration', 'functionScope' },
+            modifiers = { declaration = true, functionScope = true },
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 6,
             start_col = 7,
             end_col = 18,
-            modifiers = { 'globalScope' },
+            modifiers = { globalScope = true },
             type = 'macro',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 7,
             start_col = 4,
             end_col = 7,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             type = 'namespace',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 7,
             start_col = 9,
             end_col = 13,
-            modifiers = { 'defaultLibrary', 'globalScope' },
+            modifiers = { defaultLibrary = true, globalScope = true },
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 7,
             start_col = 17,
             end_col = 18,
-            extmark_added = true,
-            modifiers = { 'globalScope' },
+            marked = true,
+            modifiers = { globalScope = true },
             type = 'function',
           },
           {
             line = 8,
             start_col = 0,
             end_col = 5,
-            extmark_added = true,
+            marked = true,
             modifiers = {},
             type = 'comment',
           },
@@ -1158,7 +1197,7 @@ int main()
             modifiers = {},
             start_col = 0,
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           },
           {
             line = 10,
@@ -1166,7 +1205,7 @@ int main()
             end_col = 6,
             modifiers = {},
             type = 'comment',
-            extmark_added = true,
+            marked = true,
           }
         },
         expected_screen1 = function()
@@ -1228,12 +1267,12 @@ int main()
           {
             line = 0,
             modifiers = {
-              'declaration',
+              declaration = true,
             },
             start_col = 0,
             end_col = 6,
             type = 'variable',
-            extmark_added = true,
+            marked = true,
           }
         },
         expected2 = {