fix(tui): move $COLORTERM check to _defaults.lua (#29197)

We currently check $COLORTERM in the TUI process to determine if the
terminal supports 24 bit color (truecolor). If $COLORTERM is "truecolor"
or "24bit" then we automatically assume that the terminal supports
truecolor, but if $COLORTERM is set to any other value we still query
the terminal.

The `rgb` flag of the UI struct is a boolean which only indicates
whether the UI supports truecolor, but does not have a 3rd state that we
can use to represent "we don't know if the UI supports truecolor". We
currently use `rgb=false` to represent this "we don't know" state, and
we use XTGETTCAP and DECRQSS queries to determine at runtime if the
terminal supports truecolor. However, if $COLORTERM is set to a value
besides "truecolor" or "24bit" (e.g. "256" or "16) that is a clear
indication that the terminal _does not_ support truecolor, so it is
incorrect to treat `rgb=false` as "we don't know" in that case.

Instead, in the TUI process we only check for the terminfo capabilities.
This must be done in the TUI process because we do not have access to
this information in the core Neovim process when `_defaults.lua` runs.
If the TUI cannot determine truecolor support from terminfo alone, we
set `rgb=false` to indicate "we don't know if the terminal supports
truecolor yet, keep checking". When we get to `_defaults.lua`, we can
then check $COLORTERM and only query the terminal if it is unset.

This means that users can explicitly opt out of truecolor determination
by setting `COLORTERM=256` (or similar) in their environment.

1 files changed, 7 insertions, 3 deletions

diff --git a/runtime/lua/vim/_defaults.lua b/runtime/lua/vim/_defaults.lua
index 79fe5a8513..01dcd4bf74 100644
--- a/runtime/lua/vim/_defaults.lua
+++ b/runtime/lua/vim/_defaults.lua
@@ -470,10 +470,14 @@ do
     --- response indicates that it does support truecolor enable 'termguicolors',
     --- but only if the user has not already disabled it.
     do
-      if tty.rgb then
-        -- The TUI was able to determine truecolor support
+      local colorterm = os.getenv('COLORTERM')
+      if tty.rgb or colorterm == 'truecolor' or colorterm == '24bit' then
+        -- The TUI was able to determine truecolor support or $COLORTERM explicitly indicates
+        -- truecolor support
         setoption('termguicolors', true)
-      else
+      elseif colorterm == nil or colorterm == '' then
+        -- Neither the TUI nor $COLORTERM indicate that truecolor is supported, so query the
+        -- terminal
         local caps = {} ---@type table<string, boolean>
         require('vim.termcap').query({ 'Tc', 'RGB', 'setrgbf', 'setrgbb' }, function(cap, found)
           if not found then