fix(tui): move $COLORTERM check to _defaults.lua (#29197)

We currently check $COLORTERM in the TUI process to determine if the
terminal supports 24 bit color (truecolor). If $COLORTERM is "truecolor"
or "24bit" then we automatically assume that the terminal supports
truecolor, but if $COLORTERM is set to any other value we still query
the terminal.

The `rgb` flag of the UI struct is a boolean which only indicates
whether the UI supports truecolor, but does not have a 3rd state that we
can use to represent "we don't know if the UI supports truecolor". We
currently use `rgb=false` to represent this "we don't know" state, and
we use XTGETTCAP and DECRQSS queries to determine at runtime if the
terminal supports truecolor. However, if $COLORTERM is set to a value
besides "truecolor" or "24bit" (e.g. "256" or "16) that is a clear
indication that the terminal _does not_ support truecolor, so it is
incorrect to treat `rgb=false` as "we don't know" in that case.

Instead, in the TUI process we only check for the terminfo capabilities.
This must be done in the TUI process because we do not have access to
this information in the core Neovim process when `_defaults.lua` runs.
If the TUI cannot determine truecolor support from terminfo alone, we
set `rgb=false` to indicate "we don't know if the terminal supports
truecolor yet, keep checking". When we get to `_defaults.lua`, we can
then check $COLORTERM and only query the terminal if it is unset.

This means that users can explicitly opt out of truecolor determination
by setting `COLORTERM=256` (or similar) in their environment.

1 files changed, 3 insertions, 11 deletions

diff --git a/src/nvim/tui/tui.c b/src/nvim/tui/tui.c
index 650133e6a2..57696b1839 100644
--- a/src/nvim/tui/tui.c
+++ b/src/nvim/tui/tui.c
@@ -1855,20 +1855,12 @@ static int unibi_find_ext_bool(unibi_term *ut, const char *name)
   return -1;
 }
 
-/// Determine if the terminal supports truecolor or not:
+/// Determine if the terminal supports truecolor or not.
 ///
-/// 1. If $COLORTERM is "24bit" or "truecolor", return true
-/// 2. Else, check terminfo for Tc, RGB, setrgbf, or setrgbb capabilities. If
-///    found, return true
-/// 3. Else, return false
+/// If terminfo contains Tc, RGB, or both setrgbf and setrgbb capabilities, return true.
 static bool term_has_truecolor(TUIData *tui, const char *colorterm)
 {
-  // Check $COLORTERM
-  if (strequal(colorterm, "truecolor") || strequal(colorterm, "24bit")) {
-    return true;
-  }
-
-  // Check for Tc and RGB
+  // Check for Tc or RGB
   for (size_t i = 0; i < unibi_count_ext_bool(tui->ut); i++) {
     const char *n = unibi_get_ext_bool_name(tui->ut, i);
     if (n && (!strcmp(n, "Tc") || !strcmp(n, "RGB"))) {