refactor(grid): change schar_T representation to be more compact

Previously, a screen cell would occupy 28+4=32 bytes per cell
as we always made space for up to MAX_MCO+1 codepoints in a cell.

As an example, even a pretty modest 50*80 screen would consume

50*80*2*32 = 256000, i e a quarter megabyte

With the factor of two due to the TUI side buffer, and even more when
using msg_grid and/or ext_multigrid.

This instead stores a 4-byte union of either:
- a valid UTF-8 sequence up to 4 bytes
- an escape char which is invalid UTF-8 (0xFF) plus a 24-bit index to a
  glyph cache

This avoids allocating space for huge composed glyphs _upfront_, while
still keeping rendering such glyphs reasonably fast (1 hash table lookup
+ one plain index lookup). If the same large glyphs are using repeatedly
on the screen, this is still a net reduction of memory/cache
consumption. The only case which really gets worse is if you blast
the screen full with crazy emojis and zalgo text and even this case
only leads to 4 extra bytes per char.

When only <= 4-byte glyphs are used, plus the 4-byte attribute code,
i e 8 bytes in total there is a factor of four reduction of memory use.
Memory which will be quite hot in cache as the screen buffer is scanned
over in win_line() buffer text drawing

A slight complication is that the representation depends on host byte
order. I've tested this manually by compling and running this
in qemu-s390x and it works fine. We might add a qemu based solution
to CI at some point.

1 files changed, 15 insertions, 3 deletions

diff --git a/src/nvim/drawscreen.c b/src/nvim/drawscreen.c
index f71a47a596..e3e9fc8170 100644
--- a/src/nvim/drawscreen.c
+++ b/src/nvim/drawscreen.c
@@ -444,6 +444,15 @@ int update_screen(void)
   display_tick++;  // let syntax code know we're in a next round of
                    // display updating
 
+  // glyph cache full, very rare
+  if (schar_cache_clear_if_full()) {
+    // must use CLEAR, as the contents of screen buffers cannot be
+    // compared to their previous state here.
+    // TODO(bfredl): if start to cache schar_T values in places (like fcs/lcs)
+    // we need to revalidate these here as well!
+    type = MAX(type, UPD_CLEAR);
+  }
+
   // Tricky: vim code can reset msg_scrolled behind our back, so need
   // separate bookkeeping for now.
   if (msg_did_scroll) {
@@ -736,7 +745,10 @@ static void win_redr_border(win_T *wp)
 
   ScreenGrid *grid = &wp->w_grid_alloc;
 
-  schar_T *chars = wp->w_float_config.border_chars;
+  schar_T chars[8];
+  for (int i = 0; i < 8; i++) {
+    chars[i] = schar_from_str(wp->w_float_config.border_chars[i]);
+  }
   int *attrs = wp->w_float_config.border_attr;
 
   int *adj = wp->w_border_adj;
@@ -770,7 +782,7 @@ static void win_redr_border(win_T *wp)
       grid_puts_line_flush(false);
     }
     if (adj[1]) {
-      int ic = (i == 0 && !adj[0] && chars[2][0]) ? 2 : 3;
+      int ic = (i == 0 && !adj[0] && chars[2]) ? 2 : 3;
       grid_puts_line_start(grid, i + adj[0]);
       grid_put_schar(grid, i + adj[0], icol + adj[3], chars[ic], attrs[ic]);
       grid_puts_line_flush(false);
@@ -784,7 +796,7 @@ static void win_redr_border(win_T *wp)
     }
 
     for (int i = 0; i < icol; i++) {
-      int ic = (i == 0 && !adj[3] && chars[6][0]) ? 6 : 5;
+      int ic = (i == 0 && !adj[3] && chars[6]) ? 6 : 5;
       grid_put_schar(grid, irow + adj[0], i + adj[3], chars[ic], attrs[ic]);
     }