refactor!: use utf8proc full casefolding

According to `CaseFolding-15.1.0.txt`, full casefolding should be
preferred over simple casefolding as it's considered to be more correct.
Since utf8proc already provides full casefolding it makes sense to
switch to it. This will also remove a lot of unnecessary build code.

Temporary exceptions are made for two sets characters:

- `ß` will still be considered `ß` (instead of `ss`) as using a full
  casefolding requires interfering with upstream spell files in some
  form.
- `İ` will still be considered `İ` (instead of `i̇`) as using full
  casefolding requires making a value judgement on the "correct"
  behavior. There are two, equally valid case-insensetive comparison for
  this character according to unicode. It is essentially up to the
  implementor to decide which conversion is correct. For this reason it
  might make sense to allow users to decide which conversion should be
  done as an added option to `casemap` in a future PR.

1 files changed, 25 insertions, 28 deletions

diff --git a/src/nvim/mbyte.c b/src/nvim/mbyte.c
index c6cefb8a91..0c1b537f3a 100644
--- a/src/nvim/mbyte.c
+++ b/src/nvim/mbyte.c
@@ -1284,41 +1284,38 @@ bool utf_ambiguous_width(int c)
                        || intable(emoji_all, ARRAY_SIZE(emoji_all), c));
 }
 
-// Generic conversion function for case operations.
-// Return the converted equivalent of "a", which is a UCS-4 character.  Use
-// the given conversion "table".  Uses binary search on "table".
-static int utf_convert(int a, const convertStruct *const table, size_t n_items)
-{
-  // indices into table
-  size_t start = 0;
-  size_t end = n_items;
-  while (start < end) {
-    // need to search further
-    size_t mid = (end + start) / 2;
-    if (table[mid].rangeEnd < a) {
-      start = mid + 1;
-    } else {
-      end = mid;
-    }
-  }
-  if (start < n_items
-      && table[start].rangeStart <= a
-      && a <= table[start].rangeEnd
-      && (a - table[start].rangeStart) % table[start].step == 0) {
-    return a + table[start].offset;
-  }
-  return a;
-}
-
 // Return the folded-case equivalent of "a", which is a UCS-4 character.  Uses
-// simple case folding.
+// full case folding.
 int utf_fold(int a)
 {
   if (a < 0x80) {
     // be fast for ASCII
     return a >= 0x41 && a <= 0x5a ? a + 32 : a;
   }
-  return utf_convert(a, foldCase, ARRAY_SIZE(foldCase));
+
+  // TODO(dundargoc): utf8proc only does full case folding, which breaks some tests. This is a
+  // temporary workaround to circumvent failing tests.
+  //
+  // (0xdf) ß == ss in full casefolding. Using this however breaks the vim spell tests and the error
+  // E763 is thrown. This is due to the test spells relying on the vim spell files.
+  //
+  // (0x130) İ == i̇ in full casefolding.
+  if (a == 0xdf || a == 0x130) {
+    return a;
+  }
+
+  utf8proc_uint8_t input_str[16] = { 0 };
+  utf8proc_encode_char(a, input_str);
+
+  utf8proc_uint8_t *fold_str_utf;
+  utf8proc_map((utf8proc_uint8_t *)input_str, 0, &fold_str_utf,
+               UTF8PROC_NULLTERM | UTF8PROC_CASEFOLD);
+
+  int fold_codepoint_utf = utf_ptr2char((char *)fold_str_utf);
+
+  xfree(fold_str_utf);
+
+  return fold_codepoint_utf;
 }
 
 // Vim's own character class functions.  These exist because many library