fnamemodify: fix handling of :r after :e #11165

- Test fnamemodify()
- Test handling of `expand("%:e:e:r")`.
- Fix :e:e:r on filenames with insufficiently many extensions

During `fnamemodify()`, ensuring that we don't go before the filename's
tail is insufficient in cases where we've already handled a ":e"
modifier, for example:

```
"path/to/this.file.ext" :e:e:r:r
         ^    ^-------- *fnamep
         +------------- tail
```

This means for a ":r", we'll go before `*fnamep`, and outside the bounds
of the filename. This is both incorrect and causes neovim to exit with
an allocation error.

We exit because we attempt to calculate `s - *fnamep` (line 23948).
Since `s` is before `*fnamep`, we caluclate a negative length, which
ends up being interpreted as an amount to allocate, causing neovim to
exit with ENOMEM (`memory.c:xmalloc`).

We must instead ensure we don't go before `*fnamep` nor `tail`.
The check for `tail` is still relevant, for example:

```
"path/to/this.file.ext" :r:r:r
 ^       ^------------- tail
 +--------------------- *fnamep
```
Here we don't want to go before `tail`.

close #11165

1 files changed, 36 insertions, 11 deletions

diff --git a/src/nvim/eval.c b/src/nvim/eval.c
index 7d641f1295..a899dbcd3a 100644
--- a/src/nvim/eval.c
+++ b/src/nvim/eval.c
@@ -24051,22 +24051,47 @@ repeat:
      * - for second :e: before the current fname
      * - otherwise: The last '.'
      */
-    if (src[*usedlen + 1] == 'e' && *fnamep > tail)
+    const bool is_second_e = *fnamep > tail;
+    if (src[*usedlen + 1] == 'e' && is_second_e) {
       s = *fnamep - 2;
-    else
+    } else {
       s = *fnamep + *fnamelen - 1;
-    for (; s > tail; --s)
-      if (s[0] == '.')
+    }
+
+    for (; s > tail; s--) {
+      if (s[0] == '.') {
         break;
-    if (src[*usedlen + 1] == 'e') {             /* :e */
-      if (s > tail) {
-        *fnamelen += (size_t)(*fnamep - (s + 1));
-        *fnamep = s + 1;
-      } else if (*fnamep <= tail)
+      }
+    }
+    if (src[*usedlen + 1] == 'e') {
+      if (s > tail || (0 && is_second_e && s == tail)) {
+        // we stopped at a '.' (so anchor to &'.' + 1)
+        char_u *newstart = s + 1;
+        size_t distance_stepped_back = *fnamep - newstart;
+        *fnamelen += distance_stepped_back;
+        *fnamep = newstart;
+      } else if (*fnamep <= tail) {
         *fnamelen = 0;
-    } else {                          /* :r */
-      if (s > tail)             /* remove one extension */
+      }
+    } else {
+      // :r - Remove one extension
+      //
+      // Ensure that `s` doesn't go before `*fnamep`,
+      // since then we're taking too many roots:
+      //
+      // "path/to/this.file.ext" :e:e:r:r
+      //          ^    ^-------- *fnamep
+      //          +------------- tail
+      //
+      // Also ensure `s` doesn't go before `tail`,
+      // since then we're taking too many roots again:
+      //
+      // "path/to/this.file.ext" :r:r:r
+      //  ^       ^------------- tail
+      //  +--------------------- *fnamep
+      if (s > MAX(tail, *fnamep)) {
         *fnamelen = (size_t)(s - *fnamep);
+      }
     }
     *usedlen += 2;
   }