Fix error-handling of strtoimax boundary conditions

strtoimax is only required to set errno if there is an
underflow/overflow.  In those conditions, strtoimax returns
INTMAX_MIN/INTMAX_MAX respectively, so that's the only time we should be
checking the value of errno.

Even in those conditions, errno needs to be set to a known good value
before calling strtoimax to differentiate between "value is actually
INTMAX_MAX/MIN" and "value over/underflows".

Closes #5279

1 files changed, 4 insertions, 1 deletions

diff --git a/src/nvim/charset.c b/src/nvim/charset.c
index 22ca0fb0cc..b5154819b9 100644
--- a/src/nvim/charset.c
+++ b/src/nvim/charset.c
@@ -1739,8 +1739,11 @@ char_u* skiptowhite_esc(char_u *p) {
 /// @return Number read from the string.
 intmax_t getdigits(char_u **pp)
 {
+  errno = 0;
   intmax_t number = strtoimax((char *)*pp, (char **)pp, 10);
-  assert(errno != ERANGE);
+  if (number == INTMAX_MAX || number == INTMAX_MIN) {
+    assert(errno != ERANGE);
+  }
   return number;
 }