vim-patch:9.1.1243: diff mode is lacking for changes within lines

Problem:  Diff mode's inline highlighting is lackluster. It only
          performs a line-by-line comparison, and calculates a single
          shortest range within a line that could encompass all the
          changes. In lines with multiple changes, or those that span
          multiple lines, this approach tends to end up highlighting
          much more than necessary.

Solution: Implement new inline highlighting modes by doing per-character
          or per-word diff within the diff block, and highlight only the
          relevant parts, add "inline:simple" to the defaults (which is
          the old behaviour)

This change introduces a new diffopt option "inline:<type>". Setting to
"none" will disable all inline highlighting, "simple" (the default) will
use the old behavior, "char" / "word" will perform a character/word-wise
diff of the texts within each diff block and only highlight the
differences.

The new char/word inline diff only use the internal xdiff, and will
respect diff options such as algorithm choice, icase, and misc iwhite
options. indent-heuristics is always on to perform better sliding.

For character highlight, a post-process of the diff results is first
applied before we show the highlight. This is because a naive diff will
create a result with a lot of small diff chunks and gaps, due to the
repetitive nature of individual characters. The post-process is a
heuristic-based refinement that attempts to merge adjacent diff blocks
if they are separated by a short gap (1-3 characters), and can be
further tuned in the future for better results. This process results in
more characters than necessary being highlighted but overall less visual
noise.

For word highlight, always use first buffer's iskeyword definition.
Otherwise if each buffer has different iskeyword settings we would not
be able to group words properly.

The char/word diffing is always per-diff block, not per line, meaning
that changes that span multiple lines will show up correctly.
Added/removed newlines are not shown by default, but if the user has
'list' set (with "eol" listchar defined), the eol character will be be
highlighted correctly for the specific newline characters.

Also, add a new "DiffTextAdd" highlight group linked to "DiffText" by
default. It allows color schemes to use different colors for texts that
have been added within a line versus modified.

This doesn't interact with linematch perfectly currently. The linematch
feature splits up diff blocks into multiple smaller blocks for better
visual matching, which makes inline highlight less useful especially for
multi-line change (e.g. a line is broken into two lines). This could be
addressed in the future.

As a side change, this also removes the bounds checking introduced to
diff_read() as they were added to mask existing logic bugs that were
properly fixed in vim/vim#16768.

closes: vim/vim#16881

https://github.com/vim/vim/commit/9943d4790e42721a6777da9e12637aa595ba4965

Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>

8 files changed, 64 insertions, 15 deletions

diff --git a/runtime/colors/vim.lua b/runtime/colors/vim.lua
index f10cdedee0..1b6ab01671 100644
--- a/runtime/colors/vim.lua
+++ b/runtime/colors/vim.lua
@@ -47,6 +47,7 @@ hi('WildMenu',     { fg = 'Black', bg = 'Yellow',       ctermfg = 'Black', cterm
 hi('VertSplit',      { link = 'Normal' })
 hi('WinSeparator',   { link = 'VertSplit' })
 hi('WinBarNC',       { link = 'WinBar' })
+hi('DiffTextAdd',    { link = 'DiffText' })
 hi('EndOfBuffer',    { link = 'NonText' })
 hi('LineNrAbove',    { link = 'LineNr' })
 hi('LineNrBelow',    { link = 'LineNr' })
diff --git a/runtime/doc/diff.txt b/runtime/doc/diff.txt
index c9de54342e..2c2565985a 100644
--- a/runtime/doc/diff.txt
+++ b/runtime/doc/diff.txt
@@ -214,14 +214,29 @@ The diffs are highlighted with these groups:
 |hl-DiffAdd|	DiffAdd		Added (inserted) lines.  These lines exist in
 				this buffer but not in another.
 |hl-DiffChange|	DiffChange	Changed lines.
-|hl-DiffText|	DiffText	Changed text inside a Changed line.  Vim
-				finds the first character that is different,
-				and the last character that is different
-				(searching from the end of the line).  The
-				text in between is highlighted.  This means
-				that parts in the middle that are still the
-				same are highlighted anyway.  The 'diffopt'
-				flags "iwhite" and "icase" are used here.
+|hl-DiffText|	DiffText	Changed text inside a Changed line.  Exact
+				behavior depends on the `inline:` setting in
+				'diffopt'.
+				With `inline:` set to "simple", Vim finds the
+				first character that is different, and the
+				last character that is different (searching
+				from the end of the line).  The text in
+				between is highlighted.  This means that parts
+				in the middle that are still the same are
+				highlighted anyway.  The 'diffopt' flags
+				"iwhite" and "icase" are used here.
+				With `inline:` set to "char" or "word", Vim
+				uses the internal diff library to perform a
+				detailed diff between the changed blocks and
+				highlight the exact difference between the
+				two.  Will respect any 'diffopt' flag that
+				affects internal diff.
+				Not used when `inline:` set to "none".
+|hl-DiffTextAdd|  DiffTextAdd	Added text inside a Changed line. Similar to
+				DiffText, but used when there is no
+				corresponding text in other buffers.  Will not
+				be used when `inline:` is set to "simple" or
+				"none".
 |hl-DiffDelete|	DiffDelete	Deleted lines.  Also called filler lines,
 				because they don't really exist in this
 				buffer.
diff --git a/runtime/doc/news.txt b/runtime/doc/news.txt
index ad1481e304..aa6af1ef55 100644
--- a/runtime/doc/news.txt
+++ b/runtime/doc/news.txt
@@ -66,7 +66,7 @@ EVENTS
 
 HIGHLIGHTS
 
-• todo
+• |hl-DiffTextAdd| highlights added text within a changed line.
 
 LSP
 
@@ -78,7 +78,7 @@ LUA
 
 OPTIONS
 
-• todo
+• 'diffopt' `inline:` configures diff highlighting for changes within a line.
 
 PLUGINS
 
diff --git a/runtime/doc/options.txt b/runtime/doc/options.txt
index dd2460722e..09eeddb0f7 100644
--- a/runtime/doc/options.txt
+++ b/runtime/doc/options.txt
@@ -2056,7 +2056,7 @@ A jump table for the options with a short description can be found at |Q_op|.
 	security reasons.
 
 						*'diffopt'* *'dip'*
-'diffopt' 'dip'		string	(default "internal,filler,closeoff,linematch:40")
+'diffopt' 'dip'		string	(default "internal,filler,closeoff,inline:simple,linematch:40")
 			global
 	Option settings for diff mode.  It can consist of the following items.
 	All are optional.  Items must be separated by a comma.
@@ -2119,6 +2119,21 @@ A jump table for the options with a short description can be found at |Q_op|.
 				Use the indent heuristic for the internal
 				diff library.
 
+		inline:{text}	Highlight inline differences within a change.
+				See |view-diffs|.  Supported values are:
+
+				none    Do not perform inline highlighting.
+				simple  Highlight from first different
+					character to the last one in each
+					line. This is the default if nothing
+					is set.
+				char    Use internal diff to perform a
+					character-wise diff and highlight the
+					difference.
+				word    Use internal diff to perform a
+					|word|-wise diff and highlight the
+					difference.
+
 		internal	Use the internal diff library.  This is
 				ignored when 'diffexpr' is set.  *E960*
 				When running out of memory when writing a
diff --git a/runtime/doc/syntax.txt b/runtime/doc/syntax.txt
index faca3d88da..d44e3cea8e 100644
--- a/runtime/doc/syntax.txt
+++ b/runtime/doc/syntax.txt
@@ -5177,8 +5177,11 @@ DiffChange	Diff mode: Changed line. |diff.txt|
 DiffDelete	Diff mode: Deleted line. |diff.txt|
 							*hl-DiffText*
 DiffText	Diff mode: Changed text within a changed line. |diff.txt|
+							*hl-DiffTextAdd*
+DiffTextAdd	Diff mode: Added text within a changed line.  Linked to
+		|hl-DiffText| by default. |diff.txt|
 							*hl-EndOfBuffer*
-EndOfBuffer	Filler lines (~) after the end of the buffer.
+EndOfBuffer	Filler lines (~) after the last line in the buffer.
 		By default, this is highlighted like |hl-NonText|.
 							*hl-TermCursor*
 TermCursor	Cursor in a focused terminal.
diff --git a/runtime/doc/vim_diff.txt b/runtime/doc/vim_diff.txt
index df79d25198..a5c72363b4 100644
--- a/runtime/doc/vim_diff.txt
+++ b/runtime/doc/vim_diff.txt
@@ -48,7 +48,7 @@ Defaults                                            *defaults* *nvim-defaults*
 - 'complete' excludes "i"
 - 'completeopt' defaults to "menu,popup"
 - 'define' defaults to "". The C ftplugin sets it to "^\\s*#\\s*define"
-- 'diffopt' defaults to "internal,filler,closeoff,linematch:40"
+- 'diffopt' defaults to "internal,filler,closeoff,inline:simple,linematch:40"
 - 'directory' defaults to ~/.local/state/nvim/swap// (|xdg|), auto-created
 - 'display' defaults to "lastline"
 - 'encoding' is UTF-8 (cf. 'fileencoding' for file-content encoding)
diff --git a/runtime/lua/vim/_meta/options.lua b/runtime/lua/vim/_meta/options.lua
index 20d8ac3058..427bbd5124 100644
--- a/runtime/lua/vim/_meta/options.lua
+++ b/runtime/lua/vim/_meta/options.lua
@@ -1716,6 +1716,21 @@ vim.go.dex = vim.go.diffexpr
 --- 			Use the indent heuristic for the internal
 --- 			diff library.
 ---
+--- 	inline:{text}	Highlight inline differences within a change.
+--- 			See `view-diffs`.  Supported values are:
+---
+--- 			none    Do not perform inline highlighting.
+--- 			simple  Highlight from first different
+--- 				character to the last one in each
+--- 				line. This is the default if nothing
+--- 				is set.
+--- 			char    Use internal diff to perform a
+--- 				character-wise diff and highlight the
+--- 				difference.
+--- 			word    Use internal diff to perform a
+--- 				`word`-wise diff and highlight the
+--- 				difference.
+---
 --- 	internal	Use the internal diff library.  This is
 --- 			ignored when 'diffexpr' is set.  *E960*
 --- 			When running out of memory when writing a
@@ -1766,7 +1781,7 @@ vim.go.dex = vim.go.diffexpr
 ---
 ---
 --- @type string
-vim.o.diffopt = "internal,filler,closeoff,linematch:40"
+vim.o.diffopt = "internal,filler,closeoff,inline:simple,linematch:40"
 vim.o.dip = vim.o.diffopt
 vim.go.diffopt = vim.o.diffopt
 vim.go.dip = vim.go.diffopt
diff --git a/runtime/syntax/vim.vim b/runtime/syntax/vim.vim
index 7fe22190b7..29d9856b3d 100644
--- a/runtime/syntax/vim.vim
+++ b/runtime/syntax/vim.vim
@@ -60,7 +60,7 @@ syn case ignore
 syn keyword vimGroup contained	Comment Constant String Character Number Boolean Float Identifier Function Statement Conditional Repeat Label Operator Keyword Exception PreProc Include Define Macro PreCondit Type StorageClass Structure Typedef Special SpecialChar Tag Delimiter SpecialComment Debug Underlined Ignore Error Todo
 
 " Default highlighting groups {{{2
-syn keyword vimHLGroup contained ErrorMsg IncSearch ModeMsg NonText StatusLine StatusLineNC EndOfBuffer VertSplit DiffText PmenuSbar TabLineSel TabLineFill Cursor lCursor QuickFixLine CursorLineSign CursorLineFold CurSearch PmenuKind PmenuKindSel PmenuMatch PmenuMatchSel PmenuExtra PmenuExtraSel ComplMatchIns Normal Directory LineNr CursorLineNr MoreMsg Question Search SpellBad SpellCap SpellRare SpellLocal PmenuThumb Pmenu PmenuSel SpecialKey Title WarningMsg WildMenu Folded FoldColumn SignColumn Visual DiffAdd DiffChange DiffDelete TabLine CursorColumn CursorLine ColorColumn MatchParen StatusLineTerm StatusLineTermNC CursorIM LineNrAbove LineNrBelow
+syn keyword vimHLGroup contained ErrorMsg IncSearch ModeMsg NonText StatusLine StatusLineNC EndOfBuffer VertSplit DiffText DiffTextAdd PmenuSbar TabLineSel TabLineFill Cursor lCursor QuickFixLine CursorLineSign CursorLineFold CurSearch PmenuKind PmenuKindSel PmenuMatch PmenuMatchSel PmenuExtra PmenuExtraSel ComplMatchIns Normal Directory LineNr CursorLineNr MoreMsg Question Search SpellBad SpellCap SpellRare SpellLocal PmenuThumb Pmenu PmenuSel SpecialKey Title WarningMsg WildMenu Folded FoldColumn SignColumn Visual DiffAdd DiffChange DiffDelete TabLine CursorColumn CursorLine ColorColumn MatchParen StatusLineTerm StatusLineTermNC CursorIM LineNrAbove LineNrBelow
 syn match vimHLGroup contained "\<Conceal\>"
 syn keyword vimOnlyHLGroup contained	Menu Scrollbar ToolbarButton ToolbarLine Tooltip VisualNOS
 syn keyword nvimHLGroup contained	FloatBorder FloatFooter FloatTitle MsgSeparator NormalFloat NormalNC Substitute TermCursor VisualNC Whitespace WinBar WinBarNC WinSeparator