| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
The magic globals `it`, `describe`, etc., are more trouble than they are
worth.
- Hooking into `after_each` requires `getfenv()` hacks.
- They confuse luals/emmylua, because the top-level `.luarc.json` isn't
merged with `test/.luarc.json` (apparently a luals limitation?)
- They totally defeat discoverability because the user just has to
"know" about the various magic symbols.
So they harm DX, which means they serve no purpose at all.
Solution:
- Expose the test API from `testutil`, so tests can call `t.it()`,
`t.describe()`, etc., in the conventional way.
- Drop `getfenv()` hacks.
- Drop the `setfenv()` injection in `load_chunk`.
- Drop `test/_meta.lua`.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
parser_gc() calls ts_parser_delete() but leaves the userdata pointer
pointing to freed memory. If the GC finalizer runs at an unexpected time
(e.g. inside nvim_buf_get_lines #39411), a stale pointer could cause a crash.
Solution:
- NULL out `*ud` after ts_parser_delete() in parser_gc()
- Update parser_check() to handle NULL with a clear error message,
guarding all parser methods against UAF
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Signed-off-by: Szymon Wilczek <swilczek.lx@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
`get_node_text()` returned inconsistent results between buffer and
string sources when a node's range ends at `end_col == 0` (i.e. the node
ends with a newline). The buffer path dropped the trailing newline; the
string path included it correctly.
Solution:
Append `'\n'` in `buf_range_get_text()` when `end_col == 0` and
`start_row ~= end_row`. The `start_row ~= end_row` guard excludes
zero-width nodes at column 0, which should return `""`.
Remove the workaround in the `#trim!` directive that manually
compensated for the missing newline.
Strip whitespace in `resolve_lang()` so injection language nodes ending
at `end_col == 0` (e.g. `">lua\n"`) still resolve correctly.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After an edit, LanguageTree:_edit() updates the current trees. When the
LanguageTree manages explicit regions, _edit() also refreshes _regions
from tree:included_ranges(true), so those regions have the edited byte
offsets.
A later injection pass may call set_included_regions() with a different
number of child regions. That path discards the old trees and emits
changedtree callbacks for them. invalidate(true) does the same when a
buffer is reloaded. Before this change, both discard paths called
tree:included_ranges(true) for every old tree, even if _edit() had just
collected those exact ranges.
That duplicate range extraction is expensive with many injection trees.
Realistic shapes include generated C files with many macro bodies parsed
by the C preproc_arg injection, Markdown documents with many fenced blocks
of the same language, and template files with many embedded-language
islands. The stock highlighter registers recursive changedtree callbacks,
so this is on the normal highlighting edit path.
Track whether _regions currently came from tree:included_ranges(true)
with _regions_from_tree_ranges. _do_changedtree_callbacks() reuses
_regions only in that state; otherwise it falls back to calling
tree:included_ranges(true). Clear the marker when regions are replaced by
injection ranges, when a tree is reparsed, or when trees are discarded.
This avoids keeping a second copy of the ranges while preserving callback
precision: changedtree still receives tree:included_ranges(true) for the
old tree, not the broader managed region.
Benchmark on 100k C macro injections, one-line edit, recursive
changedtree callback:
- HEAD median: edited parse 84.2 ms, child region replacement 58.7 ms
- This change: edited parse 34.6 ms, child region replacement 8.5 ms
That is about 2.4x faster for the edit parse and 6.9x faster for child
region replacement in this workload.
Add a regression test that replacing injection regions still fires
changedtree and still reports the old tree's exact included ranges.
AI-assisted: Codex
|
| |
|
|
|
| |
Problem: Treesitter highlighting regressed on 32-bit builds because ranges that should cover the whole buffer were corrupted when passed into Lua.
Solution: Round-trip those range values through Lua and validate them so treesitter sees the same ranges on 32 and 64-bit builds.
|
| | |
|
| |
|
|
|
| |
gen_vimdoc.lua: In prepare for the upcoming release, comment-out the
"Experimental" warning for prerelease features.
|
| |
|
|
|
|
|
|
|
|
|
| |
Hyphenated language names are silently dropped when used as injections
(see #38132).
This combines the normalization of language aliases into `resolve_lang`,
and also adds the normalization of hyphens to underscores, which allows
for handling of injected language tags with hyphens in their names.
Fixes #38132.
|
| |
|
| |
This reverts commit 2a7cb32959b4c616bd2c76ae1933f8e068e391ad.
|
| |
|
|
|
|
|
|
|
|
| |
This commit changes `languagetree.lua` so that it creates a scratch
buffer under the hood when dealing with string parsers. This will make
it much easier to just use extmarks whenever we need to track injection
trees in `languagetree.lua`. This also allows us to remove the
`treesitter.c` code for parsing a string directly.
Note that the string parser's scratch buffer has `set noeol nofixeol` so
that the parsed source exactly matches the passed in string.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
**Problem(?):** Buffers that (for whatever reason) aren't meant to have
a final newline are still parsed with a final newline in `treesitter.c`.
**Solution:** Don't add the newline to the last buffer line if it
shouldn't be there. (This more closely matches the approach of
`read_buffer_into()`.)
This allows us to, say, use a scratch buffer with `noeol` and `nofixeol`
behind the scenes in `get_string_parser()`.
...which would allow us to track injection trees with extmarks in that
case.
...which would allow us to not drop previous trees after reparsing a
different range with `get_parser():parse()`.
...which would prevent flickering when editing a buffer that has 2+
windows to it in view at a time.
...which would allow us to keep our sanity!!!
(one step at a time...)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit changes the `offset!` directive so that instead of setting a
`metadata.range` value for the entire pattern, it will set a
`metadata.offset` value. This offset will be applied to the range only
in `vim.treesitter.get_range()`, rather than at directive application
time. This allows the offset to be applied to any and all nodes captured
by the given pattern, and removes the requirement that `#offset!` be
applied to only a single node.
The downside of this change is that plugins which read from
`metadata.range` may be thrown off course, but such plugins should
prefer `vim.treesitter.get_range()` when retrieving ranges anyway.
Note that `#trim!` still sets `metadata.range`, and
`vim.treesitter.get_range()` still reads from `metadata.range`, if it
exists.
|
| | |
|
| |
|
|
|
|
|
| |
Simplify the logic for retrieving the injection ranges for the language
tree. The trees are now also sorted by starting position, regardless of
whether they are part of a combined injection or not. This would be
helpful if ranges are ever to be stored in an interval tree or other
kind of sorted tree structure.
|
| |
|
|
|
| |
* refactor: rewrite test without trailing whitespace
* test: combined injection tests
|
| |
|
|
| |
Co-authored-by: Jaehwang Jung <tomtomjhj@gmail.com>
|
| |
|
|
|
|
| |
When given, only that range will be checked for validity rather than the
entire tree. This is used in the highlighter to save CPU cycles since we
only need to parse a certain region at a time anyway.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
**Problem:** `vim.treesitter.get_parser()` and `vim.treesitter.start()`
both parse the tree before returning it. This is problematic because if
this is a sync parse, it will stall the editor on large files. If it is
an async parse, the functions return stale trees.
**Solution:** Remove this parsing side effect and leave it to the user
to parse the returned trees, either synchronously or asynchronously.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
**Problem:** Parsing can be slow for large files, and it is a blocking
operation which can be disruptive and annoying.
**Solution:** Provide a function for asynchronous parsing, which accepts
a callback to be run after parsing completes.
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Co-authored-by: Luuk van Baal <luukvbaal@gmail.com>
Co-authored-by: VanaIgr <vanaigranov@gmail.com>
|
| |
|
|
|
|
|
|
|
| |
Problem:
char-wise folding for `#trim!` ranges are improperly calculated for nodes that
end at column 0, due to the way `get_node_text` works.
Solution:
Add the blank line that `get_node_text` removes for for nodes ending at column
0. Also properly set column positions when performing linewise trims.
|
| |
|
|
| |
The util file, for now, just abstracts the common `run_query` function.
|
| |
|
|
|
|
|
|
|
|
| |
This commit also implements more generic trimming, acting on all
whitespace (charwise) rather than just empty lines.
It will unblock
https://github.com/nvim-treesitter/nvim-treesitter/pull/3442 and allow
for properly concealing markdown bullet markers regardless of indent
width, e.g.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
**Problems:**
- `vim.treesitter.language.inspect()` returns duplicate
symbol names, sometimes up to 6 of one kind in the case of `markdown`
- The list-like `symbols` table can have holes and is thus not even a
valid msgpack table anyway, mentioned in a test
**Solution:** Return symbols as a map, rather than a list, where field
names are the names of the symbol. The boolean value associated with the
field encodes whether or not the symbol is named.
Note that anonymous nodes are surrounded with double quotes (`"`) to
prevent potential collisions with named counterparts that have the same
identifier.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
**Problem:** `vim.treesitter.get_parser` will throw an error if no parser
can be found.
- This means the caller is responsible for wrapping it in a `pcall`,
which is easy to forget
- It also makes it slightly harder to potentially memoize `get_parser`
in the future
- It's a bit unintuitive since many other `get_*` style functions
conventionally return `nil` if no object is found (e.g. `get_node`,
`get_lang`, `query.get`, etc.)
**Solution:** Return `nil` if no parser can be found or created
- This requires a function signature change, and some new assertions in
places where the parser will always (or should always) be found.
- This commit starts by making this change internally, since it is
breaking. Eventually it will be rolled out to the public API.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
For context, see https://github.com/neovim/neovim/pull/24738. Before
that PR, Nvim did not correctly handle captures with quantifiers. That
PR made the correct behavior opt-in to minimize breaking changes, with
the intention that the correct behavior would eventually become the
default. Users can still opt-in to the old (incorrect) behavior for now,
but this option will eventually be removed completely.
BREAKING CHANGE: Any plugin which uses `Query:iter_matches()` must
update their call sites to expect an array of nodes in the `match`
table, rather than a single node.
|
| |
|
|
|
|
|
|
|
|
|
| |
Problem:
Tests have lots of exec_lua calls which input blocks of code
provided as unformatted strings.
Solution:
Teach exec_lua how to handle functions.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Specifically, functions that are run in the context of the test runner
are put in module `test/testutil.lua` while the functions that are run
in the context of the test session are put in
`test/functional/testnvim.lua`.
Closes https://github.com/neovim/neovim/issues/27004.
|
| | |
|
| |
|
|
| |
Work on https://github.com/neovim/neovim/issues/27004.
|
| |
|
|
|
|
| |
Move test cases that are more about treesitter query API rather than
parser API or LanguageTree out of "treesitter/parser_spec", and collect
them in another test suite "treesitter/query_spec".
|
| |
|
|
|
|
|
|
|
|
|
| |
General refactoring, including:
- Improve whitespace and indentation
- Prefix captures with `@`
- Add more comments on `iter_capture()` tests
- Move `test_query` up closer to the fixture source string
No behavioral changes are made.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Query patterns can contain quantifiers (e.g. (foo)+ @bar), so a single
capture can map to multiple nodes. The iter_matches API can not handle
this situation because the match table incorrectly maps capture indices
to a single node instead of to an array of nodes.
The match table should be updated to map capture indices to an array of
nodes. However, this is a massively breaking change, so must be done
with a proper deprecation period.
`iter_matches`, `add_predicate` and `add_directive` must opt-in to the
correct behavior for backward compatibility. This is done with a new
"all" option. This option will become the default and removed after the
0.10 release.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
Co-authored-by: MDeiml <matthias@deiml.net>
Co-authored-by: Gregory Anders <greg@gpanders.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
A region managed by an injected parser may shrink after re-running the
injection query. If the updated region goes out of the range to be
parsed, then the corresponding tree will remain outdated, possibly
retaining the nodes that shouldn't exist anymore. This results in
outdated highlights.
Solution:
Re-parse an invalid tree if its region intersects the range to be
parsed.
|
| |
|
|
|
|
|
|
|
| |
When parsing with a range, languagetree looks up injections and adds
them if needed. This explicitly invalidates parser, making `is_valid`
report `false` both when including and excluding children.
This is an attempt to describe desired behaviour of `is_valid` in tests,
with what ended up being a single line change to satisfy them.
|
| | |
|
| | |
|
| |
|
|
| |
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
Treesitter highlighting is slow for large files with lots of injections.
Solution:
Only parse injections we are going to render during a redraw cycle.
---
- `LanguageTree:parse()` will no longer parse injections by default and
now requires an explicit range argument to be passed.
- `TSHighlighter` now parses injections incrementally during on_win
callbacks for the line range being rendered.
- Plugins which require certain injections to be parsed must run
`parser:parse({ start_row, end_row })` before using the tree.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* feat(treesitter): add injection language fallback
Problem: injection languages are often specified via aliases (e.g.,
filetype or in upper case), requiring custom directives.
Solution: include lookup logic (try as parser name, then filetype, then
lowercase) in LanguageTree itself and remove `#inject-language`
directive.
Co-authored-by: Lewis Russell <me@lewisr.dev>
|
