diff options
Diffstat (limited to 'test/README.md')
| -rw-r--r-- | test/README.md | 284 |
1 files changed, 263 insertions, 21 deletions
diff --git a/test/README.md b/test/README.md index df66f24626..010a2c9c12 100644 --- a/test/README.md +++ b/test/README.md @@ -1,34 +1,267 @@ -# Tests +Tests +===== -Tests are run by `/cmake/RunTests.cmake` file, using busted. +Tests are run by `/cmake/RunTests.cmake` file, using `busted`. -## Directory structure +For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight. -Directories with tests: `/test/benchmark` for benchmarks, `/test/functional` for -functional tests, `/test/unit` for unit tests. `/test/config` contains `*.in` -files (currently a single one) which are transformed into `*.lua` files using -`configure_file` CMake command: this is for acessing CMake variables in lua -tests. `/test/includes` contains include files for use by luajit `ffi.cdef` -C definitions parser: normally used to make macros not accessible via this -mechanism accessible the other way. +--- -Files `/test/*/preload.lua` contain modules which will be preloaded by busted, -via `--helper` option. `/test/**/helpers.lua` contain various “library” -functions, (intended to be) used by a number of tests and not just a single one. +- [Running tests](#running-tests) +- [Unit tests](#unit-tests) +- [Lint](#lint) +- [Environment variables](#environment-variables) -`/test/*/**/*_spec.lua` are files containing actual tests. Files that do not end -with a `_spec.lua` are libraries like `/test/**/helpers.lua`, except that they -have some common topic. +--- -Tests inside `/test/unit` and `/test/functional` are normally divided into -groups by the semantic component they are testing. +Running tests +------------- -## Environment variables +Neovim uses third-party tooling to execute tests. So be sure, from the +repository directory, to build the tools before testing: + + make cmake + +## Executing Tests + +To run all _non-legacy_ (unit + functional) tests: + + make test + +To run only _unit_ tests: + + make unittest + +To run only _functional_ tests: + + make functionaltest + +--- + +## Filter Tests + +### Filter by name + +Another filter method is by setting a pattern of test name to `TEST_FILTER`. + +``` lua +it('foo api',function() + ... +end) +it('bar api',function() + ... +end) +``` + +To run only test with filter name: + + TEST_TAG='foo.*api' make functionaltest + +### Filter by file + +To run a *specific* unit test: + + TEST_FILE=test/unit/foo.lua make unittest + +To run a *specific* functional test: + + TEST_FILE=test/functional/foo.lua make functionaltest + +To *repeat* a test many times: + + .deps/usr/bin/busted --filter 'foo' --repeat 1000 test/functional/ui/foo_spec.lua + +### Filter by tag + +Tests can be "tagged" by adding `#` before a token in the test description. + +``` lua +it('#foo bar baz', function() + ... +end) +it('#foo another test', function() + ... +end) +``` + +To run only the tagged tests: + + TEST_TAG=foo make functionaltest + +**NOTES**: +* Tags are mainly used for testing issues (ex: `#1234`), so use the following + method. +* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The + given value to `TEST_FILE` must be a path to an existing file. +* Both `TEST_TAG` and `TEST_FILTER` filter tests by the strings from either + `it()` or `describe()` functions. + +--- + +### Legacy + +To run all legacy (Vim) integration tests: + + make oldtest + +To run a *single* legacy test, run `make` with `TEST_FILE=test_name.res`. E.g. +to run `test_syntax.vim`: + + TEST_FILE=test_syntax.res make oldtest + +- The `.res` extension (instead of `.vim`) is required. +- Specify only the test file name, not the full path. + +### Functional tests + +`$GDB` can be set to [run tests under +gdbserver](https://github.com/neovim/neovim/pull/1527). If `$VALGRIND` is also +set, it will add the `--vgdb=yes` option to valgrind instead of +starting gdbserver directly. + +Unit tests +---------- + +Tests are broadly divided into *unit tests* +([test/unit](https://github.com/neovim/neovim/tree/master/test/unit) directory) +and *functional tests* +([test/functional](https://github.com/neovim/neovim/tree/master/test/functional) +directory). Use any of the existing tests as a template to start writing new +tests. + +- _Unit_ testing is achieved by compiling the tests as a shared library which is + loaded and called by LuaJit [FFI](http://luajit.org/ext_ffi.html). +- _Functional_ tests are driven by RPC, so they do not require LuaJit (as + opposed to Lua). + +You can learn the [key concepts of Lua in 15 +minutes](http://learnxinyminutes.com/docs/lua/). + +## Guidelines for writing tests + +- Consider [BDD](http://en.wikipedia.org/wiki/Behavior-driven_development) + guidelines for organization and readability of tests. Describe what you're + testing (and the environment if applicable) and create specs that assert its + behavior. +- For testing static functions or functions that have side effects visible only + in module-global variables, create accessors for the modified variables. For + example, say you are testing a function in misc1.c that modifies a static + variable, create a file `test/c-helpers/misc1.c` and add a function that + retrieves the value after the function call. Files under `test/c-helpers` will + only be compiled when building the test shared library. +- Luajit needs to know about type and constant declarations used in function + prototypes. The + [helpers.lua](https://github.com/neovim/neovim/blob/master/test/unit/helpers.lua) + file automatically parses `types.h`, so types used in the tested functions + must be moved to it to avoid having to rewrite the declarations in the test + files (even though this is how it's currently done currently in the misc1/fs + modules, but contributors are encouraged to refactor the declarations). + - Macro constants must be rewritten as enums so they can be "visible" to the + tests automatically. +- Busted supports various "output providers". The + **[gtest](https://github.com/Olivine-Labs/busted/pull/394) output provider** + shows verbose details that can be useful to diagnose hung tests. Either modify + the Makefile or compile with `make + CMAKE_EXTRA_FLAGS=-DBUSTED_OUTPUT_TYPE=gtest` to enable it. +- **Use busted's `pending()` feature** to skip tests + ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). + Do not silently skip the test with `if-else`. If a functional test depends on + some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you + can't mock or fake the dependency, then skip the test via `pending()` if the + external factor is missing. This ensures that the *total* test-count (success + + fail + error + pending) is the same in all environments. + - *Note:* `pending()` is ignored if it is missing an argument _unless_ it is + [contained in an `it()` + block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11). + Provide empty function argument if the `pending()` call is outside of + `it()` + ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). +- Use `make testlint` for using the shipped luacheck program ([supported by + syntastic](https://github.com/scrooloose/syntastic/blob/d6b96c079be137c83009827b543a83aa113cc011/doc/syntastic-checkers.txt#L3546)) + to lint all tests. + +### Where tests go + +- _Unit tests_ + ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should + match 1-to-1 with the structure of `src/nvim/`, because they are testing + functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in + `test/unit/undo_spec.lua`. +- _Functional tests_ + ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional)) + are higher-level (plugins and user input) than unit tests; they are organized + by concept. + - Try to find an existing `test/functional/*/*_spec.lua` group that makes + sense, before creating a new one. + +## Checklist for migrating legacy tests + +**Note:** Only "old style" (`src/testdir/*.in`) legacy tests should be +converted. Please _do not_ convert "new style" Vim tests (`src/testdir/*.vim`). +The "new style" Vim tests are faster than the old ones, and converting them +takes time and effort better spent elsewhere. + +- Remove the test from the Makefile (`src/nvim/testdir/Makefile`). +- Remove the associated `test.in`, `test.out`, and `test.ok` files from + `src/nvim/testdir/`. +- Make sure the lua test ends in `_spec.lua`. +- Make sure the test count increases accordingly in the build log. +- Make sure the new test contains the same control characters (`^]`, ...) as the + old test. + - Instead of the actual control characters, use an equivalent textual + representation (e.g. `<esc>` instead of `^]`). The + `scripts/legacy2luatest.pl` script does some of these conversions + automatically. + +## Tips + +- Really long `source([=[...]=])` blocks may break syntax highlighting. Try + `:syntax sync fromstart` to fix it. + + +Lint +---- + +`make lint` (and `make testlint`) runs [luacheck](https://github.com/mpeterv/luacheck) +on the test code. + +If a luacheck warning must be ignored, specify the warning code. Example: + + -- luacheck: ignore 621 + +http://luacheck.readthedocs.io/en/stable/warnings.html + +Ignore the smallest applicable scope (e.g. inside a function, not at the top of +the file). + +Layout +------ + +- `/test/benchmark` : benchmarks +- `/test/functional` : functional tests +- `/test/unit` : unit tests +- `/test/config` : contains `*.in` files which are transformed into `*.lua` + files using `configure_file` CMake command: this is for acessing CMake + variables in lua tests. +- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions + parser: normally used to make macros not accessible via this mechanism + accessible the other way. +- `/test/*/preload.lua` : modules preloaded by busted `--helper` option +- `/test/**/helpers.lua` : common utility functions for test code +- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with + `_spec.lua` are libraries like `/test/**/helpers.lua`, except that they have + some common topic. + +Tests in `/test/unit` and `/test/functional` are normally divided into groups +by the semantic component they are testing. + +Environment variables +--------------------- Test behaviour is affected by environment variables. Currently supported (Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, -treated as Integer; when defined, treated as String; !must be defined to -function properly): +treated as Integer; when defined, treated as String; when defined, treated as +Number; !must be defined to function properly): `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote @@ -99,3 +332,12 @@ get backtrace from). approximately 90% of the tests. Should be used when finding cores is too hard for some reason. Normally (on OS X or when `NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores are checked for after each test. + +`NVIM_TEST_RUN_TESTTEST` (U) (1): allows running `test/unit/testtest_spec.lua` +used to check how testing infrastructure works. + +`NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level: `0` +disables tracing (the fastest, but you get no data if tests crash and there was +no core dump generated), `1` or empty/undefined leaves only C function cals and +returns in the trace (faster then recording everything), `2` records all +function calls, returns and lua source lines exuecuted. |