channel/msgpack_rpc: Refactor API dispatching

This is how API dispatching worked before this commit:

- The generated `msgpack_rpc_dispatch` function receives a the `msgpack_packer`
  argument.
- The response is incrementally built while validating/calling the API.
- Return values/errors are also packed into the `msgpack_packer` while the
  final response is being calculated.

Now the `msgpack_packer` argument is no longer provided, and the
`msgpack_rpc_dispatch` function returns `Object`/`Error` values to
`msgpack_rpc_call`, which will use those values to build the response in a
single pass.

This was done because the new `channel_send_call` function created the
possibility of having recursive API invocations, and this wasn't possible when
sharing a single `msgpack_sbuffer` across call frames(it was shared implicitly
through the `msgpack_packer` instance).

Since we only start to build the response when the necessary information has
been computed, it's now safe to share a single `msgpack_sbuffer` instance
across all channels and API invocations.

Some other changes also had to be performed:

- Handling of the metadata discover was moved to `msgpack_rpc_call`
- Expose more types as subtypes of `Object`, this was required to forward the
  return value from `msgpack_rpc_dispatch` to `msgpack_rpc_call`
- Added more helper macros for casting API types to `Object`
  any

1 files changed, 17 insertions, 1 deletions

diff --git a/src/nvim/api/private/defs.h b/src/nvim/api/private/defs.h
index ee0fc02c4d..b049412014 100644
--- a/src/nvim/api/private/defs.h
+++ b/src/nvim/api/private/defs.h
@@ -65,8 +65,16 @@ typedef enum {
   kObjectTypeInteger,
   kObjectTypeFloat,
   kObjectTypeString,
+  kObjectTypeBuffer,
+  kObjectTypeWindow,
+  kObjectTypeTabpage,
   kObjectTypeArray,
-  kObjectTypeDictionary
+  kObjectTypeDictionary,
+  kObjectTypePosition,
+  kObjectTypeStringArray,
+  kObjectTypeBufferArray,
+  kObjectTypeWindowArray,
+  kObjectTypeTabpageArray,
 } ObjectType;
 
 struct object {
@@ -76,8 +84,16 @@ struct object {
     Integer integer;
     Float floating;
     String string;
+    Buffer buffer;
+    Window window;
+    Tabpage tabpage;
     Array array;
     Dictionary dictionary;
+    Position position;
+    StringArray stringarray;
+    BufferArray bufferarray;
+    WindowArray windowarray;
+    TabpageArray tabpagearray;
   } data;
 };