build: make generated source files reproducible #21586

Problem:
Build is not reproducible, because generated source files (.c/.h/) are not
deterministic, mostly because Lua pairs() is unordered by design (for security).

https://github.com/LuaJIT/LuaJIT/issues/626#issuecomment-707005671
https://www.lua.org/manual/5.1/manual.html#pdf-next
> The order in which the indices are enumerated is not specified [...]
>
>> The hardening of the VM deliberately randomizes string hashes. This in
>> turn randomizes the iteration order of tables with string keys.

Solution:
- Update the code generation scripts to be deterministic.
    - That is only a partial solution: the exported function
      (funcs_metadata.generated.h) and ui event
      (ui_events_metadata.generated.h) metadata have some mpack'ed
      tables, which are not serialized deterministically.
    - As a workaround, introduce `PRG_GEN_LUA` cmake setting, so you can
      inject a modified build of luajit (with LUAJIT_SECURITY_PRN=0)
      that preserves table order.
    - Longer-term we should change the mpack'ed data structure so it no
      longer uses tables keyed by strings.

Closes #20124

Co-Authored-By: dundargoc <gocdundar@gmail.com>
Co-Authored-By: Arnout Engelen <arnout@bzzt.net>

1 files changed, 4 insertions, 2 deletions

diff --git a/scripts/genvimvim.lua b/scripts/genvimvim.lua
index 868084a583..3e9e7077be 100644
--- a/scripts/genvimvim.lua
+++ b/scripts/genvimvim.lua
@@ -11,6 +11,8 @@ local funcs_file = arg[3]
 
 package.path = nvimsrcdir .. '/?.lua;' .. package.path
 
+_G.vim = loadfile(nvimsrcdir..'/../../runtime/lua/vim/shared.lua')()
+
 local lld = {}
 local syn_fd = io.open(syntax_file, 'w')
 lld.line_length = 0
@@ -115,7 +117,7 @@ end
 local nvimau_start = 'syn keyword nvimAutoEvent contained '
 w('\n\n' .. nvimau_start)
 
-for au, _ in pairs(auevents.nvim_specific) do
+for au, _ in vim.spairs(auevents.nvim_specific) do
   if lld.line_length > 850 then
     w('\n' .. nvimau_start)
   end
@@ -126,7 +128,7 @@ w('\n\nsyn case match')
 local vimfun_start = 'syn keyword vimFuncName contained '
 w('\n\n' .. vimfun_start)
 local funcs = mpack.unpack(io.open(funcs_file, 'rb'):read("*all"))
-for name, _ in pairs(funcs) do
+for _, name in ipairs(funcs) do
   if name then
     if lld.line_length > 850 then
       w('\n' .. vimfun_start)