build: make generated source files reproducible #21586

Problem:
Build is not reproducible, because generated source files (.c/.h/) are not
deterministic, mostly because Lua pairs() is unordered by design (for security).

https://github.com/LuaJIT/LuaJIT/issues/626#issuecomment-707005671
https://www.lua.org/manual/5.1/manual.html#pdf-next
> The order in which the indices are enumerated is not specified [...]
>
>> The hardening of the VM deliberately randomizes string hashes. This in
>> turn randomizes the iteration order of tables with string keys.

Solution:
- Update the code generation scripts to be deterministic.
    - That is only a partial solution: the exported function
      (funcs_metadata.generated.h) and ui event
      (ui_events_metadata.generated.h) metadata have some mpack'ed
      tables, which are not serialized deterministically.
    - As a workaround, introduce `PRG_GEN_LUA` cmake setting, so you can
      inject a modified build of luajit (with LUAJIT_SECURITY_PRN=0)
      that preserves table order.
    - Longer-term we should change the mpack'ed data structure so it no
      longer uses tables keyed by strings.

Closes #20124

Co-Authored-By: dundargoc <gocdundar@gmail.com>
Co-Authored-By: Arnout Engelen <arnout@bzzt.net>

1 files changed, 12 insertions, 0 deletions

diff --git a/runtime/doc/lua.txt b/runtime/doc/lua.txt
index 49b5c9da70..f325c58efb 100644
--- a/runtime/doc/lua.txt
+++ b/runtime/doc/lua.txt
@@ -1694,6 +1694,18 @@ pesc({s})                                                         *vim.pesc()*
     See also: ~
         https://github.com/rxi/lume
 
+spairs({t})                                                     *vim.spairs()*
+    Enumerate a table sorted by its keys.
+
+    Parameters: ~
+      • {t}  (table) List-like table
+
+    Return: ~
+        iterator over sorted keys and their values
+
+    See also: ~
+        Based on https://github.com/premake/premake-core/blob/master/src/base/table.lua
+
 split({s}, {sep}, {kwargs})                                      *vim.split()*
     Splits a string at each instance of a separator.