This doesn't seem like something that should require generating assembly to solve. Couldn't `CALLBACK` just return a table, which contains both the userdata pointer to `REAL_CALLBACK` and a value for `findex`, eliminating the global variable? Then `Add` could extract that. You could even make the table returned by `CALLBACK` callable in Lua with a metatable.
Or, if you're worried about performance/memory, you could allocate a struct for `CALLBACK` to return as userdata, containing the function pointer and `findex`. If you made a little allocator for these it would be extremely fast.
I'm sure I'm missing things, but the solution you chose feels like the nuclear option.
The code is passing the function pointers into Win32 APIs, so the caller side isn’t controlled; the callbacks have to work as native C function pointers.
This was probably posted in response to this other link from two days ago, which is about about JIT-compiling wndproc callbacks in particular; the comments discuss the “proper” way to do it, which is to use GWLP_USERDATA:
At least, that’s the proper way to do it if you control the entire application. But for what’s apparently a thin wrapper that exposes Win32 APIs directly to Lua, I can understand picking an approach that makes life easier for the Lua code author, even if it’s hackier under the hood. It also avoids the need to write custom behavior for each API.
It was inspired by that post, but not in response to it. Over the 3 or 4 months that I took writing lowkPRO, one of the features I'm most proud of is this, the ability for it to create C callbacks on the fly that just call Lua callbacks and return the value to C. And I would argue that it's not hacky but just well-engineered. After all, everything compiles to assembly at the end of the day, right?
Well designed C APIs have a context/userdata parameter on their callbacks, which is registered and stored alongside the function pointer. Unfortunately WNDPROC lacks this parameter.
GWLP_USERDATA should be the best option, though the API for setting it and setting the WNDPROC being separate looks error prone.
The bigger problem is that I would have to automate this for every callback type in the Windows API, and there's no guarantee that all of them follow the same 2 or 3 patterns for passing the context pointer into the callback. This solution works great, even if it is a bit wasteful of the ctx ptr.
Lua has its own allocator, which will also collect for you. lua_newuserdata. At the expense of having to set executable yourself, but without all the inbuilt inefficiencies the article points out.
Can you set arbitrary memory allocated by malloc (e.g. HeapAlloc in Windows) to executable, via VirtualProtect or something else? If so, that's news to me. I thought it had to be memory allocated by VirtualAlloc only.
That said, I'm not sure that solution beats out this one. I'm using a linked list on top of an arena allocator in practice, which means allocations happen once every 0x10000 bytes. (A single C closure of mine takes up exactly 16 bytes, which includes the pointer to the next linked list node. I'm very happy with how it's designed.)
I wonder if there would be a way to piggyback on top of GCC's nested function extension. It does something a little bit similar, with the dynamically generated functions.
This doesn't seem like something that should require generating assembly to solve. Couldn't `CALLBACK` just return a table, which contains both the userdata pointer to `REAL_CALLBACK` and a value for `findex`, eliminating the global variable? Then `Add` could extract that. You could even make the table returned by `CALLBACK` callable in Lua with a metatable.
Or, if you're worried about performance/memory, you could allocate a struct for `CALLBACK` to return as userdata, containing the function pointer and `findex`. If you made a little allocator for these it would be extremely fast.
I'm sure I'm missing things, but the solution you chose feels like the nuclear option.
The code is passing the function pointers into Win32 APIs, so the caller side isn’t controlled; the callbacks have to work as native C function pointers.
This was probably posted in response to this other link from two days ago, which is about about JIT-compiling wndproc callbacks in particular; the comments discuss the “proper” way to do it, which is to use GWLP_USERDATA:
https://news.ycombinator.com/item?id=46259334
At least, that’s the proper way to do it if you control the entire application. But for what’s apparently a thin wrapper that exposes Win32 APIs directly to Lua, I can understand picking an approach that makes life easier for the Lua code author, even if it’s hackier under the hood. It also avoids the need to write custom behavior for each API.
It was inspired by that post, but not in response to it. Over the 3 or 4 months that I took writing lowkPRO, one of the features I'm most proud of is this, the ability for it to create C callbacks on the fly that just call Lua callbacks and return the value to C. And I would argue that it's not hacky but just well-engineered. After all, everything compiles to assembly at the end of the day, right?
Well designed C APIs have a context/userdata parameter on their callbacks, which is registered and stored alongside the function pointer. Unfortunately WNDPROC lacks this parameter.
GWLP_USERDATA should be the best option, though the API for setting it and setting the WNDPROC being separate looks error prone.
The bigger problem is that I would have to automate this for every callback type in the Windows API, and there's no guarantee that all of them follow the same 2 or 3 patterns for passing the context pointer into the callback. This solution works great, even if it is a bit wasteful of the ctx ptr.
Fun fact: LuaJIT FFI actually has a similar feature. You can do funky things like detour hooking with this functionality too.
https://luajit.org/ext_ffi_semantics.html
I feel libffi is better suited and supports most platforms.
Tsoding made a video about it : https://www.youtube.com/watch?v=0o8Ex8mXigU
The ffcall library even more
wasn't aware of ffcall, it's definitely a better fit
What’s the thing emulating so much activity around closures lately?
https://news.ycombinator.com/item?id=46228597
https://news.ycombinator.com/item?id=46259334
Also talking about the Knuth boy/man test: https://news.ycombinator.com/item?id=46020151
Not a bad thing, but that really question if there is some active micro-
Why VirtualAlloc?
Lua has its own allocator, which will also collect for you. lua_newuserdata. At the expense of having to set executable yourself, but without all the inbuilt inefficiencies the article points out.
Can you set arbitrary memory allocated by malloc (e.g. HeapAlloc in Windows) to executable, via VirtualProtect or something else? If so, that's news to me. I thought it had to be memory allocated by VirtualAlloc only.
That said, I'm not sure that solution beats out this one. I'm using a linked list on top of an arena allocator in practice, which means allocations happen once every 0x10000 bytes. (A single C closure of mine takes up exactly 16 bytes, which includes the pointer to the next linked list node. I'm very happy with how it's designed.)
Lua lets you set the allocator, so you don't have to manage the lifetime.
I wonder if there would be a way to piggyback on top of GCC's nested function extension. It does something a little bit similar, with the dynamically generated functions.