[Request] cURL compile-time option

[TurtleP]

Heya,

I spoke with @MikuAuahDark briefly about this on Discord. It would be nice if lua-https could be statically compiled instead of dynamically linked. It makes it a hassle to update my fork for homebrew usage. I'm hoping I can make a pull request at some point, but I'm not great at dealing with fixing something like this, unless I use a ton of #ifdef markings which can look ugly.

Statically linking would let me at least only need to update the config.h file (and rename main.cpp to https.cpp to not conflict names just as a safety measure).

[slime73]

As far as I know in order for the code to be statically linked to another dll/executable which uses it, nothing would need to change except the CMakeLists file. love already uses the unmodified code with its own build system to compile it into the rest of love's library (which itself is statically linked to the main executable on some platforms), for example.

[TurtleP]

Hm, I see. Although looking again, the init of the cURL struct does try to load the symbols from the DLL or shared library. If it can't load a handle from one of those, it will not load symbols.

[MikuAuahDark]

I think she meant this part of the code.

lua-https/src/generic/CurlClient.cpp

Lines 39 to 82 in 9611092

	CurlClient::Curl::Curl()
	: handle(nullptr)
	, loaded(false)
	, global_cleanup(nullptr)
	, easy_init(nullptr)
	, easy_cleanup(nullptr)
	, easy_setopt(nullptr)
	, easy_perform(nullptr)
	, easy_getinfo(nullptr)
	, slist_append(nullptr)
	, slist_free_all(nullptr)
	{
	#ifdef _WIN32
	handle = (void *) LoadLibraryA("libcurl.dll");
	#else
	handle = dlopen("libcurl.so.4", RTLD_LAZY);
	#endif
	if (!handle)
	return;
	// Load symbols
	decltype(&curl_global_init) global_init = nullptr;
	if (!loadSymbol(global_init, handle, "curl_global_init"))
	return;
	if (!loadSymbol(global_cleanup, handle, "curl_global_cleanup"))
	return;
	if (!loadSymbol(easy_init, handle, "curl_easy_init"))
	return;
	if (!loadSymbol(easy_cleanup, handle, "curl_easy_cleanup"))
	return;
	if (!loadSymbol(easy_setopt, handle, "curl_easy_setopt"))
	return;
	if (!loadSymbol(easy_perform, handle, "curl_easy_perform"))
	return;
	if (!loadSymbol(easy_getinfo, handle, "curl_easy_getinfo"))
	return;
	if (!loadSymbol(slist_append, handle, "curl_slist_append"))
	return;
	if (!loadSymbol(slist_free_all, handle, "curl_slist_free_all"))
	return;
	global_init(CURL_GLOBAL_DEFAULT);
	loaded = true;
	}

Currently we're loading cURL dynamically so users doesn't have to install cURL for lua-https to work, in which case it will fall back to OpenSSL on Linux.

[slime73]

Is dlsym not supported on your platform?

(I know there's a dlopen in the code as well, but that probably isn't really necessary and maybe we can remove the if (!handle) return in favour of also checking the global namespace, or something.)

[TurtleP]

No, dlsym is not supported. Homebrew does not support dynamically linking/opening of libraries (except on Wii U but I'm still linking statically on there).

[slime73]

dlsym doesn't actually dynamically link with or open libraries itself, it just looks through already loaded symbols (which could be from a static library as long as they weren't stripped away as an optimization step). It can look through the global symbol list for the program rather than inside a specific library handle.

So it's much more likely to be supported on limited platforms compared to dlopen - for example iOS used to disallow dlopen entirely but still allowed dlsym, which let LuaJIT's FFI work even though dynamically loading a library did not.

[TurtleP]

Gotcha, but official comment from the homebrew toolchain maintainers is that dlsym is not supported. They also don't keep/maintain a symbol table at runtime either.

[bartbes]

Whilst working on an OpenSSL3 backend, I once again had to duplicate the code to load symbols, and I thought of a solution that might work for you as well:

Instead of having 3 or 4 ad-hoc implementations of library loading (dlopen/dlsym/dlclose + loadSymbol), we could extract this to its own header and source files.
We could then make windows and unix version of this (GetProcAddress vs dlsym), abstracting that logic away from the implementation.
This would then also allow you to provide your own loader, which could no-op dlopen/dlclose, and just look up directly linked libraries in a string table for dlsym. Slightly more (and uglier) work on that side, but it means implementations are oblivious to the changes.

I've got a some of this work done already, in my local tree.

[MikuAuahDark]

That's great idea. What do you think @TurtleP?

[TurtleP]

look up directly linked libraries in a string table for dlsym

Only question I have is if this requires dlsym, etc to exist. On the platforms I am working on (3DS, Switch, and Wii U) dlsym, dlopen, dlclose.. these don't exist. We don't have dynamic linking library support so I end up building this library statically linked against the binary. That's the main concern. If your solution allows me to keep doing that, then I am fine with the proposal, @bartbes.

[MikuAuahDark]

The bartbes's idea is basically abstracting away that dlsym, dlopen, and dlclose function.

In your platform, you can implement dlopen that only return opaque handle when certain string is passed in (e.g. if it's "curl" then pass opaque pointer that denotes "curl" library). With the same opaque pointer, you implement dlsym such that it just return the function pointer directly based on string name lookup (e.g. dlsym(handle, "curl_easy_setopt") is as simple as return &curl_easy_setopt). You then can implement dlclose as no-op.

Note that the name dlopen, dlsym, and dlclose will be different depending how bartbes implement things. In overall it's a good idea though.

[TurtleP]

Seems reasonable, although I would prefer that upstream (this repo) be able to just set that up for me, rather than me needing to patch lua-https for static linking. This way I can set an option and be done.

[MikuAuahDark]

Well, we can't do that because we don't really know how your platform behaves (and we don't have the toolchain and hardware to test it), so it's up to you to implement them.