[c++] Support dynamic linking to libspdlog

[jdblischak]

When installing libtiledbsoma in a conda environment, we want to dynamically link to the binaries installed by conda. Currently our conda-build in tiledbsoma-feedstock dynamically links to the conda-installed libfmt but not to the conda-installed libspdlog.

This was discovered in the PR to add tiledbsoma to conda-forge (conda-forge/staged-recipes#28828). The contributor added a patch that allowed libtiledbsoma to dynamically link to the conda-installed libspdlog. We need to evaluate this patch. While we want to enable dynamically linking to an existing libspdlog installed in the system, we also need to ensure that we continue to support a bootstrapped installation where we download libspdlog and statically link it.

For more details, please see the discussion starting at conda-forge/staged-recipes#28828 (comment)

[jdblischak]

Now that I am aware that we aren't dynamically linking against the conda-installed libspdlog, this is allowing me to re-interpret our conversation in the Issue dedicated to troubleshooting the C++20 compilation errors. My mental model was that we were linking against the conda-installed libspdlog, but it appears that we were explicitly not doing that for some reason.

It's a super-long Issue that discusses various things. Below are comments relevant to the discussion of linking against spdlog:

• I noted that the passing centralized nightly build (ie non conda) builds spdlog as an external project (Nightly feedstock build failed TileDB-Inc/tiledbsoma-feedstock#230 (comment))
• In hindsight I mistakenly argued that we've been building against conda-installed spdlog (turns out we've been installing it in the conda env but then ignoring it during the build) Nightly feedstock build failed TileDB-Inc/tiledbsoma-feedstock#230 (comment)
• Intriguingly, @nguyenv explicitly argued for not using the conda-installed spdlog in Nightly feedstock build failed TileDB-Inc/tiledbsoma-feedstock#230 (comment)

  I think we should build spdlog as an external project rather than relying on the conda-forge version.

This raises a lot of questions for me. What are our requirements in regards to linking against spdlog that we have to statically link it instead of dynamically linking against a system version (eg installed by conda)? Does the patch used in the conda-forge recipe reveal how we could accomplish linking against a system version?

[johnkerl]

This raises a lot of questions for me

I defer to @nguyenv and/or @XanthosXanthopoulos (in addition to @jdblischak ).

All I want is green CI, in this repo as well as the tiledbsoma-feedstock repo, using C++20.

I'm happy with whatever it takes to get us there.

[nguyenv]

Sorry for any confusion, but I only suggested building spdlog as an external project because it failed with the conda-forge version but passed when built externally. I assumed that meant there was some incompatibility with using C++20 and the conda-forge version of spdlog. There wasn’t any other reason behind the suggestion.

[jdblischak]

Sorry for any confusion, but I only suggested building spdlog as an external project because it failed with the conda-forge version but passed when built externally. I assumed that meant there was some incompatibility with using C++20 and the conda-forge version of spdlog. There wasn’t any other reason behind the suggestion.

@nguyenv thanks for the clarification! Now I better understand your previous suggestion.

Thus I think we should be able to fix the CMake files to support either static linking an external project or dynamically linking a system library for libspdlog. I am not the CMake expert among us, so if someone better understands the patch I linked above, please feel free to experiment. Otherwise I can attempt to investigate more next week.

[XanthosXanthopoulos]

These finding match what I saw with some experiments when debugging the C++20 builds. There where some linking errors regarding spdlog when I was simulating the local build as a single output recipe. I will experiment with the patch locally and report back any findings.

[XanthosXanthopoulos]

These finding match what I saw with some experiments when debugging the C++20 builds. There where some linking errors regarding spdlog when I was simulating the local build as a single output recipe. I will experiment with the patch locally and report back any findings.

Replicating the steps in the patch file, I haven't managed to dynamically link spdlog. When adding add_definitions(-DSPDLOG_COMPILED_LIB) the are linker errors. Also following the examples in spdlog's github page the linker error persist. In any variation of the CMAKE build that managed to link successfully, libspdlog was missing from the dependencies.

[jdblischak]

Some more thoughts after having seen TileDB-Inc/TileDB#5427:

1. We may not be able to fix this only in SOMA. We need all our software that interacts with spdlog to do so uniformly
2. IIUC the goal of our source builds is to statically link spdlog and prevent any of its symbols from leaking between binaries (eg between libtiledb and libtiledbsoma). Ideally we can configure our conda builds to behave differently than this default: instead we would want all of our conda binaries to be dynamically linked to the same major.minor version of spdlog (which the conda-smithy infrastructure already manages for us via the pins in .ci_support/)

[XanthosXanthopoulos]

Following the discussion in TileDB-Inc/TileDB/pull/5427 and spdlog/pull/3322 I managed to build locally and found the issue. When we migrated to C++20 we added a flag -DSPDLOG_USE_STD_FORMAT to use std::format inside spdlog. The binaries however are built without this flag and that resulted in the missing symbols I was seeing.

The recipe in conda-forge succeeded because TileDB-SOMA 1.15 is still in C++17 and without this flag so the linking was succeesful. So in order to support dynamic linking this flag needs to be unset and only be enabled if we are using the header only version or we are compiling spdlog ourselves.