cpython-3.7.7-linux64-20200409T0045 links against libnsl.so.1, which Fedora no longer has

[durin42]

I'm not sure what to do about this offhand, but it looks like Fedora dropped libnsl.so.1 in Fedora 28 or so (32 is current). Filing this so we at least don't lose track. STR:

[greenwood:Programming/hg/crew] augie% pyoxidizer build --path ./rust/hgcli --release
resolving 1 targets
resolving target app_posix
resolving target distribution_posix
resolving target exe_posix
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20200408/cpython-3.7.7-linux64-20200409T0045.tar.zst", sha256: "74799ae3b7f3ddc2d118516d65d46356fb3ef3ff3c4c4591a0dde073c413aff0" }
Python distribution available at /home/augie/Programming/hg/crew/build/pyoxidizer/python_distributions/cpython-3.7.7-linux64-20200409T0045.tar.zst
reading data from Python distribution...
/home/augie/Programming/hg/crew/build/pyoxidizer/python_distributions/python.74799ae3b7f3/python/install/bin/python3.7m: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory
error[PYOXIDIZER_BUILD]: EOF while parsing a value at line 1 column 0
  --> ./rust/hgcli/pyoxidizer.bzl:26:11
   |
26 |       exe = dist.to_python_executable(
   |  ___________^
27 | |         name = "hg",
28 | |         resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
29 | |         config = config,
...  |
32 | |         extension_module_filter = "all",
33 | |     )
   | |_____^ resolve_distribution()


error: EOF while parsing a value at line 1 column 0

[durin42]

There's a libnsl.so.2, so presumably we have to use that instead. I see various rumors that libnsl can't be statically linked too, so this might be a weird issue to tackle.

[indygreg]

Hmmm. This may be a recent regression. I had documented way back when that libnsl.so.1 was only required by the nis extension, which is a shared library extension module. But you are correct that the python3.8 binary is depending on it.

I think you can install a legacy version of a package on Fedora to provide the missing library. But I could be conflating this with another core system library.

[durin42]

Yes, there's a legacy package, but I figured you'd still want to know. :)

…

On Sat, Aug 22, 2020, 19:40 Gregory Szorc ***@***.***> wrote: Hmmm. This may be a recent regression. I had documented way back when that libnsl.so.1 was only required by the nis extension, which is a shared library extension module. But you are correct that the python3.8 binary is depending on it. I *think* you can install a legacy version of a package on Fedora to provide the missing library. But I could be conflating this with another core system library. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE6LLORHKOHCAU3UWY663SCBJN3ANCNFSM4OZA6R5Q> .

[hakostra]

I just tested the Python 3.8 builds on Centos 8 recently, and it fails there as well, missing libnsl.so.1.

Apparently the libnsl.so.1 were installed as parts of the minimal base OS in Centos/RHEL 7, but this is now an optional package. Being on a public system where I cannot "just install" libnsl myself, this is a (huge) problem...

[indygreg]

So that's Fedora and Centos/RHEL dropping libnsl. I guess we better deal with it.

libnsl isn't part of the LSB specification, so removal of it from the base system and as a required dependency of python-build-standalone distributions seems justified to me.

The question becomes: how? Right now, we statically link all but the test extensions into libpython to provide a fully-features distribution. That's a really nice feature! Unfortunately, that means the nis extension and its libnsl.so.1 dependency is also in libpython. We can't have that if the installed distribution is to be portable. (Although it does appear that libnsl isn't in the PYTHON.json metadata, so downstream reconsumers like PyOxidizer may avoid the dependency assuming they didn't link the nis extension.)

My knee jerk is to prevent the nis extension from being compiled into libpython, making it a shared library extension in the default install. Downstream reconsumers could still statically link via its object files if they wanted. But at least it wouldn't poison the pre-built Python. If we went this route, we may also want an additional annotation in PYTHON.json to indicate a library dependency is non-standard / non-LSB. That we downstream consumers can filter without having to maintain their own lists.

I could also drop the nis extension completely. We already do that on macOS distributions. But this may anger people. Who, I'm not sure: the nis extension and the APIs it is using are old and python-build-standalone is kind of a niche project. I'm willing to bet nobody would notice...

Anyone have any opinions?

[durin42]

I just looked at the nis module, and I can't imagine it's widely used. I'd say remove it and if someone wants to package something that actually _uses_ nis with pyoxidizer, they should pay the costs?

…

On Thu, Mar 4, 2021 at 7:45 PM Gregory Szorc ***@***.***> wrote: So that's Fedora and Centos/RHEL dropping libnsl. I guess we better deal with it. libnsl isn't part of the LSB specification, so removal of it from the base system and as a required dependency of python-build-standalone distributions seems justified to me. The question becomes: how? Right now, we statically link all but the test extensions into libpython to provide a fully-features distribution. That's a really nice feature! Unfortunately, that means the nis extension and its libnsl.so.1 dependency is also in libpython. We can't have that if the installed distribution is to be portable. (Although it does appear that libnsl isn't in the PYTHON.json metadata, so downstream reconsumers like PyOxidizer may avoid the dependency assuming they didn't link the nis extension.) My knee jerk is to prevent the nis extension from being compiled into libpython, making it a shared library extension in the default install. Downstream reconsumers could still statically link via its object files if they wanted. But at least it wouldn't poison the pre-built Python. If we went this route, we may also want an additional annotation in PYTHON.json to indicate a library dependency is non-standard / non-LSB. That we downstream consumers can filter without having to maintain their own lists. I could also drop the nis extension completely. We already do that on macOS distributions. But this may anger people. Who, I'm not sure: the nis extension and the APIs it is using are old and python-build-standalone is kind of a niche project. I'm willing to bet nobody would notice... Anyone have any opinions? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[indygreg]

The reason why PyOxidizer doesn't run into this problem: it has a hard-coded drop list for the _crypt and nis extensions on Linux! I completely forgot those lists existed.

Also, another potential solution here is to build our own static libnsl and distribute that. But if my cursory research is correct, libnsl was split out of glibc. If it is using some of the coding conventions prevalent in glibc, a statically linked library might be a bad idea (tm).

I'm just going to drop the nis extension from our builds. If someone complains, we can add complexity later.

[indygreg]

nis/libnsl should be gone now.

You should be able to get builds from CI in a few hours or build locally, of course. I'll likely perform an official release in the next few days to track down a regression in macOS builds affecting PyOxidizer users.