socket.getfqdn() doesn't cope properly with purely DNS-based setups #49254
Comments
|
On Linux and presumably on other POSIX-like systems, socket.getfqdn() My system's FQDN is 'fugue.tank.wellohorld.com'. My /etc/hosts is empty dfranke@fugue:~/Python-2.6.1$ hostname
fugue
dfranke@fugue:~/Python-2.6.1$ hostname -f
fugue.tank.wellohorld.com
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue'
>>>
dfranke@fugue:~/Python-2.6.1$ echo -e '$a\n172.17.0.120
fugue.tank.wellohorld.com fugue\n.\nwq' | sudo ed /etc/hosts
305
350
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue.tank.wellohorld.com'
>>>
dfranke@fugue:~/Python-2.6.1$ |
|
Would someone with appropriate knowledge please take a look to see if this is still an issue. |
|
I think anybody willing to invest the time could acquire the appropriate knowledge, at least to determine whether it's still an issue (i.e. trying to reproduce it). To fix it, one would then need to read the source code of hostname, and find out what they do differently; strace might be sufficient already. |
|
I cannot reproduce this issue. I just tested this on my mac. atoshniw@prusev-mn:~/Documents/code/python-dev/bin #hostname -f
prusev-mn.helloworld.com
atoshniw@prusev-mn:~/Documents/code/python-dev/bin #python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'prusev-mn.helloworld.com' |
|
Gave this a go myself... $ ./python
Python 3.4.0a0 (default:57a33af85407, Oct 27 2012, 21:26:30)
[GCC 4.4.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'host.domain.com'
>>> $ hostname -f
host.domain.com
$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost# The following lines are desirable for IPv6 capable hosts Linux host 3.5.2-x86_64 #1 SMP Wed Aug 15 14:31:07 EDT 2012 x86_64 GNU/Linux According to strace, both rely on DNS: recvfrom(3, "Wj\201\200\0\1\0\1\0\5\0\0\00219\003134\003230\003173\7in-a"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.2.3.4")}, [16]) = 176 Same behavior on both 2.6 & hg tip. I think this is a non-issue. |
|
Still seeing this on Fedora 18 / Python 2.7.3. I only have loopback in /etc/hosts [TUE\shoop@pclin281] <~> cat /etc/hosts I search in .campus.tue.nl and .win.tue.nl: [TUE\shoop@pclin281] <~> grep search /etc/resolv.conf Hostname -f reliably returns .campus.tue.nl as it should [TUE\shoop@pclin281] < But socket.getfqdn disagrees, even with itself when run multiple times: [TUE\shoop@pclin281] <~> python
Python 2.7.3 (default, Aug 9 2012, 17:23:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> Note that pclin281.win.tue.nl is in fact also a valid DNS entry, but not one that I would expect the function to ever return given the search order. |
|
Note that socket.getfqdn is a wrapper around a couple of socket calls that are just wrappers of OS level socket calls. If you take a look at socket.py you'll see the definition. As Martin said earlier, if you (or anyone else) can figure out what hostname does differently and suggest how to patch our getfqdn method to behave similarly, I'm sure the patch will be welcome. Unfortunately there won't be any good way to write a test for this. |
|
OK, fair enough. From reading sources, it appears that hostname is using getaddrinfo(3) on kernelhostname with hints->ai_flags & AI_CANONNAME, while Lib/socket.py simply uses gethostbyaddr(kernelhostname), and falls back on kernelhostname in case of errors. Unfortunately I am not entirely sure who is "correct" here, as I don't know the intent of socket.getfqdn(). In my case, kernelhostname is set to 'pclin281' e.g. without the dots. I believe this to be correct, but I know that this is already "controversial" as in there exists software that expects an FQDN there, and internet folklore is split about 50/50 about this necessity. Then, apparently, there is confusion about AI_CANONNAME and what it actually should do. glibc upstream does address lookups but Fedora patches this out. See this recent glibc bug for more pointers: http://sourceware.org/bugzilla/show_bug.cgi?id=15218 As mentioned in that bug, a lot of software runs on Fedora and works using that definition of AI_CANONNAME. However, switching Lib/socket.py / getfqdn from gethostbyaddr to getaddrinfo might have more implications than just fixing this case. I can try to write a patch, but is this the right direction? |
|
Attached is a very lightly tested patch that matches hostname -f behaviour on my system. I suspect this should be OK but it definitely needs more testing than just my system... |
|
The problem with your patch is that it changes the (effective) meaning of the 'name' parameter. Before the patch, name can be an IP address. After the patch, that will fail on Fedora. (It also fails on my Gentoo system). It is interesting to note, as well, that the documentation for gethostbyaddr says that it is obsolete and getaddrinfo should be used instead. Could we use the getaddrinfo call if we don't get an FQDN back from gethostbyaddr? It doesn't look like that would completely solve your problem, though, given your example output. Have you figured out why that is happening? Alternatively, perhaps we could fall back to gethostbyaddr if we don't get an fqdn from the getaddrinfo call. However, given that the documentation actually specifies the algorithm used by getfqdn, I'm not sure if we can make either change in a bugfix version. |
|
OK, dumping my current findings here, as I'm still not sure what the expected results should be. First of all, Lib/socket.py calls gethostbyaddr with a name. As in, gethostby _ADDR_ with a name. This works because Modules/socketmodule.c internally uses setipaddr() to resolve the name to an address. setipaddr() does this using a call to getaddrinfo() with hints.ai_family == AF_UNSPEC and no further flags. On my system (confirmed using the test program attached) this results in SIX entries, and this is the part that confused me. Due to virtualization I have a virtual bridge virbr0 configured with an internal IP address 192.168.122.1, as well as my LAN-connected bridge br0 with IP address 131.155.71.8. Both of these addresses are returned in the call to getaddrinfo() (each one 3 times), but NOT ALWAYS IN THE SAME ORDER. And this is the clue as to why python's socket.getfqdn() does not behave consistently. For 192.168.122.1 does not resolve to anything, hence it will return "pclin281". And 131.155.71.8 will backwards resolve to pclin281.win.tue.nl as the PTR record points to that entry. Now, again, I'm not entirely sure what to do here. I agree that this is not a simple bugfix. I also think that, apart from the weirdness of getaddrinfo() return order, socket.getfqdn() is doing it's documented job of returning /an/ FQDN for a given host. But in case of the guaranteed LOCAL canonical hostname, another function is warranted, imho. Does this make sense? For the record, output of a given run on my system: [TUE\shoop@pclin281] <~/tmp> ./test |
|
Yeah, a new function was a thought that had crossed my mind as well. getfqdnbyname, maybe? Or gethostnamefqdn? Then deprecate calling getfqdn without an argument. I agree that gethostbyaddr accepting a non-IP is weird. I have no idea why it was implemented that way, much less why it is *used* that way. It's been that way for a long time, though. |
|
So after a good nights sleep: does it not make sense to use the canonical hostname iff the name argument is not present / empty? Otherwise, fall back to the documented steps? That way extra API is avoided, and I can't think of a case where you would rather have my weird results vs "the output of hostname -f". |
|
That is an interesting proposal, yes. I suppose someone that needs the getaddrinfo semantics for something other than the local host can just call it directly. Now, do we add the fact that we are doing this to the current alogarithmic documention? :) |
|
According to the man page for gethostbyaddr "The gethostbyname*() and gethostbyaddr*() functions are obsolete. Applications should use getaddrinfo(3) and getnameinfo(3) instead." - so perhaps using the correct API call might be a good start to resolving this issue, but I found that in my case, I needed to chase the problem upstream instead of downstream. On my Red Hat box, the kernel.hostname value with sysctl was incorrect. I had to re-set it with a sysctl kernel.hostname=hostname.example.com. This overrides /etc/hosts, so I suspect this is probably not an issue on other distros that do not use sysctl. The moral of the story being garbage in, garbage out. |
|
In my case, /etc/hostname, /proc/sys/kernel/hostname, |
|
Embarassing as always to stumble over some stuff and then find a 9y old ticket here, where it is discussed and even (almost) solved. Our ticket: borgbackup/borg#3471 Fixed getfqdn we use now instead of socket.getfqdn(): borgbackup/borg@9b0d0f3#diff-4b53f84e19a3bb376bf2202371ed269aR188 Note: no "else: name = hostname" at the end, that is a bug in the patch attached to this ticket (hostname is undefined after applying the patch). |
|
Confirming the fixed version linked in previous comment by Thomas Waldmann is correct and matches what |
|
I just ran into this 12 year old issue. Can this be merged please? |
|
Could you or somebody else please create a PR with patch and a test case? |
|
Here is the updated patch. Is python5004-test.c enough as a test case? |
|
We no longer accept patches. Contributors have to create a PR on GitHub, so we can record contributions and verify the contributor license agreement. |
|
In that case Stijn Hope should create the PR since he wrote the patch. Anyone else could get in trouble for using his code without proper permission. |
|
I hereby put my patch in the public domain and/or under any desired Regards, Stijn Hoop On Fri, 22 Oct 2021 21:03:26 +0000
|
We keep on having repeated issue reports about non-matching hostname of workers. This seems to be trceable to getfqdn method of socket in Kubernetes that in some circumstances (race condition with netwrking setup when starting) can return different hostname at different times. There seems to be a related issue in Python that has not been resolved in more than 13 years (!) python/cpython#49254 The error seems to be related to the way how canonicalname is derived by getfqdn (it uses gethostbyaddr which sometimes provides different name than canonical name (it returns the first DNS name resolved that contains "."). We are fixing it in two ways: * instead of using gethostbyaddr we are using getadddrinfo with AI_CANONNAME flag which (according to the docs): https://man7.org/linux/man-pages/man3/getaddrinfo.3.html If hints.ai_flags includes the AI_CANONNAME flag, then the ai_canonname field of the first of the addrinfo structures in the returned list is set to point to the official name of the host. * we are caching the name returned by first time retrieval per interpreter. This way at least inside the same interpreter, the name of the host should not change.
We keep on having repeated issue reports about non-matching hostname of workers. This seems to be trceable to getfqdn method of socket in Kubernetes that in some circumstances (race condition with netwrking setup when starting) can return different hostname at different times. There seems to be a related issue in Python that has not been resolved in more than 13 years (!) python/cpython#49254 The error seems to be related to the way how canonicalname is derived by getfqdn (it uses gethostbyaddr which sometimes provides different name than canonical name (it returns the first DNS name resolved that contains "."). We are fixing it in two ways: * instead of using gethostbyaddr we are using getadddrinfo with AI_CANONNAME flag which (according to the docs): https://man7.org/linux/man-pages/man3/getaddrinfo.3.html If hints.ai_flags includes the AI_CANONNAME flag, then the ai_canonname field of the first of the addrinfo structures in the returned list is set to point to the official name of the host. * we are caching the name returned by first time retrieval per interpreter. This way at least inside the same interpreter, the name of the host should not change.
We keep on having repeated issue reports about non-matching hostname of workers. This seems to be trceable to getfqdn method of socket in Kubernetes that in some circumstances (race condition with netwrking setup when starting) can return different hostname at different times. There seems to be a related issue in Python that has not been resolved in more than 13 years (!) python/cpython#49254 The error seems to be related to the way how canonicalname is derived by getfqdn (it uses gethostbyaddr which sometimes provides different name than canonical name (it returns the first DNS name resolved that contains "."). We are fixing it in two ways: * instead of using gethostbyaddr we are using getadddrinfo with AI_CANONNAME flag which (according to the docs): https://man7.org/linux/man-pages/man3/getaddrinfo.3.html If hints.ai_flags includes the AI_CANONNAME flag, then the ai_canonname field of the first of the addrinfo structures in the returned list is set to point to the official name of the host. * we are caching the name returned by first time retrieval per interpreter. This way at least inside the same interpreter, the name of the host should not change. GitOrigin-RevId: a3f2df0f45973ddb97990361fdc5caa256c175ca
We keep on having repeated issue reports about non-matching hostname of workers. This seems to be trceable to getfqdn method of socket in Kubernetes that in some circumstances (race condition with netwrking setup when starting) can return different hostname at different times. There seems to be a related issue in Python that has not been resolved in more than 13 years (!) python/cpython#49254 The error seems to be related to the way how canonicalname is derived by getfqdn (it uses gethostbyaddr which sometimes provides different name than canonical name (it returns the first DNS name resolved that contains "."). We are fixing it in two ways: * instead of using gethostbyaddr we are using getadddrinfo with AI_CANONNAME flag which (according to the docs): https://man7.org/linux/man-pages/man3/getaddrinfo.3.html If hints.ai_flags includes the AI_CANONNAME flag, then the ai_canonname field of the first of the addrinfo structures in the returned list is set to point to the official name of the host. * we are caching the name returned by first time retrieval per interpreter. This way at least inside the same interpreter, the name of the host should not change. GitOrigin-RevId: a3f2df0f45973ddb97990361fdc5caa256c175ca
We keep on having repeated issue reports about non-matching hostname of workers. This seems to be trceable to getfqdn method of socket in Kubernetes that in some circumstances (race condition with netwrking setup when starting) can return different hostname at different times. There seems to be a related issue in Python that has not been resolved in more than 13 years (!) python/cpython#49254 The error seems to be related to the way how canonicalname is derived by getfqdn (it uses gethostbyaddr which sometimes provides different name than canonical name (it returns the first DNS name resolved that contains "."). We are fixing it in two ways: * instead of using gethostbyaddr we are using getadddrinfo with AI_CANONNAME flag which (according to the docs): https://man7.org/linux/man-pages/man3/getaddrinfo.3.html If hints.ai_flags includes the AI_CANONNAME flag, then the ai_canonname field of the first of the addrinfo structures in the returned list is set to point to the official name of the host. * we are caching the name returned by first time retrieval per interpreter. This way at least inside the same interpreter, the name of the host should not change. GitOrigin-RevId: a3f2df0f45973ddb97990361fdc5caa256c175ca
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: