problems with import

Post by Neal Becker
One problem I often encounter is with import search.
An example of the problem is with the package mercurial. It has
extensions in a subdirectory called 'hgext'.
On fedora, I install mercurial using the vendor package. This creates
/usr/lib64/python2.7/site-packages/hgext/
Later, I want to try out an extension as a non-privileged user.
cd <extension>
python setup.py install --user
Now I also have
~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system
hgext directory, it won't look also in the local one.
Any thoughts?

Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?

http://legacy.python.org/dev/peps/pep-0420/

Chris Angelico

2014-06-27 13:44:03 UTC

Post by Peter Otten
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
http://legacy.python.org/dev/peps/pep-0420/

Unfortunately for the OP, that doesn't seem to be applicable to Python
2.x; also, changes made to Python in response to this list don't
usually apply to 2.x either. Mercurial doesn't work with Python 3,
currently, although some of the improvements to the latest Pythons
have been in response to reported issues with hg, so it's possible
that hg might support 3.5 or 3.6 at some point.

I don't know if there's a 2.x-compatible solution to this problem.

ChrisA

Peter Otten

2014-06-27 14:01:13 UTC

On Fri, Jun 27, 2014 at 11:34 PM, Peter Otten

Post by Peter Otten
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
http://legacy.python.org/dev/peps/pep-0420/

Unfortunately for the OP, that doesn't seem to be applicable to Python
2.x; also, changes made to Python in response to this list don't
usually apply to 2.x either. Mercurial doesn't work with Python 3,
currently, although some of the improvements to the latest Pythons
have been in response to reported issues with hg, so it's possible
that hg might support 3.5 or 3.6 at some point.
I don't know if there's a 2.x-compatible solution to this problem.

http://legacy.python.org/dev/peps/pep-0420/#namespace-packages-today

But this is likely off-topic for python-ideas.

Steven D'Aprano

2014-06-27 17:12:15 UTC

On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote:
[...]

Post by Neal Becker
Now I also have
~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext
directory, it won't look also in the local one.

Re-arrange sys.path so that the local site-packages comes first,
before the global site-packages. (I'm surprised Python doesn't
already do this.)

--
Steven

Antoine Pitrou

2014-06-27 18:33:07 UTC

Re-arrange sys.path so that the local site-packages comes first,
before the global site-packages. (I'm surprised Python doesn't
already do this.)

Then he would have the reverse problem: once he installs a user-local hg
extension, the bundled (official) hg extensions wouldn't be reachable
anymore.

The answer here comes into two possibilities, both of which have to do
with Mercurial and none with Python itself:

1) Mercurial could make hgext a namespace package (see Peter Otten's
answer above)

2) third-party extensions for Mercurial should never install into the
"hgext" package but rather in a separate top-level package or module
(presumable called "hgsomething")

Regards

Antoine.

Steven D'Aprano

2014-06-27 19:19:26 UTC

Re-arrange sys.path so that the local site-packages comes first,
before the global site-packages. (I'm surprised Python doesn't
already do this.)

Then he would have the reverse problem: once he installs a user-local hg
extension, the bundled (official) hg extensions wouldn't be reachable
anymore.

Naturally, but I assumed that the only reason you would install
something locally was if you intended it to over-ride the global
version. If that's not the case, then you're right, it's an issue for
Mercurial to solve.

--
Steven

Neal Becker

2014-06-27 21:02:14 UTC

Re-arrange sys.path so that the local site-packages comes first,
before the global site-packages. (I'm surprised Python doesn't
already do this.)

Then he would have the reverse problem: once he installs a user-local hg
extension, the bundled (official) hg extensions wouldn't be reachable
anymore.

I don't think this is unique to mercurial.

I'd like to have 2 areas for installing extensions to a package:
a system wide and a local.

I think the semantics we'd want is that the 2 trees are effectively merged,
with the local overriding in the event of a conflict

Guido van Rossum

2014-06-27 21:06:29 UTC

Yeah, so in Python 3.3 this is possible through namespace packages (see PEP
420 -- tldr: remove the empty __init__.py). It has been supported for a
long time in Python 2 by setuptools, and you can even do it yourself by
setting the package's __path__ attribute. See also pkgutil.py in the Python
2 stdlib.

Post by Neal Becker

Post by Neal Becker
Now I also have
~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the

system

Post by Neal Becker
hgext
directory, it won't look also in the local one.

Re-arrange sys.path so that the local site-packages comes first,
before the global site-packages. (I'm surprised Python doesn't
already do this.)

Then he would have the reverse problem: once he installs a user-local hg
extension, the bundled (official) hg extensions wouldn't be reachable
anymore.

I don't think this is unique to mercurial.
a system wide and a local.
I think the semantics we'd want is that the 2 trees are effectively merged,
with the local overriding in the event of a conflict
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--
--Guido van Rossum (python.org/~guido)

Antoine Pitrou

2014-06-27 23:52:11 UTC

Post by Neal Becker
I don't think this is unique to mercurial.
a system wide and a local.
I think the semantics we'd want is that the 2 trees are effectively merged,
with the local overriding in the event of a conflict

Who is "we"? Mercurial extensions have to be enabled manually in your
.hgrc, so whether they live in the hgext namespace or anywhere else is
quite irrelevant.

Regards

Antoine.

Stephen J. Turnbull

2014-06-28 15:48:01 UTC

Post by Neal Becker
I think the semantics we'd want is that the 2 trees are effectively
merged, with the local overriding in the event of a conflict

Maybe. XEmacs does such overriding, and my experience is that users
expect DWIM behavior (the "best" version gets used). Typically local
trees get out of date and may not be compatible with newer versions of
the main tree updated by the OS's PMS, etc. It makes things hard to
diagnose.

Andrew Barnert

2014-06-28 22:02:29 UTC

Post by Neal Becker
I think the semantics we'd want is that the 2 trees are effectively
merged, with the local overriding in the event of a conflict

Isn't Python already flexible enough here? Whichever one comes first in sys.path wins. If you (as a distro packager, a sysadmin, a user, or even a program at runtime) want to customize that order, it's trivial to do so. If even that isn't good enough, you can replace almost any piece of the import machinery pretty easily—you could write a custom finder that finds all versions and picks the one with the newest timestamp, or whatever you think is better. So, what do people want here that Python doesn't do?

Meanwhile, what I personally prefer is to let local beat global, as long as I have an easy way to check when that may not be a good idea anymore. For example, I had PyObjC 2.5.0 installed for my system Python 2.7, back when Apple was using 2.4.something. At some point, Apple switched to 2.5.1, so after installing that system upgrade, my local copy was no longer needed, or wanted. Fortunately, I have a script that I wrote for just that occasion that told me about it so I could uninstall it.

I definitely wouldn't want Python to automatically start ignoring my 2.5.0 because there's a 2.5.1. I'd love it if something (Python, Apple's installer, a script that came with either Python or OS X, whatever) would alert me to the problem so I didn't need my own script, but I'm not expecting that.

Also, what's right for my primary dev machine is not necessarily what's right for my company's testing systems or live deployed servers, or our customers' disparate systems; all I can really do there is require PyObjC 2.5.0 and let whoever's in charge of the machine figure out that they may be able to get that by uninstalling instead of upgrading. Anything that tries to DWIM its way out of that problem is going to get things mysteriously wrong as often as it helps.

Nick Coghlan

2014-06-27 22:18:27 UTC

Post by Steven D'Aprano
Naturally, but I assumed that the only reason you would install
something locally was if you intended it to over-ride the global
version. If that's not the case, then you're right, it's an issue for
Mercurial to solve.

Local installs are supported to *add* non-conflicting user specific
packages to the system Python installation, not to override them. The fact
it's tricky to override the standard library and system provided libraries
helps reduce the attack surface when running software with elevated access,
but still as a specific user.

Packages *can* opt-in to allowing contributions of submodules from later
sys.path entries by declaring that package as a namespace package. In
Python 3.3+ that's as simple as leaving __init__.py out entirely. In any
version, it can be done explicitly using pkgutil.extend_path() (standard
library) or pkg_resources.declare_namespace() (published as part of
setuptools)

Even with namespace packages, though, *modules* earlier in sys.path still
take precedence over later entries.

Cheers,
Nick.

Post by Steven D'Aprano
--
Steven
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Eric Snow

2014-06-27 21:39:48 UTC