Accepting keyword arguments for __getitem__

[Stefano Borini]

Dear all,

At work we use a notation like LDA[Z=5] to define a specific level of accuracy for our evaluation. This notation is used
just for textual labels, but it would be nice if it actually worked at the scripting level, which led me to think to the following:
at the moment, we have the following
... def __getitem__(self, y):
... print(y)
...

a=A()
a[2]

2

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}. In the case of a[1:3, 4, Z=3, R=5], the value of y would
be a tuple containing (slice(1,3,None), 4, {"Z": 3}, {"R": 5}). This allows to
preserve the ordering as specified (e.g. a[Z=3, R=4] vs a[R=4, Z=3]).

Do you think it would be a good/useful idea? Was this already discussed or proposed in a PEP?
Google did not help on this regard.

Thank you,

Stefano Borini

[Chris Angelico]

On Mon, Jun 23, 2014 at 10:06 PM, Stefano Borini

Post by Stefano Borini
At work we use a notation like LDA[Z=5] to define a specific level of accuracy for our evaluation. This notation is used
at the moment, we have the following
... print(y)
...

a=A()
a[2]

2

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}.

The obvious way to accept that would be to support keyword arguments,
and then it begins looking very much like a call. Can you alter your
notation very slightly to become LDA(Z=5) instead? Then you can accept
that with your class thus:

class A:
def __call__(self, Z):
print(Z)

Or you can accept it generically with keyword arg collection:

class A:
def __call__(self, **kw):
print(kw)

Post by Stefano Borini

a=A()
a(Z=3)

{'Z': 3}

Requires a small change to notation, but no changes to Python, ergo it
can be done without waiting for a new release!

ChrisA

[Stefano Borini]

Post by Chris Angelico
The obvious way to accept that would be to support keyword arguments,
and then it begins looking very much like a call. Can you alter your
notation very slightly to become LDA(Z=5) instead?

We certainly can, but I was wondering if such extension would be useful in other contexts.
Also, with the function solution, you would lose the order of the entries. You can't distinguish
foo(z=3, r=4) from foo(r=4, z=3)

[Ian Cordasco]

On Mon, Jun 23, 2014 at 7:53 AM, Stefano Borini

Post by Stefano Borini

Post by Chris Angelico
The obvious way to accept that would be to support keyword arguments,
and then it begins looking very much like a call. Can you alter your
notation very slightly to become LDA(Z=5) instead?

We certainly can, but I was wondering if such extension would be useful in other contexts.
Also, with the function solution, you would lose the order of the entries. You can't distinguish
foo(z=3, r=4) from foo(r=4, z=3)

Chris may have missed that requirement (as I did) when they first read
your email. Your desired behaviour matches no other known behaviour in
Python. The only way to achieve that would be to do something akin to:

foo(dict(z=3), dict(r=4))

And the same would be true of your proposed feature for __getitem__
because all keyword arguments would be collected into one dictionary.
It would be unreasonable for just one method to behave totally
differently from the standard behaviour in Python. It would be
confusing for only __getitem__ (and ostensibly, __setitem__) to take
keyword arguments but instead of turning them into a dictionary, turn
them into individual single-item dictionaries.

[Chris Angelico]

On Mon, Jun 23, 2014 at 10:53 PM, Stefano Borini

Post by Stefano Borini

Post by Chris Angelico
The obvious way to accept that would be to support keyword arguments,
and then it begins looking very much like a call. Can you alter your
notation very slightly to become LDA(Z=5) instead?

We certainly can, but I was wondering if such extension would be useful in other contexts.
Also, with the function solution, you would lose the order of the entries. You can't distinguish
foo(z=3, r=4) from foo(r=4, z=3)

Then you're asking for something where the syntax->semantics
translation is very different from the rest of Python. I suspect that
won't fly.

As an alternative, you may want to look into a preprocessor - some
sort of source code or concrete syntax tree transformation (you can't
use an AST transform unless you start with valid, compilable Python).
Translate this:

LDA[z=3, r=4]

into this:

LDA(("z",3),("r",4))

and then parse it off like this:

class A:
def __call__(self, *args):
for name, value in args:
blah blah blah

I rather doubt your proposal would see much support in the rest of the
Python world, so a solution that's specific to your codebase would be
the way to go.

ChrisA

[Stefano Borini]

Post by Ian Cordasco
Chris may have missed that requirement (as I did) when they first read
your email.

I blame my poor choice of subject on that misunderstanding. Apologies.

Post by Ian Cordasco
Your desired behaviour matches no other known behaviour in
foo(dict(z=3), dict(r=4))
And the same would be true of your proposed feature for __getitem__
because all keyword arguments would be collected into one dictionary.
It would be unreasonable for just one method to behave totally
differently from the standard behaviour in Python.
It would be confusing for only __getitem__ (and ostensibly, __setitem__) to take
keyword arguments but instead of turning them into a dictionary, turn
them into individual single-item dictionaries.

I tend to agree, however, the fact is that when you say

a[2,3,4]

__getitem__ is not called with four arguments. It's called with one tuple
argument, which puts it already in a different category than a(2,3,4), where
each entry is bound to individual arguments. It makes sense if you
understand
the comma as a tuple production. With keyword arguments, it would
resemble more
of a namedtuple, at least partially.

The alternative, and accidentally proposed by my subject, would be to have
__getitem__(self, y, **kwargs) and have a[1,2,Z=3,R=4] produce

y=(1,2)
kwargs = {"Z":3, "R": 4}

but that would be equally heterogeneous (no *args), and it would not
preserve ordering.

I am not a big fan either of my own idea. I just threw a bone to see if
it has
already been discussed or if anyone would envision other possible use
cases for
this notation.

[Chris Angelico]

On Tue, Jun 24, 2014 at 1:59 AM, Stefano Borini

I am not a big fan either of my own idea. I just threw a bone to see if it
has
already been discussed or if anyone would envision other possible use cases
for
this notation.

Best place to go from here would be a preparser, which you can then
publish. If, at some later date, someone else has a similar need, s/he
can see what you did and either (1) use it as-is and utter a prayer of
thanks that someone's done the work already; (2) tweak it to fit the
exact situation required; or (3) grumble at your code, and come back
to python-ideas with a proposal.

The proposal from #3 would sound something like this: "Here's my
use-case. There's this recipe on the internet <link> but it's awkward
because X and Y, and it'd be so much better if this could be supported
by the core language." And then we'd have this long and fruitful
discussion (Sir Humphrey Appleby would approve!), figuring out whether
it's of value or not, all with the solid basis of two separate
use-cases for the same new syntax.

ChrisA

[Andrew Barnert]

Post by Stefano Borini

Post by Chris Angelico
The obvious way to accept that would be to support keyword arguments,
and then it begins looking very much like a call. Can you alter your
notation very slightly to become LDA(Z=5) instead?

We certainly can, but I was wondering if such extension would be useful in other contexts.
Also, with the function solution, you would lose the order of the entries. You can't distinguish
foo(z=3, r=4) from foo(r=4, z=3)

That last problem is a more general one, which applies to function calls at least as much as to your proposed use case, and there's an open PEP (466) that could probably use more use cases to convince people. With that PEP, you wouldn't get {'z': 3}, {'r': 4}, but OrderedDict(('z', 3), ('r', 4)) or something equivalent. I think that would make the function-calling workaround much more usable. And it would definitely make your additional proposal a lot simpler: add kwargs--which then work exactly the same as in function calls--to getitem.

There's also a proposal for namedtuple literals, which seems like it fit your use case a lot better (especially if, like a regular tuple literal, the parens were optional). Unfortunately, if I remember right, nobody was able to come up with a good enough solution to the semantic problems to make it worth writing a PEP. But you could find that in the archives and see if you can come up with a workable version of that idea.

[Stefano Borini]

This post might be inappropriate. Click to display it.

[Eric V. Smith]

Post by Stefano Borini

Post by Andrew Barnert
That last problem is a more general one, which applies to function
calls at least as much as to your proposed use case, and there's an
open PEP (466) that could probably use more use cases to convince
people.

Sorry, I cannot find it. PEP 466 is about network security. It's the
first time I engage in active python proposals, so I might be a bit
clueless if for python3 there's a different repository/numbering system.

That's PEP 468: http://legacy.python.org/dev/peps/pep-0468/

[Terry Reedy]

Post by Stefano Borini
Dear all,
At work we use a notation like LDA[Z=5] to define a specific level
of accuracy for our evaluation. This notation is used
just for textual labels, but it would be nice if it actually worked
at the moment, we have the following

This actually says that y can be passed by position or name ;-)

Post by Stefano Borini
... print(y)
...

a=A()
a[2]

2

a.__getitem__(y=2)

2

Post by Stefano Borini

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}. In the case of a[1:3, 4, Z=3, R=5], the value of y would
be a tuple containing (slice(1,3,None), 4, {"Z": 3}, {"R": 5}). This allows to
preserve the ordering as specified (e.g. a[Z=3, R=4] vs a[R=4, Z=3]).

As others have pointed out, you are not actually asking that __getitem__
'accept keyword arguments'. Rather you are asking that "x=y" be seen as
an abbreviation for "{'x':y}" in a very rare usage in a particular
context to save 4 (admittedly awkward) keystrokes. The resulting
confusion is not worth it. Saving 4 of 7 might seem worth it, but it
real cases, like "precision=4" versus "{'precision':4}" the ratio is
lower. I also wonder whether you might sometimes us the same spec in
multiple subscriptings, so that you might define "p = {'precision': 4}"
once and use it multiple times.

In your introductory paragraph, you only specify one optional parameter
-- accuracy. So it is not clear why you do not just write a .get(self,
ob, accuaracy=default) method. If their are multiple options, make them
keyword only.

[Paul Moore]

Post by Terry Reedy
As others have pointed out, you are not actually asking that __getitem__
'accept keyword arguments'. Rather you are asking that "x=y" be seen as an
abbreviation for "{'x':y}" in a very rare usage in a particular context to
save 4 (admittedly awkward) keystrokes.

The point here is that the OP is viewing Python syntax as (in effect)
a DSL[1] for his application, and is looking for syntactical support
for constructs that make sense in the context of that DSL. It's not
about saving keystrokes, but about expressing things in a way that
matches the problem space. The problem is that Python doesn't really
support use as a DSL (as opposed to, say Ruby and Perl, which have
syntax that is explicitly designed for use as a DSL).

Trying to add on DSL-style syntax into Python is always going to be
difficult, because that's not how the language was designed. On the
other hand, writing a parser or preprocessor that handles a specific
DSL is entirely possible - just painful because you need to handle all
the niggly details of expression parsers, etc.

Maybe a better approach would be to add features to the Python parser
to allow it to be used in 3rd party code and customised. Applications
could then more easily write their own Python-derived syntax, with a
parser that can read from a string, or even implement an import hook
to allow directly importable DSL files. I don't know how practical
this solution is, or how much of it is already available, but it might
be a more productive way of directing people who are looking for
"python-like" syntax for their application languages, rather than
simply leaving them with writing their own parser, or trying to get
Python's syntax changed (which is essentially not going to happen).

Just a thought...

Paul

[1] Domain Specific Language, just in case the term isn't familiar.

[Joseph Martinot-Lagarde]

Post by Stefano Borini
Dear all,
At work we use a notation like LDA[Z=5] to define a specific level of accuracy for our evaluation. This notation is used
at the moment, we have the following
... print(y)
...

a=A()
a[2]

2

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}. In the case of a[1:3, 4, Z=3, R=5], the value of y would
be a tuple containing (slice(1,3,None), 4, {"Z": 3}, {"R": 5}). This allows to
preserve the ordering as specified (e.g. a[Z=3, R=4] vs a[R=4, Z=3]).
Do you think it would be a good/useful idea? Was this already discussed or proposed in a PEP?
Google did not help on this regard.
Thank you,
Stefano Borini
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Actually I proposed a similar functionnality a few months ago:
http://thread.gmane.org/gmane.comp.python.ideas/27584

The main point is not saving a few keystrokes but increase readability.
It is indeed possible to use __call__ (that's what I'm doing in some
cases), but then the indexing part is lost. Using a dictionnary is not
clear either. Compare:

table[x=8, y=11]
table[{x: 8}, {y: 11}]

You could argue that keyword arguments are useless since you can always
add a dictionary as last argument...

Before using python I was using Matlab. One very annoying thing in
Matlab is that both indexing and function call use parenthesis. Code
mixing both is really hard to understand. Coming to python was a relief
on this aspect, where [] and () makes really clear whether the operation
is a call or indexing.

Now that I know python better, it bothers me that indexing doesn't have
the same semantics a a function call. To me their intentions are
different but their use should be the same. I guess that the equivalence
between a[1, 2] and a[(1, 2)] is for backward compatibility, but it
shouldn't stop from adding keywords arguments.

Using a preprocessor seems fine when building a full application, but is
really impracticable when crunching numbers from scripts or ipython.
Also, using a preprocessor for something as simple as indexing seems
really overkill.

Now, I don't understand why you need to know the ordering of keyword
arguments, since they are clearly labeled ? I'd hate to have to manually
parse (slice(1,3,None), 4, {"Z": 3}, {"R": 5}).

Joseph

[Joseph Martinot-Lagarde]

Post by Joseph Martinot-Lagarde

Post by Stefano Borini
Dear all,
At work we use a notation like LDA[Z=5] to define a specific level of
accuracy for our evaluation. This notation is used
just for textual labels, but it would be nice if it actually worked at
at the moment, we have the following
... print(y)
...

a=A()
a[2]

2

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}. In the case of a[1:3, 4, Z=3, R=5], the value of y would
be a tuple containing (slice(1,3,None), 4, {"Z": 3}, {"R": 5}). This allows to
preserve the ordering as specified (e.g. a[Z=3, R=4] vs a[R=4, Z=3]).
Do you think it would be a good/useful idea? Was this already
discussed or proposed in a PEP?
Google did not help on this regard.
Thank you,
Stefano Borini
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

http://thread.gmane.org/gmane.comp.python.ideas/27584
The main point is not saving a few keystrokes but increase readability.
It is indeed possible to use __call__ (that's what I'm doing in some
cases), but then the indexing part is lost. Using a dictionnary is not
table[x=8, y=11]
table[{x: 8}, {y: 11}]
You could argue that keyword arguments are useless since you can always
add a dictionary as last argument...
Before using python I was using Matlab. One very annoying thing in
Matlab is that both indexing and function call use parenthesis. Code
mixing both is really hard to understand. Coming to python was a relief
on this aspect, where [] and () makes really clear whether the operation
is a call or indexing.
Now that I know python better, it bothers me that indexing doesn't have
the same semantics a a function call. To me their intentions are
different but their use should be the same. I guess that the equivalence
between a[1, 2] and a[(1, 2)] is for backward compatibility, but it
shouldn't stop from adding keywords arguments.
Using a preprocessor seems fine when building a full application, but is
really impracticable when crunching numbers from scripts or ipython.
Also, using a preprocessor for something as simple as indexing seems
really overkill.
Now, I don't understand why you need to know the ordering of keyword
arguments, since they are clearly labeled ? I'd hate to have to manually
parse (slice(1,3,None), 4, {"Z": 3}, {"R": 5}).
Joseph
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

I forgot to add that slice notation can't be used in function calls, you
have to use the very less readable slice() function.

For a live example of something that would use keyword arguments in
__getitem__, there is numpy.r_ :
http://docs.scipy.org/doc/numpy/reference/generated/numpy.r_.html.

Joseph

[Devin Jeanpierre]

What about using slices instead?

Post by Stefano Borini

a['Z': 3, 'B': 2]

(slice('Z', 3, None), slice('B', 2, None))

-- Devin

On Mon, Jun 23, 2014 at 5:06 AM, Stefano Borini

Post by Stefano Borini
Dear all,
At work we use a notation like LDA[Z=5] to define a specific level of accuracy for our evaluation. This notation is used
at the moment, we have the following
... print(y)
...

a=A()
a[2]

2

a[2,3]

(2, 3)

a[1:3]

slice(1, 3, None)

a[1:3, 4]

(slice(1, 3, None), 4)
I would propose to add the possibility for a[Z=3], where y would then be a
dictionary {"Z": 3}. In the case of a[1:3, 4, Z=3, R=5], the value of y would
be a tuple containing (slice(1,3,None), 4, {"Z": 3}, {"R": 5}). This allows to
preserve the ordering as specified (e.g. a[Z=3, R=4] vs a[R=4, Z=3]).
Do you think it would be a good/useful idea? Was this already discussed or proposed in a PEP?
Google did not help on this regard.
Thank you,
Stefano Borini
_______________________________________________
Python-ideas mailing list
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Guido van Rossum]

I'm not sure yet what to think of the proposal (the proposed workarounds
sound pretty reasonable) but it looks to me like the OP (Stefano) did a
pretty good and careful analysis of the existing API, and his actual
proposal does make the most sense if we wanted to add such a feature at
all. (And yes, the subject was a little misleading. :-)