Some interesting behavior from .groupby().apply( .to_records() )

[alexlenail]

Why is it that

df.groupby('A')[['B','C']].apply(lambda x: x.to_records())

throws: TypeError: Series.name must be a hashable type while

df.groupby('A').apply(lambda x: x[['B','C']].to_records())

works?

[jreback]

pls show a copy pastable example

[alexlenail]

df = pd.DataFrame({'A':{'a':1,'b':2},'B':{'a':1,'b':2},'C':{'a':1,'b':2}})
df.groupby('A')[['B','C']].apply(lambda x: x.to_records())  # => TypeError
df.groupby('A').apply(lambda x: x[['B','C']].to_records())  # => Succeeds

[bobhaffner]

I had some time this morning so I looked at this. Here's what I found out.

This looks like the line where things head south.

pandas/pandas/core/series.py

Line 248 in ee9c7e9

self.name = name

The successful line below has name = None
df.groupby('A').apply(lambda x: x[['B','C']].to_records()) # => Succeeds

and the line that errors has a name = ['B', 'C']
df.groupby('A')[['B','C']].apply(lambda x: x.to_records()) # => TypeError

ps. swapping out to_records for something similar like to_dict does work
df.groupby('A')['B','C'].apply(lambda x: x.to_dict())

[jreback]

hmm, it maybe that the rec-array is not handled correctly.

[bobhaffner]

Here's what I'm thinking

Modify the NDFrameGroupBy class method called _wrap_applied_output to deal with recarrays

Maybe something like the following which fixes this issue and all current tests in test_groupby.py still pass.

Any feedback?

git diff ..upstream/master -- pandas/core/groupby.py 

diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index 6a2f49a89..041239ed0 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -3977,8 +3977,6 @@ class NDFrameGroupBy(GroupBy):
                         return self._concat_objects(
                             keys, values, not_indexed_same=True,
                         )
-                if isinstance(v, np.recarray):
-                    return Series(values, index=key_index)
 
                 try:
                     if self.axis == 0:

[jreback]

NO. don't specifically deal with recarrays, you can possibly handle an array separaly though (recarray is a sub-class).

[bobhaffner]

Hi Jeff, thanks for the quick reply. Let me attempt to add some context. I was trying to follow the existing logic found here

if isinstance(v, (np.ndarray, Index, Series)):
    if isinstance(v, Series):
        ...
    if isinstance(v, np.recarray):  #proposed addition
        return Series(values, index=key_index)                

[bobhaffner]

Hi @zfrenchee probably above mine too :-)

but yes, it's typical to add tests to cover additions. Thanks for submitting this by the way.

[yarimiz]

Hey, are there any news regarding this issue? Currently, I'm facing is with this code
df.groupby('type')[['c1', 'c2']].apply(lambda x: ((x.c1.shift(1) - x.c2) / x.c1))

that is raising
TypeError: Series.name must be a hashable type

I haven't found a solution for that.

@jreback

[damon-chem-e]

I'm having the same issue
small['ret'] = small.groupby('isin')[['closingprice_eur','price_lag']].apply(lambda g: (g.closingprice_eur-g.price_lag)/g.price_lag)

raises
TypeError: Series.name must be a hashable type

[javiqm12]

Having same issue here :( with pandas 1.3.5