如何使用pyarrow将Pandas数据帧设置获取到Redis中

如何使用pyarrow将Pandas数据帧设置/获取到Redis中

这是一个使用pyarrow序列化熊猫数据帧以存储在Redis中的完整示例

apt-get install python3 python3-pip redis-serverpip3 install pandas pyarrow redis

然后在python中

import pandas as pdimport pyarrow as paimport redisdf=pd.Dataframe({'A':[1,2,3]})r = redis.Redis(host='localhost', port=6379, db=0)context = pa.default_serialization_context()r.set("key", context.serialize(df).to_buffer().to_pybytes())context.deserialize(r.get("key"))   A0  11  22  3

我刚刚向熊猫提交了PR 28494，以便在文档中包含这个pyarrow示例。

参考文档：

• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Dataframe.to_msgpack.html
• https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization
• https://arrow.apache.org/docs/python/memory.html#pyarrow-buffer
• https://stackoverflow.com/a/37957490/4126114