Gensim:TypeError:doc2bow期望输入的是Unicode标记数组,而不是单个字符串

Gensim:TypeError:doc2bow期望输入的是Unicode标记数组,而不是单个字符串,第1张

Gensim:TypeError:doc2bow期望输入的是Unicode标记数组,而不是单个字符串

在dictionary.py中,初始化函数为:

def __init__(self, documents=None):    self.token2id = {} # token -> tokenId    self.id2token = {} # reverse mapping for token2id; only formed on request, to save memory    self.dfs = {} # document frequencies: tokenId -> in how many documents this token appeared    self.num_docs = 0 # number of documents processed    self.num_pos = 0 # total number of corpus positions    self.num_nnz = 0 # total number of non-zeroes in the BOW matrix    if documents is not None:        self.add_documents(documents)

函数add_documents从文档集合构建字典。每个文档都是令牌列表:

def add_documents(self, documents):    for docno, document in enumerate(documents):        if docno % 10000 == 0: logger.info("adding document #%i to %s" % (docno, self))        _ = self.doc2bow(document, allow_update=True) # ignore the result, here we only care about updating token ids    logger.info("built %s from %i documents (total %i corpus positions)" %      (self, self.num_docs, self.num_pos))

因此,如果以此方式初始化Dictionary,则必须传递文档,但不能传递单个文档。例如,

dic = corpora.Dictionary([a.split()])

还可以



欢迎分享,转载请注明来源:内存溢出

原文地址: https://outofmemory.cn/zaji/5645551.html

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2022-12-16
下一篇 2022-12-16

发表评论

登录后才能评论

评论列表(0条)

保存