def __init__(self, documents=None): self.token2id = {} # token -> tokenId self.id2token = {} # reverse mapping for token2id; only formed on request, to save memory self.dfs = {} # document frequencies: tokenId -> in how many documents this token appeared self.num_docs = 0 # number of documents processed self.num_pos = 0 # total number of corpus positions self.num_nnz = 0 # total number of non-zeroes in the BOW matrix if documents is not None: self.add_documents(documents)
函数add_documents从文档集合构建字典。每个文档都是令牌列表:
def add_documents(self, documents): for docno, document in enumerate(documents): if docno % 10000 == 0: logger.info("adding document #%i to %s" % (docno, self)) _ = self.doc2bow(document, allow_update=True) # ignore the result, here we only care about updating token ids logger.info("built %s from %i documents (total %i corpus positions)" % (self, self.num_docs, self.num_pos))
因此,如果以此方式初始化Dictionary,则必须传递文档,但不能传递单个文档。例如,
dic = corpora.Dictionary([a.split()])
还可以
欢迎分享,转载请注明来源:内存溢出
评论列表(0条)