使用solr的DIHandler 构建mysql大表全量索引，内存溢出问题的解决方法

概述solr官方给出的解决方式是：DataImportHandler is designed to stream row one-by-one. It passes a fetch size value (default: 500) to Statement#setFetchSize which some drivers do not honor. For MySQL, add batchSize property to dataSource configuration with value -1. This will pass Integer.MIN_VALUE to the driver as the fetch size and keep it from going out of memory for large tables.Should look like:<dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:8889/mysqldatabase" batchSize="-1" user="root" password="root"/>说明：DataImportHandler 设计是支持按行获取的。它通过Statement#setFetchSize来设置每次获取的数量，默认是500个。然而一些驱动不支持设置fetchSize。对mysql来说，传递fetchSize属性值-1到Datasource配置中。它将将Integer.MIN_VALUE（-231，-2147483648 [0x80000000]）传给驱动作为fetchsize，此时确保大表不会造成大表移除。mysql官方给出的解释是：ResultSetBy default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. If you are working with ResultSets that have a large number of rows or large values and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.To enable this functionality, create a Statement instance in the following manner:stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);stmt.setFetchSize(Integer.MIN_VALUE);The combination of a forward-only, read-only result set, with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this, any result sets created with the statement will be retrieved row-by-row.There are some caveats with this approach. You must read all of the rows in the result set (or close it) before you can issue any other queries on the connection, or an exception will be thrown.The earliest the locks these statements hold can be released (whether they be MyISAM table-level locks or row-level locks in some other storage engine such as InnoDB) is when the statement completes.If the statement is within scope of a transaction, then locks are released when the transaction completes (which implies that the statement needs to complete first). As with most other databases, statements are not complete until all the results pending on the statement are read or the active result set for the statement is closed.Therefore, if using streaming results, process them as quickly as possible if you want to maintain concurrent access to the tables referenced by the statement producing the result set.通过联合使用forward-only,read-only resultSet和fetchsize值为Integer.MIN_VALUE作为驱动一行行获取结果流的信号。设置完以后，所有statement创建的resultSet将会一行行的获取结果集。参考文献：【1】 https://wiki.apache.org/solr/DataImportHandlerFaq【2】http://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html

solr官方给出的解决方式是：

dataimporthandler is designed to stream row one-by-one. It passes a fetch size value (default: 500) to Statement#setFetchSize which some drivers do not honor. For MysqL,add batchSize property to dataSource configuration with value -1Should look like:

<dataSource type="JdbcdataSource" name="ds-2" driver="com.MysqL.jdbc.Driver" url="jdbc:MysqL://localhost:8889/MysqLdatabase" batchSize="-1" user="root" password="root"/>

说明：dataimporthandler 设计是支持按行获取的。它通过Statement#setFetchSize来设置每次获取的数量，默认是500个。然而一些驱动不支持设置fetchSize。对MysqL来说，传递fetchSize属性值-1到Datasource配置中。它将将Integer.MIN_VALUE（-2³¹，-2147483648 [0x80000000]）传给驱动作为fetchsize，此时确保大表不会造成大表移除。

MysqL官方给出的解释是：

By default,ResultSets are completely retrIEved and stored in memory. In most cases this is the most efficIEnt way to operate and,due to the design of the MysqL network protocol,is easIEr to implement. If you are working with ResultSets that have a large number of rows or large values and cannot allocate heap space in your JVM for the memory required,<span > you can tell the driver to stream the results back one row at a time.

To enable this functionality,<span > create a Statement instance in the following manner:

<span >stmt <span >= conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
<span >The combination of a forward-only,read-only result set,with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this,<span ><span > any result sets created with the statement will be retrIEved row-by-row.

There are some caveats with this approach. You must read all of the rows in the result set (or close it) before you can issue any other querIEs on the connection,<span > or an exception will be thrown.

The earlIEst the locks these statements hold can be released (whether they be MyISAM table-level locks or row-level locks in some other storage engine such as InnoDB) is when the statement completes.

If the statement is within scope of a transaction,then locks are released when the transaction completes (which implIEs that the statement needs to complete first). As with most other databases,<span > statements are not complete until all the results pending on the statement are read or the active result set for the statement is closed.

Therefore,if using streaming results,process them as quickly as possible if you want to maintain concurrent access to the tables referenced by the statement producing the result set.

通过联合使用forward-only,read-only resultSet和fetchsize值为Integer.MIN_VALUE作为驱动一行行获取结果流的信号。设置完以后，所有statement创建的resultSet将会一行行的获取结果集。

参考文献：

【1】 https://wiki.apache.org/solr/dataimporthandlerFaq

【2】http://dev.MysqL.com/doc/connector-j/en/connector-j-reference-implementation-notes.HTML

总结

以上是内存溢出为你收集整理的使用solr的DIHandler 构建mysql大表全量索引，内存溢出问题的解决方法全部内容，希望文章能够帮你解决使用solr的DIHandler 构建mysql大表全量索引，内存溢出问题的解决方法所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。

欢迎分享，转载请注明来源：内存溢出

原文地址: https://outofmemory.cn/sjk/1169167.html