presto-0.147+postgresql-9.5.3+msql-5.0.7+hadoop-2.5.2+hive-1.2.1环境构筑以及测试

概述背景 每个支持SQL的数据库，都有一个强大的SQL引擎。 而对于SQL引擎，基本都是大同小异的，负责SQL文法解析，语意分析，指定查询树，优化查询树，再到最终的执行，客户端返回结果。 而presto的也跟一般的是一样的。 架构如下： 准备 1.postgresql-9.5.3 2.mysql-5.0.7 3.hadoop-2.5.2 4.hive-1.2.1 5.presto-server-0.1 背景

每个支持sql的数据库，都有一个强大的sql引擎。

而对于sql引擎，基本都是大同小异的，负责sql文法解析，语意分析，指定查询树，优化查询树，再到最终的执行，客户端返回结果。

而presto的也跟一般的是一样的。

架构如下：

准备

1.postgresql-9.5.3

2.MysqL-5.0.7

3.hadoop-2.5.2

4.hive-1.2.1

5.presto-server-0.147

6.presto-cli-0.147-executable.jar

且注意系统要求：

Mac OS X or linuxJava 8 Update 60 or higher (8u60+),64-bitMaven 3.3.9+ (for building)Python 2.4+ (for running with the launcher script)

环境搭建_1

MysqL,postgresql都是在windows这边搭建的，直接就可以使用。

hadoop-2.5.2的搭建手顺之前的博文中已经记载了，此处不再说明。

hive的环境，解压后就可以使用了。

这里主要说一下hive的两种Cli工具：

1.hive shell

2.beeline

现在官网标记beeline是new，hive shell是older。建议使用beeline，结果的现实比较直观易懂。跟MysqL的比较像。

beeline的启动，如果hive使用默认的debery数据库的话，请使用下面的方式启动

./bin/beeline -u jdbc:hive2://

另外，derby只能同时一个用户使用，否则会报错如下所示：

Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/myProject/apache-hive-1.2.1-bin/metastore_db.        at org.apache.derby.iAPI.error.StandardException.newException(UnkNown Source)        at org.apache.derby.impl.store.raw.data.BaseDatafileFactory.privGetJBMSLockOnDB(UnkNown Source)        at org.apache.derby.impl.store.raw.data.BaseDatafileFactory.run(UnkNown Source)        at java.security.AccessController.doPrivileged(Native Method)        at org.apache.derby.impl.store.raw.data.BaseDatafileFactory.getJBMSLockOnDB(UnkNown Source)        at org.apache.derby.impl.store.raw.data.BaseDatafileFactory.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.topService.bootModule(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(UnkNown Source)        at org.apache.derby.iAPI.services.monitor.Monitor.bootServiceModule(UnkNown Source)        at org.apache.derby.impl.store.raw.RawStore.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.topService.bootModule(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(UnkNown Source)        at org.apache.derby.iAPI.services.monitor.Monitor.bootServiceModule(UnkNown Source)        at org.apache.derby.impl.store.access.RAMAccessManager.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.topService.bootModule(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(UnkNown Source)        at org.apache.derby.iAPI.services.monitor.Monitor.bootServiceModule(UnkNown Source)        at org.apache.derby.impl.db.Basicdatabase.bootStore(UnkNown Source)        at org.apache.derby.impl.db.Basicdatabase.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(UnkNown Source)        at org.apache.derby.impl.services.monitor.topService.bootModule(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.startProvIDerService(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.findProvIDerAndStartService(UnkNown Source)        at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(UnkNown Source)        at org.apache.derby.iAPI.services.monitor.Monitor.startPersistentService(UnkNown Source)        ... 83 moreError applying authorization policy on hive configuration: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.Metadata.SessionHivemetastoreClIEnt

处理方法也很简单：

rm -rf derby.log metastore_db

环境搭建_2 presto 解压presto-server-0.147.tar.gz mkdir presto-server-0.147/etc mkdir presto-server-0.147/catalog vim etc/node.propertIEs

node.environment=productionnode.ID=1node.data-dir=/home/myProject/presto-server-0.147/data

etc/jvm.config

-server-Xmx16G-XX:+UseG1GC-XX:G1HeapRegionSize=32M-XX:+UseGCOverheadlimit-XX:+ExplicitGCInvokesConcurrent-XX:+HeapDumpOnOutOfMemoryError-XX:OnOutOfMemoryError=kill -9 %p

etc/config.propertIEs

coordinator=truenode-scheduler.include-coordinator=truehttp-server.http.port=8080query.max-memory=5GBquery.max-memory-per-node=1GBdiscovery-server.enabled=truediscovery.uri=http://your-hive-IP:8080

etc/log.propertIEs

com.facebook.presto=INFO

下载presto-cli-0.147-executable.jar 将其放置在bin目录下 赋权限

chmod +x presto-cli-0.147-executable.jar

etc/catalog/MysqL.propertIEs

connector.name=MysqLconnection-url=jdbc:MysqL://your-MysqL-location-IP:3306connection-user=your-MysqL-usernameconnection-password=your-MysqL-password

etc/catalog/postgresql.propertIEs

connector.name=postgresqlconnection-url=jdbc:postgresql://your-postgresql-location-ip/postgresconnection-user=your-postgres-usernameconnection-password=your-postgresql-password

etc/catalog/hive.propertIEs

connector.name=hive-hadoop2hive.metastore.uri=thrift://your-hive-ip:9083hive.config.resources=/etc/hadoop/core-site.xml,/etc/hadoop/hdfs-site.xml

connector.name的选取参照如下信息

hive-hadoop1: Apache Hadoop 1.xhive-hadoop2: Apache Hadoop 2.xhive-cdh4: Cloudera CDH 4hive-cdh5: Cloudera CDH 5

启动 bin/launcher start ./presto --server localhost:8080--cataloghive --schema default

结果

MysqL

presto:test_hive> select * from MysqL.sqoop.t1; ID | int_col | char_col ----+---------+----------  1 |       1 | a          2 |       2 | b          4 |       4 | d          3 |       3 | c          5 |       5 | e        (5 rows)query 20160520_101400_00009_k46dt,FINISHED,1 nodehttp://localhost:8080/query.HTML?20160520_101400_00009_k46dtSplits: 2 total,0 done (0.00%)cpu Time: 0.0s total,0 rows/s,0B/s,100% activePer Node: 0.0 parallelism,0B/sParallelism: 0.00:29 [0 rows,0B] [0 rows/s,0B/s]

postgresql

presto:test_hive> select * from postgresql.public.test; ID | name ----+------  1 | lily   2 | Tom    3 | Jim  (3 rows)query 20160520_101503_00010_k46dt,1 nodehttp://localhost:8080/query.HTML?20160520_101503_00010_k46dtSplits: 2 total,0% activePer Node: 0.0 parallelism,0B/sParallelism: 0.00:02 [0 rows,0B/s]

MysqL&postgresql

presto:test_hive> select ID,char_col from MysqL.sqoop.t1 union select ID,name from postgresql.public.test; ID | char_col ----+----------  1 | lily       2 | Tom        3 | Jim        1 | a          2 | b          4 | d          3 | c          5 | e        (8 rows)query 20160520_101532_00011_k46dt,1 nodehttp://localhost:8080/query.HTML?20160520_101532_00011_k46dtSplits: 6 total,2 done (33.33%)cpu Time: 0.0s total,107 rows/s,17% activePer Node: 0.0 parallelism,0B/sParallelism: 0.00:28 [3 rows,0B/s]

hive

--未完待续---

总结

以上是内存溢出为你收集整理的presto-0.147+postgresql-9.5.3+msql-5.0.7+hadoop-2.5.2+hive-1.2.1环境构筑以及测试全部内容，希望文章能够帮你解决presto-0.147+postgresql-9.5.3+msql-5.0.7+hadoop-2.5.2+hive-1.2.1环境构筑以及测试所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。