解压sqoop安装包
[root@master ~]# tar zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local/src/
重命名sqoop文件夹
[root@master ~]# mv /usr/local/src/sqoop-1.4.7.bin__hadoop-2.6.0/ /usr/local/src/sqoop2、配置sqoop环境
创建sqoop的配置文件sqoop-env.sh
复制sqoop-env-template.sh模板,并将模板重命名为sqoop-env.sh
[root@master conf]# cd /usr/local/src/sqoop/conf/ [root@master conf]# cp sqoop-env-template.sh sqoop-env.sh
修改sqoop-env.sh文件,添加Hadoop,Hbase,hive等组件的安装路径
[root@master conf]# vim sqoop-env.sh export HADOOP_COMMON_HOME=/usr/local/src/hadoop export HADOOP_MAPRED_HOME=/usr/local/src/hadoop export Hbase_HOME=/usr/local/src/hbase export HIVE_HOME=/usr/local/src/hive
配置sqoop的环境变量
[root@master conf]# vim /etc/profile export SQOOP_HOME=/usr/local/src/sqoop export PATH=$PATH:$SQOOP_HOME/bin export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib [root@master conf]# source /etc/profile
把mysql驱动复制到sqoop的lib目录下
[root@master ~]# cp mysql-connector-java-5.1.47.jar /usr/local/src/sqoop/lib/3.启动sqoop
启动Hadoop集群
[root@master ~]# su - hadoop [hadoop@master ~]$ start-all.sh [hadoop@master ~]$ jps 1286 NameNode 1897 Jps 1482 SecondaryNameNode 1630 ResourceManager
测试sqoop是否能够正常链接mysql
[hadoop@master ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306/ --username root -P Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 21/10/23 15:26:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 Enter password: 21/10/23 15:26:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. Sat Oct 23 15:26:09 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. information_schema hive mysql performance_schema sys
将hive驱动放到sqoop的lib目录下
[hadoop@master ~]$ cp /usr/local/src/hive/lib/hive-common-2.0.0.jar /usr/local/src/sqoop/lib/
利用sqoop将MySQL数据库中的表导入到hive中
sqoop import --connect jdbc:mysql://master:3306/sample --username root --password Password123$ --table book --fields-terminated-by '|' --delete-target-dir --num-mappers 1 --hive-import --hive-database sample --hive-table book
利用sqoop将MySQL数据库中的表导入到hdfs中
sqoop import --connect jdbc:mysql://master:3306/School --username root --password Password123$ --table student -m 1 -target-dir /user/hadoop/hive/exam
利用sqoop将MySQL数据库中的表新增的数据增量导入到hdfs中
sqoop import -connect jdbc:mysql://master:3306/School -username root -password Password123$ -table student -m 1 -target-dir /user/hadoop/hive/exam -check-column id -incremental append -last-value 3
欢迎分享,转载请注明来源:内存溢出
评论列表(0条)