HDP 3.1.0 集成 Sqoop2

摘要:
HDP3.1.0集成Sqoop2原始地址:https://sitoi.cn/posts/65261.html环境由三个主机组成。HDP3.1.0集群配置有时间同步步骤。下载Sqoop2的安装包,解压缩安装包,然后修改sqoop。/usr/lib目录中的sh环境变量属性配置导入第三方jar包配置第三方jar包参考路径修改Ambari上的组件配置验证配置是否正确启用以从服务器下载Sq
HDP 3.1.0 集成 Sqoop2

本文原始地址:https://sitoi.cn/posts/65261.html

环境

  • 由三台主机组成的 HDP 3.1.0 集群
  • 配置好时间同步

步骤

  • 下载 Sqoop2 的安装包
  • 解压安装包到 /usr/lib 目录下
  • 修改 sqoop.sh 环境变量
  • 修改 sqoop.properties 配置
  • 导入第三方 jar
  • 配置第三方 jar 包引用路径
  • 修改 Ambari 上组件配置
  • 验证配置是否正确
  • 开启服务器

下载 Sqoop2 的安装包

下载地址:http://mirror.bit.edu.cn/apache/sqoop/1.99.7/

下载命令

cd ~
wget http://mirror.bit.edu.cn/apache/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz

解压安装包到 /usr/lib 目录下

解压 Sqoop2 压缩包

tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz

移动到 /usr/lib/sqoop 目录

mv sqoop-<version>-bin-hadoop<hadoop version> /usr/lib/sqoop

修改 sqoop.sh 环境变量

编辑 /usr/lib/sqoop/bin/sqoop.sh 文件

sudo vim /usr/lib/sqoop/bin/sqoop.sh

找到 function sqoop_server_classpath_set 函数,将其中的环境变量改一下就可以了,如下:

function sqoop_server_classpath_set {
                                                                                             
  HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-${HADOOP_HOME}/share/hadoop/common}
  HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-${HADOOP_HOME}/share/hadoop/hdfs}
  HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-${HADOOP_HOME}/share/hadoop/mapreduce}
  HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-${HADOOP_HOME}/share/hadoop/yarn}

将这些环境变量都注释掉,改为下面的内容即可:

function sqoop_server_classpath_set {

  HDP=/usr/hdp/3.0.1.0-187
  HADOOP_COMMON_HOME=$HDP/hadoop
  HADOOP_HDFS_HOME=$HDP/hadoop-hdfs
  HADOOP_MAPRED_HOME=$HDP/hadoop-mapreduce
  HADOOP_YARN_HOME=$HDP/hadoop-yarn

修改 sqoop.properties 配置

修改 sqoop.properties

sudo vim /usr/lib/sqoop/conf/sqoop.properties

找到 org.apache.sqoop.submission.engine.mapreduce.configuration.directory 参数,如下:

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/

根据集群实际信息将其改为下面的内容即可:

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hdp/3.1.0.0-78/hadoop/conf/

导入第三方 jar 包

mkdir /usr/lib/sqoop/extra
cp /var/lib/ambari-server/resources/mysql-jdbc-driver.jar /usr/lib/sqoop/extra/
cp  -r /usr/lib/sqoop/extra/* /usr/lib/sqoop/server/lib/
cp  -r /usr/lib/sqoop/extra/* /usr/lib/sqoop/shell/lib/
cp  -r /usr/lib/sqoop/extra/* /usr/lib/sqoop/tools/lib/

配置第三方 jar 包引用路径

sudo vim ~/.bashrc

添加环境变量,如下:

export SQOOP_HOME=/usr/lib/sqoop
export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra
export PATH=$PATH:$SQOOP_HOME/bin

运行如下命令,使环境变量生效:

source ~/.bashrc 

修改 Ambari 上组件配置

修改组件 HDFS 配置

配置项参数名初始值修改值
Advanced hdfs-sitedfs.permissions.enabledTrueFalse
Custom core-sitehadoop.proxyuser.hive.hosts*
Custom core-sitehadoop.proxyuser.root.hosts*
Custom core-sitehadoop.proxyuser.sqoop2.groups*
Custom core-sitehadoop.proxyuser.sqoop2.hosts*
Custom core-sitehadoop.proxyuser.yarn.groups*
Custom core-sitehadoop.proxyuser.yarn.hosts*

修改组件 MapRduce2 配置

${hdp.version} 替换成实际 hdp 的版本: 3.1.0.0-78

配置项参数名初始值修改值
Advanced mapred-sitemapreduce.admin.map.child.java.opts-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.0.0-78
Advanced mapred-sitemapreduce.admin.reduce.child.java.opts-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.0.0-78
Advanced mapred-sitemapreduce.admin.user.envLD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-{{architecture}}-64LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native:/usr/hdp/3.1.0.0-78/hadoop/lib/native/Linux-{{architecture}}-64
Advanced mapred-sitemapreduce.application.classpath$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/3.1.0.0-78/hadoop/lib/hadoop-lzo-0.6.0.3.1.0.0-78.jar:/etc/hadoop/conf/secure
Advanced mapred-sitemapreduce.application.framework.path/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework/hdp/apps/3.1.0.0-78/mapreduce/mapreduce.tar.gz#mr-framework
Advanced mapred-siteyarn.app.mapreduce.am.admin-command-opts-Dhdp.version=${hdp.version}-Dhdp.version=3.1.0.0-78
Advanced mapred-siteMR AppMaster Java Heap Size-Xmx819m -Dhdp.version=${hdp.version}-Xmx819m -Dhdp.version=3.1.0.0-78

验证配置是否正确

$ sqoop2-tool verify

Setting conf dir: /usr/lib/sqoop/bin/../conf                                                 
Sqoop home directory: /usr/lib/sqoop                                                         
Sqoop tool executor:                                                                         
        Version: 1.99.7                                                                      
        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb                                   
        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine                                  
Running tool: class org.apache.sqoop.tools.tool.VerifyTool                                   
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.            
8    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config fi
le poller thread                                                                             
Verification was successful.                                                                 
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.  

开启服务器

$ sqoop2-server start

Setting conf dir: /usr/lib/sqoop/bin/../conf                                                 
Sqoop home directory: /usr/lib/sqoop                                                         
Sqoop tool executor:                                                                         
        Version: 1.99.7                                                                      
        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb                                   
        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine                                  
Running tool: class org.apache.sqoop.tools.tool.VerifyTool                                   
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.            
8    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config fi
le poller thread                                                                             
Verification was successful.                                                                 
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.                    
[root@sandbox-hdp ~]# sqoop2-server start                                                    
Setting conf dir: /usr/lib/sqoop/bin/../conf                                                 
Sqoop home directory: /usr/lib/sqoop                                                         
Starting the Sqoop2 server...                                                                
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.            
11   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config fi
le poller thread                                                                             
Sqoop2 server started. 

查看是否启动成功

$ jps | grep Sqoop

30970 SqoopJettyServer

如出现 SqoopJettyServer 进程则表示已启动成功。

免责声明:文章转载自《HDP 3.1.0 集成 Sqoop2》仅用于学习参考。如对内容有疑问,请及时联系本站处理。

上篇Qt for windows消息循环、libqxt分析和wince快捷键处理语音识别之梅尔频谱倒数MFCC(Mel Frequency Cepstrum Coefficient)下篇

宿迁高防,2C2G15M,22元/月;香港BGP,2C5G5M,25元/月 雨云优惠码:MjYwNzM=

相关文章

PHP手动搭建环境

php手动搭建环境有好多种组合,版本号不一致,会导致搭建失败。 我搭建的组合是: php5.6+MySQL5.6+Apache2.4的组合。 一、PHP语言包下载 首先从官网上下载php5.6 http://windows.php.net/download#php-5.6  选择完整版下载: 二、 Apache服务器下载 首先从官网上下载Apache2....

01 HDFS 简介

01.HDFS简介 大纲: hadoop2 介绍 HDFS概述 HDFS读写流程 hadoop2介绍 框架的核心设计是HDFS(存储),mapReduce(分布式计算),YARN(资源管理),为海量的数据提供了存储和计算。 Hadoop 1.0(MapReduce,HDFS) Hadoop2 主要改进 YARN NameNode HA HDFS fe...

hadoop 多次格式化format namenode 导致节点不能启动解决

链接:https://blog.csdn.net/qq_41059374/article/details/80695581?depth_1-utm_source=distribute.pc_relevant.none-task&utm_source=distribute.pc_relevant.none-task...

Apache设置反向代理

一、配置http反向代理地址 1、首先确保Apache有这些模块,在Apache根目录下确认有这些模块,主要包含以下模块: mod_proxy.so mod_proxy_ajp.so mod_proxy_balancer.so mod_proxy_connect.so mod_proxy_http.so  然后打开配置conf/httpd.conf文件,将...

HDFS只支持文件append操作, 而依赖HDFS的HBase如何完成数据的增删改查

转:http://www.th7.cn/db/nosql/201510/135382.shtml 1. HDFS的文件append功能 早期版本的HDFS不支持任何的文件更新操作,一旦一个文件创建、写完数据、并关闭之后,这个文件就再也不能被改变了。为什么这么设计?是为了与MapReduce完美配合,MapReduce的工作模式是接受一系列输入文件,经过ma...

hadoop伪分布式搭建

JDK安装配置执行 rpm -i jdk-8u91-linux-x64.rpm 默认安装在/usr/java/jdk1.8.0_91目录中 配置环境变量vi /etc/profile在最后一行添加 export JAVA_HOME=/usr/java/jdk1.8.0_91 export PATH=$PATH:$JAVA_HOME/bin...