CDH6.3.2集成Flink1.12.2

CDH6.3.2集成Flink1.12.2

1. 编译flink-shaded
   flink-shaded(用10.0版本，1.12对应的flink-shaded1.12版本没有flink-shaded-hadoop-2-uber-3.0.0-cdh6.3.2-10.0这个jar包)
   git clone https://github.com/apache/flink-shaded.git
   git checkout -b flinkshaded9 release-9.0
   vim pom.xml

 
            
            
                cloudera-releases
                https://repository.cloudera.com/artifactory/cloudera-repos
                
                    true
                
                
                    false
                
            
            
            
                HDPReleases
                HDP Releases
                https://repo.hortonworks.com/content/repositories/releases/
                false
                true
            
            
                HortonworksJettyHadoop
                HDP Jetty
                https://repo.hortonworks.com/content/repositories/jetty-hadoop
                false
                true
            
            
            
                mapr-releases
                https://repository.mapr.com/maven/
                false
                true
            
        

mvn -T24C clean install -DskipTests -Pvendor-repos -Dhadoop.version=3.0.0-cdh6.3.2 -Dscala-2.12 -Drat.skip=true

2. 编译 flink
   git clone https://github.com/apache/flink.git
   git checkout -b flink1.12 release-1.12.2
   vim pom.xml

 
        
            
            
                cloudera-releases
                https://repository.cloudera.com/artifactory/cloudera-repos
                
                    true
                
                
                    false
                
            
            
            
                HDPReleases
                HDP Releases
                https://repo.hortonworks.com/content/repositories/releases/
                false
                true
            
            
                HortonworksJettyHadoop
                HDP Jetty
                https://repo.hortonworks.com/content/repositories/jetty-hadoop
                false
                true
            
            
            
                mapr-releases
                https://repository.mapr.com/maven/
                false
                true
            
        

mvn clean install -DskipTests -Dfast -Drat.skip=true -Dhaoop.version=3.0.0-cdh6.3.2 -Pvendor-repos -Dinclude-hadoop -Dscala-2.12 -T14C

3. 打包编译的Flink应用
   cd flink-dist/target/flink-1.12.2-bin
   tar -zcf flink-1.12.2-bin-scala_2.12.tgz flink-1.12.2/

4. 下载工具把flink包转换为CDH能识别的包
   git clone https://github.com/pkeropen/flink-parcel.git
   cd flink-parcel
   vim flink-parcel.properties

#Flink 下载地址
#Flink_URL=https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.12.tgz
Flink_URL=./flink-1.12.2-bin-scala_2.12.tgz
#flink版本号
Flink_VERSION=1.12.2

#扩展版本号
EXTENS_VERSION=BIN-SCALA_2.12

# *** 作系统版本，以centos为例
OS_VERSION=7

#CDH 小版本
CDH_MIN_FULL=6.1
CDH_MAX_FULL=6.5

#CDH大版本
CDH_MIN=6
CDH_MAX=6

bash ./build.sh parcel
Flink-1.12.2-BIN-SCALA_2.12_build

bash ./build.sh csd_on_yarn
Flink_ON_YARN-1.12.2.jar

5. 打包拷贝到CDH集群
   tar -zcvf Flink-1.12.2-BIN-SCALA_2.12_build.tar.gz Flink-1.12.2-BIN-SCALA_2.12_build
   cp Flink-1.12.2-BIN-SCALA_2.12_build.tar.gz Flink_ON_YARN-1.12.2.jar /opt/cloudera/parcel-repo
   cp Flink_ON_YARN-1.12.2.jar /opt/cloudera/csd
   tar -zxvf Flink-1.12.2-BIN-SCALA_2.12_build.tar.gz -C ./

6. web 页面激活分配

7. 激活分配后把flink-shaded-hadoop-2-uber-3.0.0-cdh6.3.2-10.0.jar文件移动到/opt/cloudera/parcels/Flink/lib/flink/lib目录下，不然add服务启动会报错

8. 报错解决
   在 ‘select integral(events_informational_rate) where entityName=“flink_on_yarn”’ 中未找到与筛选器 ‘entityName = “flink_on_yarn”’ 匹配的实体
   重启 Cloudera Management Service 解决问题