大数据学习教程SD版第十二篇【Canal】

大数据学习教程SD版第十二篇【Canal】

Canal 数据实时采集工具

Alibaba 解析MySQL Binlog

1. MySQL Binlog

包括两类文件：日志索引文件、 *** 作日志文件

• 使用场景

  1. 主从数据一致
  2. 数据恢复

• 分类

  1. statement 语句级 节省空间，但可能造成数据不一致now()
  2. row 行级 数据绝对一致，占用空间大
  3. mixed statement升级版

2. Canal 工作原理

与MySQL主从复制一致，自己伪装成Slave

3. Canal 使用场景

1. 更新缓存
2. 抓取动态数据

4. Canal 使用步骤

1. 开启MySQL的binlog，并重启MySQL

server-id=1
log_bin=mysql-bin
binlog_format=row
binlog-do-db=canal01

systemctl restart mysqld

2. 赋予Canal 读数据权限

grant select,replication slave,replication client on *.* to 'canal'@'%' identified by 'canal';

5. Canal 安装

1. 下载并解压安装包，先创建一个解压目录
2. 修改配置

instance.properties

canal.instance.mysql.slaveId=0
canal.instance.master.address=hadoop102:3306

3. 启动canal

bin/startup.sh

6. Canal 使用 6.1 TCP模式

使用代码方式获取采集的数据：

当 EntryType 为 RowDATA ，获取Entry 里的Storevalue，反序列化为RowChange，有变化前信息和变化后信息

新增数据：insert after

删除数据：delete before

修改数据：update before after

6.2 Kafka模式

修改配置为Kafka模式

canal.properties

canal.serverMode = kafka
canal.mq.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092

instance.properties

canal.mq.topic=canal_test

1. 开启Canal
2. 开启Kfka ，并开启Kafka Consumer

null:{"data":[{"id":"1003","name":"liuliu","sex":"male"}],"database":"canal01","es":1640935296000,"id":3,"isDdl":false,"mysqlType":{"id":"int(11)","name":"varchar(255)","sex":"varchar(255)"},"old":[{"name":"wangwu"}],"sql":"","sqlType":{"id":4,"name":12,"sex":12},"table":"canal_test","ts":1640935296674,"type":"UPDATE"}

null:{"data":[{"id":"1004","name":"aaa","sex":"male"}],"database":"canal01","es":1640935350000,"id":4,"isDdl":false,"mysqlType":{"id":"int(11)","name":"varchar(255)","sex":"varchar(255)"},"old":null,"sql":"","sqlType":{"id":4,"name":12,"sex":12},"table":"canal_test","ts":1640935350765,"type":"INSERT"}

null:{"data":[{"id":"1004","name":"aaa","sex":"male"}],"database":"canal01","es":1640935382000,"id":5,"isDdl":false,"mysqlType":{"id":"int(11)","name":"varchar(255)","sex":"varchar(255)"},"old":null,"sql":"","sqlType":{"id":4,"name":12,"sex":12},"table":"canal_test","ts":1640935383165,"type":"DELETE"}