V2.2版本有用户遇到接口自动化调试出现死锁的问题,问题现象:除了接口自动化模块,单接口用例、用例执行都是正常的。此处简单记录下死锁出现及排查过程。
1、参考历史出现死锁问题https://github.com/metersphere/metersphere/issues/17972查询了相关数据库相关死锁事务,未发现具体死锁错误 {#1%E3%80%81%E5%8F%82%E8%80%83%E5%8E%86%E5%8F%B2%E5%87%BA%E7%8E%B0%E6%AD%BB%E9%94%81%E9%97%AE%E9%A2%98https%3A%2F%2Fgithub.com%2Fmetersphere%2Fmetersphere%2Fissues%2F17972%E6%9F%A5%E8%AF%A2%E4%BA%86%E7%9B%B8%E5%85%B3%E6%95%B0%E6%8D%AE%E5%BA%93%E7%9B%B8%E5%85%B3%E6%AD%BB%E9%94%81%E4%BA%8B%E5%8A%A1%EF%BC%8C%E6%9C%AA%E5%8F%91%E7%8E%B0%E5%85%B7%E4%BD%93%E6%AD%BB%E9%94%81%E9%94%99%E8%AF%AF}
a、查看正在锁的事务, select * from performance_schema.data_locks;
b、查看等待锁的事务,select * from performance_schema.data_lock_waits;
c、查看正在运行中的事务 SELECT * FROM information_schema.innodb_trx;
d、查看是否锁表,show open tables where IN_use > 0
e、查看正在进行的进程 show processlist
f、show status like "%lock%"
2、配置并开启mysql的BinLog日志 {#2%E3%80%81%E9%85%8D%E7%BD%AE%E5%B9%B6%E5%BC%80%E5%90%AFmysql%E7%9A%84binlog%E6%97%A5%E5%BF%97}
参考了一篇博文《MySQL死锁系列-线上死锁问题排查思路》https://www.cnblogs.com/remcarpediem/p/13843180.html
给数据库配置了binlog日志方法,通过相关的binlog 日志会完整记录事务执行的所有 SQL,借助它找到最终获取锁事务所执行的全部 SQL。然后再进行具体的锁冲突分析。
(1)通过navicat 连接工具执行如下命令,开启binlog开关 {#%EF%BC%881%EF%BC%89%E9%80%9A%E8%BF%87navicat-%E8%BF%9E%E6%8E%A5%E5%B7%A5%E5%85%B7%E6%89%A7%E8%A1%8C%E5%A6%82%E4%B8%8B%E5%91%BD%E4%BB%A4%EF%BC%8C%E5%BC%80%E5%90%AFbinlog%E5%BC%80%E5%85%B3}
-
执行相关sql即可看到记录的binlog日志:
-- 开启标准监控 set GLOBAL innodb_status_output=ON; -- 关闭标准监控 set GLOBAL innodb_status_output=OFF; -- 开启锁监控 set GLOBAL innodb_status_output_locks=ON; -- 关闭锁监控 set GLOBAL innodb_status_output_locks=OFF;
set GLOBAL innodb_print_all_deadlocks=ON; -- 查询binglog日志数据 show master logs;
(2)当死锁出现时,可以进入mysql容器/var/lib/mysql/下找到最近时间的死锁binlog日志。可以相关命令将文件转换为文本格式方便查看Mysqlbinlog -h127.0.0.1 -u root -p --read-from-remote-server binlog.000001 --base64-output=decode-rows -v {#%EF%BC%882%EF%BC%89%E5%BD%93%E6%AD%BB%E9%94%81%E5%87%BA%E7%8E%B0%E6%97%B6%EF%BC%8C%E5%8F%AF%E4%BB%A5%E8%BF%9B%E5%85%A5mysql%E5%AE%B9%E5%99%A8%2Fvar%2Flib%2Fmysql%2F%E4%B8%8B%E6%89%BE%E5%88%B0%E6%9C%80%E8%BF%91%E6%97%B6%E9%97%B4%E7%9A%84%E6%AD%BB%E9%94%81binlog%E6%97%A5%E5%BF%97%E3%80%82%E5%8F%AF%E4%BB%A5%E7%9B%B8%E5%85%B3%E5%91%BD%E4%BB%A4%E5%B0%86%E6%96%87%E4%BB%B6%E8%BD%AC%E6%8D%A2%E4%B8%BA%E6%96%87%E6%9C%AC%E6%A0%BC%E5%BC%8F%E6%96%B9%E4%BE%BF%E6%9F%A5%E7%9C%8Bmysqlbinlog--h127.0.0.1--u-root--p---read-from-remote-server-binlog.000001---base64-output%3Ddecode-rows--v}
-
因为此处MeterSphere没有默认安装Mysqlbinlog命令,我的处理方式是将容器日志拷贝出来,用windows上安装的mysql8.0自带的Mysqlbinlog 命令转换为log日志
- docker cp mysql:/var/lib/mysql/mysql-bin.000040 ./
- mysqlbinlog.exe --base64-output=decode-rows -v "C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql-bin.000024" >mysqlbin.log