ORACLE 19C RAC DIAG进程消耗大量内存的分析

2024-05-09 07:04
文章标签 oracle 分析 内存 进程 19c 大量 rac 消耗 diag

本文主要是介绍ORACLE 19C RAC DIAG进程消耗大量内存的分析，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

近期一个ORACLE 19C的RAC环境，多次出现数据库实例的后台进程DIAG消耗很多内存（达到20G+），节点1、节点2都出现过次问题。
问题分析：通过对DIAG进程TRACE分析，结合在ORACLE官方后台进行问题、BUG查询匹配，该进程占用内存的现象不是持续增大，而是突然增大。目前分析是心跳网络不稳定时DIAG进程产生大量TRACE时的内存泄漏，导致DIAG进程占用很多内存（mos:Database Ora_dia0_* Process Consumes High Memory Utilization (文档 ID 2787583.1),此问题一般为私网的性能较差导致）。
处理办法：
1、之前是对DIAG进程人工KILL后释放内存（进程KILL后会自动启动不影响业务）；后续考虑增加脚本在后台KILL此进程。
2、增加主机层的OSW监控工具监控网络性能，监控数据保留15天，持续监控分析。

相关日志如下：

1、消耗很多内存的DIAG进行信息截图，截图如下：

2、分析DIAG进程的TRACE日志，大量的IPCLW开头的日志

*** 2024-04-12T21:20:03.747180+08:00
2024-04-12 21:20:03.747 :kjzgrcfg_finalize(): Reconfig finished inc:8 state:5
2024-04-12 21:20:03.789 :kjzgrcfg_msghdlr(): Step 1 remote_inc:8 prop_inc:8 state:5 master:0
IPCLW:[0.123777436]{E}[WAIT]:PROTO: [1712928004190595]RETRANS DBG local acnh 0x7fcfc4721ca8 dump:
IPCLW:[0.123777437]{-}[WAIT]:UTIL: [1712928004190595]  ACNH 0x7fcfc4721ca8 State: 1 SMSN: 69887638 PKT(69906133.325238309) # Pending: 4096
IPCLW:[0.123777438]{-}[WAIT]:UTIL: [1712928004190595]   Peer: DIA0.KSXP_diag.67881 AckSeq: 325234213
IPCLW:[0.123777439]{-}[WAIT]:UTIL: [1712928004190595]   Flags: 0x20000000 IHint: 0x5e64fe2a0000001f THint: 0x3218cc490000001c 
IPCLW:[0.123777440]{-}[WAIT]:UTIL: [1712928004190595]   Local Address: 169.254.12.225:60038 Remote Address: 169.254.1.19:51696
IPCLW:[0.123777441]{-}[WAIT]:UTIL: [1712928004190595]   Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0 xdata3 28b1 xdata2 e85e640e
IPCLW:[0.123777442]{-}[WAIT]:UTIL: [1712928004190595]             : mmsz 32768 mmr 4096 mms 4096 xdata 101630a2
IPCLW:[0.123777443]{-}[WAIT]:UTIL: [1712928004190595]   IVPort: 63416 TVPort: 12450 IMPT: 47425 RMPT: 10417   Pending Sends: Yes Unacked Sends: Yes 
IPCLW:[0.123777444]{-}[WAIT]:UTIL: [1712928004190595]   Send Engine Queued: Yes sshdl -1 ssts 0 rtts 1712928004190837 snderrchk 5 creqcnt 1 credits 0/8
IPCLW:[0.123777445]{-}[WAIT]:UTIL: [1712928004190595]   Unackd Messages 69902037 -> 69906132. SSEQ 325234213 Send Time: INVALID TIME SMSN # Xmits: 3977664 EMSN INVALID TIME
IPCLW:[0.123777446]{-}[WAIT]:UTIL: [1712928004190595]  Pending send queue: 
IPCLW:[0.123777447]{-}[WAIT]:UTIL: [1712928004190595]    [0] mbuf 0x7fcfc47467d0 MSN 69902037 Seq 325234213 -> 325234214 # XMits: 3977664
IPCLW:[0.123777448]{-}[WAIT]:UTIL: [1712928004190595]    [1] mbuf 0x7fcfc4746bf0 MSN 69902038 Seq 325234214 -> 325234215 # XMits: 3977663
IPCLW:[0.123777449]{-}[WAIT]:UTIL: [1712928004190595]    [2] mbuf 0x7fcfc47463b0 MSN 69902039 Seq 325234215 -> 325234216 # XMits: 3977662
IPCLW:[0.123777450]{-}[WAIT]:UTIL: [1712928004190595]    [3] mbuf 0x7fcfc4745f90 MSN 69902040 Seq 325234216 -> 325234217 # XMits: 3977661
IPCLW:[0.123777451]{-}[WAIT]:UTIL: [1712928004190595]    [4] mbuf 0x7fcfc4745b70 MSN 69902041 Seq 325234217 -> 325234218 # XMits: 3977660
IPCLW:[0.123777452]{-}[WAIT]:UTIL: [1712928004190595]    [5] mbuf 0x7fcfc4745750 MSN 69902042 Seq 325234218 -> 325234219 # XMits: 3977659
IPCLW:[0.123777453]{-}[WAIT]:UTIL: [1712928004190595]    [6] mbuf 0x7fcfc4745330 MSN 69902043 Seq 325234219 -> 325234220 # XMits: 3977658
IPCLW:[0.123777454]{-}[WAIT]:UTIL: [1712928004190595]    [7] mbuf 0x7fcfc4744f10 MSN 69902044 Seq 325234220 -> 325234221 # XMits: 3977657
IPCLW:[0.123777455]{-}[WAIT]:UTIL: [1712928004190595]    [8] mbuf 0x7fcfc4744af0 MSN 69902045 Seq 325234221 -> 325234222 # XMits: 0
IPCLW:[0.123777456]{-}[WAIT]:UTIL: [1712928004190595]    [9] mbuf 0x7fcfc47446d0 MSN 69902046 Seq 325234222 -> 325234223 # XMits: 0
IPCLW:[0.123777457]{-}[WAIT]:UTIL: [1712928004190595]    [10] mbuf 0x7fcfc47442b0 MSN 69902047 Seq 325234223 -> 325234224 # XMits: 0
IPCLW:[0.123777458]{-}[WAIT]:UTIL: [1712928004190595]    [11] mbuf 0x7fcfc4743e90 MSN 69902048 Seq 325234224 -> 325234225 # XMits: 0
IPCLW:[0.123777459]{-}[WAIT]:UTIL: [1712928004190595]    [12] mbuf 0x7fcfc4743a70 MSN 69902049 Seq 325234225 -> 325234226 # XMits: 0
IPCLW:[0.123777460]{-}[WAIT]:UTIL: [1712928004190595]    [13] mbuf 0x7fcfc4743650 MSN 69902050 Seq 325234226 -> 325234227 # XMits: 0
IPCLW:[0.123777461]{-}[WAIT]:UTIL: [1712928004190595]    [14] mbuf 0x7fcfc4743230 MSN 69902051 Seq 325234227 -> 325234228 # XMits: 0
IPCLW:[0.123777462]{-}[WAIT]:UTIL: [1712928004190595]    [15] mbuf 0x7fcfc4742e10 MSN 69902052 Seq 325234228 -> 325234229 # XMits: 0
IPCLW:[0.123777463]{-}[WAIT]:UTIL: [1712928004190595]    [16] mbuf 0x7fcfc47429f0 MSN 69902053 Seq 325234229 -> 325234230 # XMits: 0
IPCLW:[0.123777464]{-}[WAIT]:UTIL: [1712928004190595]    [17] mbuf 0x7fcfc47425d0 MSN 69902054 Seq 325234230 -> 325234231 # XMits: 0
IPCLW:[0.123777465]{-}[WAIT]:UTIL: [1712928004190595]    [18] mbuf 0x7fcfc47421b0 MSN 69902055 Seq 325234231 -> 325234232 # XMits: 0
IPCLW:[0.123777466]{-}[WAIT]:UTIL: [1712928004190595]    [19] mbuf 0x7fcfc4741d90 MSN 69902056 Seq 325234232 -> 325234233 # XMits: 0
IPCLW:[0.123777467]{-}[WAIT]:UTIL: [1712928004190595]    [20] mbuf 0x7fcfc4741970 MSN 69902057 Seq 325234233 -> 325234234 # XMits: 0
IPCLW:[0.123777468]{-}[WAIT]:UTIL: [1712928004190595]    [21] mbuf 0x7fcfc4741550 MSN 69902058 Seq 325234234 -> 325234235 # XMits: 0
IPCLW:[0.123777469]{-}[WAIT]:UTIL: [1712928004190595]    [22] mbuf 0x7fcfc4741130 MSN 69902059 Seq 325234235 -> 325234236 # XMits: 0
IPCLW:[0.123777470]{-}[WAIT]:UTIL: [1712928004190595]    [23] mbuf 0x7fcfc4740d10 MSN 69902060 Seq 325234236 -> 325234237 # XMits: 0
IPCLW:[0.123777471]{-}[WAIT]:UTIL: [1712928004190595]    [24] mbuf 0x7fcfc47408f0 MSN 69902061 Seq 325234237 -> 325234238 # XMits: 0
IPCLW:[0.123777472]{-}[WAIT]:UTIL: [1712928004190595]    [25] mbuf 0x7fcfc47404d0 MSN 69902062 Seq 325234238 -> 325234239 # XMits: 0
IPCLW:[0.123777473]{-}[WAIT]:UTIL: [1712928004190595]    [26] mbuf 0x7fcfc47400b0 MSN 69902063 Seq 325234239 -> 325234240 # XMits: 0
IPCLW:[0.123777474]{-}[WAIT]:UTIL: [1712928004190595]    [27] mbuf 0x7fcfc473fc90 MSN 69902064 Seq 325234240 -> 325234241 # XMits: 0
IPCLW:[0.123777475]{-}[WAIT]:UTIL: [1712928004190595]    [28] mbuf 0x7fcfc473f870 MSN 69902065 Seq 325234241 -> 325234242 # XMits: 0
IPCLW:[0.123777476]{-}[WAIT]:UTIL: [1712928004190595]    [29] mbuf 0x7fcfc473f450 MSN 69902066 Seq 325234242 -> 325234243 # XMits: 0
IPCLW:[0.123777477]{-}[WAIT]:UTIL: [1712928004190595]    [30] mbuf 0x7fcfc473f030 MSN 69902067 Seq 325234243 -> 325234244 # XMits: 0
IPCLW:[0.123777478]{-}[WAIT]:UTIL: [1712928004190595]    [31] mbuf 0x7fcfc473ec10 MSN 69902068 Seq 325234244 -> 325234245 # XMits: 0
IPCLW:[0.123777479]{-}[WAIT]:UTIL: [1712928004190595]    [32] mbuf 0x7fcfc473e7f0 MSN 69902069 Seq 325234245 -> 325234246 # XMits: 0
IPCLW:[0.123777480]{-}[WAIT]:UTIL: [1712928004190595]    [33] mbuf 0x7fcfc473e3d0 MSN 69902070 Seq 325234246 -> 325234247 # XMits: 0
IPCLW:[0.123777481]{-}[WAIT]:UTIL: [1712928004190595]    [34] mbuf 0x7fcfc473dfb0 MSN 69902071 Seq 325234247 -> 325234248 # XMits: 0
IPCLW:[0.123777482]{-}[WAIT]:UTIL: [1712928004190595]    [35] mbuf 0x7fcfc473db90 MSN 69902072 Seq 325234248 -> 325234249 # XMits: 0
IPCLW:[0.123777483]{-}[WAIT]:UTIL: [1712928004190595]    [36] mbuf 0x7fcfc473d770 MSN 69902073 Seq 325234249 -> 325234250 # XMits: 0
IPCLW:[0.123777484]{-}[WAIT]:UTIL: [1712928004190595]    [37] mbuf 0x7fcfc473d350 MSN 69902074 Seq 325234250 -> 325234251 # XMits: 0
IPCLW:[0.123777485]{-}[WAIT]:UTIL: [1712928004190595]    [38] mbuf 0x7fcfc473cf30 MSN 69902075 Seq 325234251 -> 325234252 # XMits: 0
IPCLW:[0.123777486]{-}[WAIT]:UTIL: [1712928004190595]    [39] mbuf 0x7fcfc473cb10 MSN 69902076 Seq 325234252 -> 325234253 # XMits: 0
IPCLW:[0.123777487]{-}[WAIT]:UTIL: [1712928004190595]    [40] mbuf 0x7fcfc473c6f0 MSN 69902077 Seq 325234253 -> 325234254 # XMits: 0
IPCLW:[0.123777488]{-}[WAIT]:UTIL: [1712928004190595]    [41] mbuf 0x7fcfc473c2d0 MSN 69902078 Seq 325234254 -> 325234255 # XMits: 0
IPCLW:[0.123777489]{-}[WAIT]:UTIL: [1712928004190595]    [42] mbuf 0x7fcfc473beb0 MSN 69902079 Seq 325234255 -> 325234256 # XMits: 0
IPCLW:[0.123777490]{-}[WAIT]:UTIL: [1712928004190595]    [43] mbuf 0x7fcfc473ba90 MSN 69902080 Seq 325234256 -> 325234257 # XMits: 0
IPCLW:[0.123777491]{-}[WAIT]:UTIL: [1712928004190595]    [44] mbuf 0x7fcfc473b670 MSN 69902081 Seq 325234257 -> 325234258 # XMits: 0
IPCLW:[0.123777492]{-}[WAIT]:UTIL: [1712928004190595]    [45] mbuf 0x7fcfc473b250 MSN 69902082 Seq 325234258 -> 325234259 # XMits: 0
IPCLW:[0.123777493]{-}[WAIT]:UTIL: [1712928004190595]    [46] mbuf 0x7fcfc473ae30 MSN 69902083 Seq 325234259 -> 325234260 # XMits: 0
IPCLW:[0.123777494]{-}[WAIT]:UTIL: [1712928004190595]    [47] mbuf 0x7fcfc473aa10 MSN 69902084 Seq 325234260 -> 325234261 # XMits: 0
IPCLW:[0.123777495]{-}[WAIT]:UTIL: [1712928004190595]    [48] mbuf 0x7fcfc473a5f0 MSN 69902085 Seq 325234261 -> 325234262 # XMits: 0
IPCLW:[0.123777496]{-}[WAIT]:UTIL: [1712928004190595]    [49] mbuf 0x7fcfc473a1d0 MSN 69902086 Seq 325234262 -> 325234263 # XMits: 0
IPCLW:[0.123777497]{-}[WAIT]:UTIL: [1712928004190595]    [50] mbuf 0x7fcfc4739db0 MSN 69902087 Seq 325234263 -> 325234264 # XMits: 0
IPCLW:[0.123777498]{-}[WAIT]:UTIL: [1712928004190595]    [51] mbuf 0x7fcfc4739990 MSN 69902088 Seq 325234264 -> 325234265 # XMits: 0
IPCLW:[0.123777499]{-}[WAIT]:UTIL: [1712928004190595]    [52] mbuf 0x7fcfc4739570 MSN 69902089 Seq 325234265 -> 325234266 # XMits: 0
IPCLW:[0.123777500]{-}[WAIT]:UTIL: [1712928004190595]    [53] mbuf 0x7fcfc4739150 MSN 69902090 Seq 325234266 -> 325234267 # XMits: 0
IPCLW:[0.123777501]{-}[WAIT]:UTIL: [1712928004190595]    [54] mbuf 0x7fcfc4738d30 MSN 69902091 Seq 325234267 -> 325234268 # XMits: 0
IPCLW:[0.123777502]{-}[WAIT]:UTIL: [1712928004190595]    [55] mbuf 0x7fcfc4738910 MSN 69902092 Seq 325234268 -> 325234269 # XMits: 0
 

3、MOS文档中关于此问题描述

Oracle Database - Enterprise Edition - Version 19.8.0.0.0 and later
Database ora_dia0_* process consumes High Memory utilization
<Instance>_dia0_21668_base_5.trc
==============================
*** 2021-06-07T23:02:45.794273-07:00 (CDB$ROOT(1))
2021-06-07 23:02:45.794 :kjzgrcfg_finalize(): Reconfig finished inc:31 state:5
2021-06-07 23:02:45.800 :kjzgrcfg_msghdlr(): Step 1 remote_inc:31 prop_inc:31 state:5 master:2
IPCLW:[0.124755339]{E}[WAIT]:PROTO: [1623132166001387]RETRANS DBG local acnh 0x7fe9aa139530 dump:
IPCLW:[0.124755340]{-}[WAIT]:UTIL: [1623132166001387] ACNH 0x7fe9aa139530 State: 1 SMSN:
1655824697 PKT(1662191011.496113055) # Pending: 4096 >>>>>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755341]{-}[WAIT]:UTIL: [1623132166001387] Peer: DIA0.KSXP_diag.30620 AckSeq: 496108959
IPCLW:[0.124755342]{-}[WAIT]:UTIL: [1623132166001387] Flags: 0x20000000 IHint: 0x1d07ad590000001f
THint: 0x52fe03630000001f
IPCLW:[0.124755343]{-}[WAIT]:UTIL: [1623132166001387] Local Address: 192.xxx.xx.120:39061 Remote
Address: 192.xxx.12.xxx:35575 >>>>>>>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755344]{-}[WAIT]:UTIL: [1623132166001387] Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0
xdata3 8227 xdata2 b8300501
IPCLW:[0.124755345]{-}[WAIT]:UTIL: [1623132166001387] : mmsz 32768 mmr 4096 mms 4096 xdata
b883f95f
IPCLW:[0.124755346]{-}[WAIT]:UTIL: [1623132166001387] IVPort: 55103 TVPort: 63839 IMPT: 52824
RMPT: 33319 Pending Sends: Yes Unacked Sends: Yes >>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755347]{-}[WAIT]:UTIL: [1623132166001387] Send Engine Queued: Yes sshdl -1 ssts 0 rtts
1623132166002134 snderrchk 5 creqcnt 1 credits 0/8
IPCLW:[0.124755348]{-}[WAIT]:UTIL: [1623132166001387] Unackd Messages 1662186915 ->
1662191010. SSEQ 496108959 Send Time: INVALID TIME SMSN # Xmits: 4227520 EMSN INVALID TIME
>>>>>>>>>
IPCLW:[0.124755349]{-}[WAIT]:UTIL: [1623132166001387
MOS文档中的案例信息与TRACE是匹配的，官方文档对此解释是心跳网络异常，需要检查处理网络问题：
CAUSE
Packet Reassembly Failure at Interconnect
SOLUTION
Involve System Admin for fixing the packet reassembly failure/Packet drop at interconnect.
4、检查网络状态，有少量报错，符合文档上说的现象。

这篇关于ORACLE 19C RAC DIAG进程消耗大量内存的分析的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！


http://www.chinasem.cn/article/972711。

相关文章

Go标准库常见错误分析和解决办法

Go标准库常见错误分析和解决办法

《Go标准库常见错误分析和解决办法》Go语言的标准库为开发者提供了丰富且高效的工具,涵盖了从网络编程到文件操作等各个方面,然而,标准库虽好,使用不当却可能适得其反,正所谓工欲善其事,必先利其器,本文将... 目录1. 使用了错误的time.Duration2. time.After导致的内存泄漏3. jsO
阅读更多...
Python如何使用__slots__实现节省内存和性能优化

Python如何使用__slots__实现节省内存和性能优化

《Python如何使用__slots__实现节省内存和性能优化》你有想过,一个小小的__slots__能让你的Python类内存消耗直接减半吗,没错,今天咱们要聊的就是这个让人眼前一亮的技巧,感兴趣的... 目录背景：内存吃得满满的类__slots__：你的内存管理小助手举个大概的例子：看看效果如何？1.
阅读更多...
Spring事务中@Transactional注解不生效的原因分析与解决

Spring事务中@Transactional注解不生效的原因分析与解决

《Spring事务中@Transactional注解不生效的原因分析与解决》在Spring框架中,@Transactional注解是管理数据库事务的核心方式,本文将深入分析事务自调用的底层原理,解释为... 目录1. 引言2. 事务自调用问题重现2.1 示例代码2.2 问题现象3. 为什么事务自调用会失效3
阅读更多...
找不到Anaconda prompt终端的原因分析及解决方案

找不到Anaconda prompt终端的原因分析及解决方案

《找不到Anacondaprompt终端的原因分析及解决方案》因为anaconda还没有初始化,在安装anaconda的过程中,有一行是否要添加anaconda到菜单目录中,由于没有勾选,导致没有菜... 目录问题原因问http://www.chinasem.cn题解决安装了 Anaconda 却找不到 An
阅读更多...
Spring定时任务只执行一次的原因分析与解决方案

Spring定时任务只执行一次的原因分析与解决方案

《Spring定时任务只执行一次的原因分析与解决方案》在使用Spring的@Scheduled定时任务时,你是否遇到过任务只执行一次,后续不再触发的情况？这种情况可能由多种原因导致,如未启用调度、线程... 目录1. 问题背景2. Spring定时任务的基本用法3. 为什么定时任务只执行一次？3.1 未启用
阅读更多...
C++ 各种map特点对比分析

C++ 各种map特点对比分析

《C++各种map特点对比分析》文章比较了C++中不同类型的map（如std::map,std::unordered_map,std::multimap,std::unordered_multima... 目录特点比较C++ 示例代码 ​​​​​​代码解释特点比较1. std::map底层实现：基于红黑
阅读更多...
Oracle存储过程里操作BLOB的字节数据的办法

Oracle存储过程里操作BLOB的字节数据的办法

《Oracle存储过程里操作BLOB的字节数据的办法》该篇文章介绍了如何在Oracle存储过程中操作BLOB的字节数据,作者研究了如何获取BLOB的字节长度、如何使用DBMS_LOB包进行BLOB操作... 目录一、缘由二、办法2.1 基本操作2.2 DBMS_LOB包2.3 字节级操作与RAW数据类型2.
阅读更多...
Linux中的进程间通信之匿名管道解读

Linux中的进程间通信之匿名管道解读

《Linux中的进程间通信之匿名管道解读》：本文主要介绍Linux中的进程间通信之匿名管道解读,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录一、基本概念二、管道1、温故知新2、实现方式3、匿名管道（一）管道中的四种情况（二）管道的特性总结一、基本概念我们知道多
阅读更多...
查看Oracle数据库中UNDO表空间的使用情况(最新推荐)

查看Oracle数据库中UNDO表空间的使用情况(最新推荐)

《查看Oracle数据库中UNDO表空间的使用情况(最新推荐)》Oracle数据库中查看UNDO表空间使用情况的4种方法：DBA_TABLESPACES和DBA_DATA_FILES提供基本信息,V$... 目录1. 通过 DBjavascriptA_TABLESPACES 和 DBA_DATA_FILES
阅读更多...
Linux进程终止的N种方式详解

Linux进程终止的N种方式详解

《Linux进程终止的N种方式详解》进程终止是操作系统中,进程的一个重要阶段,他标志着进程生命周期的结束,下面小编为大家整理了一些常见的Linux进程终止方式,大家可以根据需求选择... 目录前言一、进程终止的概念二、进程终止的场景三、进程终止的实现3.1 程序退出码3.2 运行完毕结果正常3.3 运行完毕
阅读更多...

Copyright China编程 23002807@qq.com

沪ICP备2023033072号-2