51工具盒子在日常巡检中发现某服务器服务无数据，通过查看java服务输出发现如下报错：
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000ffff7cab0000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
排查思路
- 减少系统内存负载
- 增加物理内存或交换空间
- 检查交换后备存储是否已满
- 在 64 位操作系统上使用 64 位 Java
- 减小 Java 堆大小 (
-Xmx/-Xms) - 减少 Java 线程数
- 减小 Java 线程堆栈大小 (
-Xss) - 设置更大的代码缓存
-XX:ReservedCodeCacheSize=
查看内存情况
由此可以看到剩余内存并不多。
[collect@localhost ~]$ free -h
total used free shared buff/cache available
Mem: 62G 53G 4.4G 3.1G 4.6G 863M
Swap: 19G 19G 256K
查看内存占用
通过top命令mem排名可以看到rsyslogd服务占用内存过多
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14088 root 20 0 68.7g 48.8g 43840 S 20.6 77.5 3933:15 rsyslogd
清理内存占用过多服务
[root@localhost ~]# systemctl status 14088
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since 一 2022-10-17 15:47:10 CST; 1 months 13 days ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 14088 (rsyslogd)
Tasks: 3
CGroup: /system.slice/rsyslog.service
└─14088 /usr/sbin/rsyslogd -n
[root@localhost ~]# systemctl restart rsyslog
查看rsyslog服务
[root@localhost ~]# journalctl -u rsyslog
-- Logs begin at 三 2022-11-30 08:33:38 CST, end at 三 2022-11-30 09:37:30 CST. --
11月 30 08:33:38 localhost.localdomain rsyslogd[14088]: sd_journal_get_cursor() failed: 'Cannot assign requested address' [v8.24.0-34.el7]
11月 30 08:33:38 localhost.localdomain rsyslogd[14088]: imjournal: journal reloaded... [v8.24.0-34.el7 try http://www.rsyslog.com/e/0 ]
11月 30 08:36:32 localhost.localdomain rsyslogd[14088]: sd_journal_get_cursor() failed: 'Cannot assign requested address' [v8.24.0-34.el7]
11月 30 08:36:32 localhost.localdomain rsyslogd[14088]: imjournal: journal reloaded... [v8.24.0-34.el7 try http://www.rsyslog.com/e/0 ]
11月 30 08:41:53 localhost.localdomain rsyslogd[14088]: imjournal: 317109 messages lost due to rate-limiting
11月 30 08:46:17 localhost.localdomain rsyslogd[14088]: imjournal: begin to drop messages due to rate-limiting
修改rsyslog服务配置
根据实际情况新增或修改MemoryAccounting、MemoryMax、MemoryHigh参数进行内存使用限制。
[root@localhost ~]# vi /usr/lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
;Requires=syslog.socket
Wants=network.target network-online.target
After=network.target network-online.target
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/
\[Service\]
Type=notify
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStart=/usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS
Restart=on-failure
UMask=0066
StandardOutput=null
Restart=on-failure
MemoryAccounting=yes
MemoryMax=100M
MemoryHigh=10M
`[Install]
WantedBy=multi-user.target
;Alias=syslog.service`
重启rsyslog服务
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart rsyslog
新增journal目录
由于不存在/var/log/journal/目录，导致rsyslog服务输出报错，占用大量存储及内存空间。
[root@localhost ~]# mkdir /var/log/journal/
参考文档
