linux mem overcommit

2023-11-30 12:48
文章标签 linux mem overcommit

本文主要是介绍linux mem overcommit,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

 linux overcommit 机制会欺骗进程,承诺分配给他的内存空间,只有当这些page被真正touch到的时候才通过缺页机制拿到真正的物理内存。

以下摘自kernel doc

The Linux kernel supports the following overcommit handling modes0	-	Heuristic overcommit handling. Obvious overcommits ofaddress space are refused. Used for a typical system. Itensures a seriously wild allocation fails while allowingovercommit to reduce swap usage.  root is allowed to allocate slightly more memory in this mode. This is the default.1	-	Always overcommit. Appropriate for some scientificapplications. Classic example is code using sparse arraysand just relying on the virtual memory consisting almostentirely of zero pages.2	-	Don't overcommit. The total address space commitfor the system is not permitted to exceed swap + aconfigurable amount (default is 50%) of physical RAM.Depending on the amount you use, in most situationsthis means a process will not be killed while accessingpages but will receive errors on memory allocation asappropriate.Useful for applications that want to guarantee theirmemory allocations will be available in the futurewithout having to initialize every page.The overcommit policy is set via the sysctl `vm.overcommit_memory'.The overcommit amount can be set via `vm.overcommit_ratio' (percentage)
or `vm.overcommit_kbytes' (absolute value).The current overcommit limit and amount committed are viewable in
/proc/meminfo as CommitLimit and Committed_AS respectively.Gotchas
-------The C language stack growth does an implicit mremap. If you want absolute
guarantees and run close to the edge you MUST mmap your stack for the 
largest size you think you will need. For typical stack usage this does
not matter much but it's a corner case if you really really careIn mode 2 the MAP_NORESERVE flag is ignored. How It Works
------------The overcommit is based on the following rulesFor a file backed mapSHARED or READ-only	-	0 cost (the file is the map not swap)PRIVATE WRITABLE	-	size of mapping per instanceFor an anonymous or /dev/zero mapSHARED			-	size of mappingPRIVATE READ-only	-	0 cost (but of little use)PRIVATE WRITABLE	-	size of mapping per instanceAdditional accountingPages made writable copies by mmapshmfs memory drawn from the same poolStatus
------o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcementTo Do
-----
o	Account ptrace pages (this is hard)
CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'),this is the total amount of  memory currently available tobe allocated on the system. This limit is only adhered toif strict overcommit accounting is enabled (mode 2 in'vm.overcommit_memory').The CommitLimit is calculated with the following formula:CommitLimit = ([total RAM pages] - [total huge TLB pages]) *overcommit_ratio / 100 + [total swap pages]For example, on a system with 1G of physical RAM and 7Gof swap with a `vm.overcommit_ratio` of 30 it wouldyield a CommitLimit of 7.3G.For more details, see the memory overcommit documentationin vm/overcommit-accounting.
Committed_AS: The amount of memory presently allocated on the system.The committed memory is a sum of all of the memory whichhas been allocated by processes, even if it has not been"used" by them as of yet. A process which malloc()'s 1Gof memory, but only touches 300M of it will show up asusing 1G. This 1G is memory which has been "committed" toby the VM and can be used at any time by the allocatingapplication. With strict overcommit enabled on the system(mode 2 in 'vm.overcommit_memory'),allocations which wouldexceed the CommitLimit (detailed above) will not be permitted.This is useful if one needs to guarantee that processes willnot fail due to lack of memory once that memory has beensuccessfully allocated.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   for(i=0; i< 1024*1024*30; i++);{   temp_malloc[i] = i&0xff;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

 如上程序以为能分配到物理内存,可是实际上free命令结果看不到used memory增加

meminfo结果对比,申请的30M都被over committed 了,所以如上程序也没有真的touch到内存? 

 

/ # busybox pmap  2655
2655: {no such process} /mnt/mmc/sdcard/largemem
00010000       4K r-xp  /mnt/mmc/sdcard/largemem
00020000       4K r--p  /mnt/mmc/sdcard/largemem
00021000       4K rw-p  /mnt/mmc/sdcard/largemem
00022000     132K rw-p  [heap]
b50d0000   30724K rw-p    [ anon ]
b6ed1000     892K r-xp  /lib/libc-2.30.so
b6fb0000      60K ---p  /lib/libc-2.30.so
b6fbf000       8K r--p  /lib/libc-2.30.so
b6fc1000       4K rw-p  /lib/libc-2.30.so
b6fc2000      12K rw-p    [ anon ]
b6fc5000     100K r-xp  /lib/ld-2.30.so
b6feb000       8K rw-p    [ anon ]
b6fed000       4K r--p  /lib/ld-2.30.so
b6fee000       4K rw-p  /lib/ld-2.30.so
bed26000     132K rw-p  [stack]
bedd7000       4K r-xp  [sigpage]
bedd8000       4K r--p  [vvar]
bedd9000       4K r-xp  [vdso]
ffff0000       4K r-xp  [vectors]
mapped: 32108K

pmap的结果也显示出了30M的内存使用

google 查到有人用mlock 来实现防止被overcommit掉

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/mman.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   if(mlock(temp_malloc, 1024*1024*30)){   printf("mlock fail\n");free(temp_malloc);return 0;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

测试结果used 果然增加了30M

/ # freetotal         used         free       shared      buffers
Mem:        352328        42864       309464            0           40
-/+ buffers:              42824       309504
Swap:            0            0            0
/ # freestart malloc
wr donetotal         used         free       shared      buffers
Mem:        352328        73576       278752            0           40
-/+ buffers:              73536       278792
Swap:            0            0            0

至于为什么真正使用了malloc的memory也会被overcommit掉的原因,需要另开一篇认真调研下

这篇关于linux mem overcommit的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/437089

相关文章

Linux命令之firewalld的用法

《Linux命令之firewalld的用法》:本文主要介绍Linux命令之firewalld的用法,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录linux命令之firewalld1、程序包2、启动firewalld3、配置文件4、firewalld规则定义的九大

Linux之计划任务和调度命令at/cron详解

《Linux之计划任务和调度命令at/cron详解》:本文主要介绍Linux之计划任务和调度命令at/cron的使用,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录linux计划任务和调度命令at/cron一、计划任务二、命令{at}介绍三、命令语法及功能 :at

Linux下如何使用C++获取硬件信息

《Linux下如何使用C++获取硬件信息》这篇文章主要为大家详细介绍了如何使用C++实现获取CPU,主板,磁盘,BIOS信息等硬件信息,文中的示例代码讲解详细,感兴趣的小伙伴可以了解下... 目录方法获取CPU信息:读取"/proc/cpuinfo"文件获取磁盘信息:读取"/proc/diskstats"文

Linux内核参数配置与验证详细指南

《Linux内核参数配置与验证详细指南》在Linux系统运维和性能优化中,内核参数(sysctl)的配置至关重要,本文主要来聊聊如何配置与验证这些Linux内核参数,希望对大家有一定的帮助... 目录1. 引言2. 内核参数的作用3. 如何设置内核参数3.1 临时设置(重启失效)3.2 永久设置(重启仍生效

kali linux 无法登录root的问题及解决方法

《kalilinux无法登录root的问题及解决方法》:本文主要介绍kalilinux无法登录root的问题及解决方法,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,... 目录kali linux 无法登录root1、问题描述1.1、本地登录root1.2、ssh远程登录root2、

Linux ls命令操作详解

《Linuxls命令操作详解》通过ls命令,我们可以查看指定目录下的文件和子目录,并结合不同的选项获取详细的文件信息,如权限、大小、修改时间等,:本文主要介绍Linuxls命令详解,需要的朋友可... 目录1. 命令简介2. 命令的基本语法和用法2.1 语法格式2.2 使用示例2.2.1 列出当前目录下的文

Linux中的计划任务(crontab)使用方式

《Linux中的计划任务(crontab)使用方式》:本文主要介绍Linux中的计划任务(crontab)使用方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录一、前言1、linux的起源与发展2、什么是计划任务(crontab)二、crontab基础1、cro

Linux换行符的使用方法详解

《Linux换行符的使用方法详解》本文介绍了Linux中常用的换行符LF及其在文件中的表示,展示了如何使用sed命令替换换行符,并列举了与换行符处理相关的Linux命令,通过代码讲解的非常详细,需要的... 目录简介检测文件中的换行符使用 cat -A 查看换行符使用 od -c 检查字符换行符格式转换将

Linux系统配置NAT网络模式的详细步骤(附图文)

《Linux系统配置NAT网络模式的详细步骤(附图文)》本文详细指导如何在VMware环境下配置NAT网络模式,包括设置主机和虚拟机的IP地址、网关,以及针对Linux和Windows系统的具体步骤,... 目录一、配置NAT网络模式二、设置虚拟机交换机网关2.1 打开虚拟机2.2 管理员授权2.3 设置子

Linux系统中卸载与安装JDK的详细教程

《Linux系统中卸载与安装JDK的详细教程》本文详细介绍了如何在Linux系统中通过Xshell和Xftp工具连接与传输文件,然后进行JDK的安装与卸载,安装步骤包括连接Linux、传输JDK安装包... 目录1、卸载1.1 linux删除自带的JDK1.2 Linux上卸载自己安装的JDK2、安装2.1