linux – OOM杀手用大量(？)的空闲内存杀死东西

概述尽管我的系统上有足够的可用RAM,OOM杀手似乎正在杀人： (Full resolution) (Full resolution) 稍后27分钟和408个进程,系统再次开始响应.我大约一小时后重新启动它,不久之后内存利用率恢复正常(对于这台机器). 经过检查,我的盒子上运行了一些有趣的进程： USER PID %CPU %MEM VSZ RSS TTY STAT S 尽管我的系统上有足够的可用RAM,OOM杀手似乎正在杀人：

(Full resolution)

(Full resolution)

稍后27分钟和408个进程,系统再次开始响应.我大约一小时后重新启动它,不久之后内存利用率恢复正常(对于这台机器).

经过检查,我的盒子上运行了一些有趣的进程：

USER       PID %cpu %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND[...snip...]root      1399 60702042  0.2 482288 1868 ?     Sl   Feb21 21114574:24 /sbin/rsyslogd -i /var/run/syslogd.pID -c 4[...snip...]MysqL     2022 60730428  5.1 1606028 38760 ?   Sl   Feb21 21096396:49 /usr/libexec/MysqLd --basedir=/usr --datadir=/var/lib/MysqL --user=MysqL --log-error=/var/log/MysqLd.log --pID-file=/var/run/MysqLd/MysqLd.pID --socket=/var/lib/MysqL/MysqL.sock[...snip...]

此特定服务器已运行约. 8个小时,这是唯一有两个奇怪值的过程.我怀疑“其他东西”正在发生,可能与这些非感性价值观有关.具体来说,我认为系统认为它是内存不足,而实际上并非如此.毕竟,它认为rsyslogd一直使用55383984％的cpu,而理论上最大值是400％.

这是一个完全最新的CentOS 6安装(6.2),内存为768MB.任何关于如何弄清楚为什么会发生这种情况的建议将不胜感激！

编辑：附加虚拟机. sysctl可调参数..我一直在玩swappiness(显然它是100),而且我还运行一个绝对可怕的脚本,它转储我的缓冲区和缓存(由vm.drop_caches显示为3)同步磁盘每15分钟.这就是为什么重新启动后,缓存的数据增长到一个正常的大小,但然后再次迅速下降.我认识到缓存是一件非常好的事情,但直到我弄明白……

同样有趣的是,当我的页面文件在事件期间增长时,它只达到总可能利用率的约20％,这与真正的OOM事件不同.在频谱的另一端,磁盘在同一时期绝对坚果,这是页面文件播放时OOM事件的特征.

sysctl -a 2> / dev / null | grep’^ vm’：

vm.overcommit_memory = 1vm.panic_on_oom = 0vm.oom_kill_allocating_task = 0vm.extfrag_threshold = 500vm.oom_dump_tasks = 0vm.would_have_oomkilled = 0vm.overcommit_ratio = 50vm.page-cluster = 3vm.dirty_background_ratio = 10vm.dirty_background_bytes = 0vm.dirty_ratio = 20vm.dirty_bytes = 0vm.dirty_writeback_centisecs = 500vm.dirty_expire_centisecs = 3000vm.nr_pdflush_threads = 0vm.swappiness = 100vm.nr_hugepages = 0vm.hugetlb_shm_group = 0vm.hugepages_treat_as_movable = 0vm.nr_overcommit_hugepages = 0vm.lowmem_reserve_ratio = 256   256     32vm.drop_caches = 3vm.min_free_kbytes = 3518vm.percpu_Pagelist_fraction = 0vm.max_map_count = 65530vm.laptop_mode = 0vm.block_dump = 0vm.vfs_cache_pressure = 100vm.legacy_va_layout = 0vm.zone_reclaim_mode = 0vm.min_unmapped_ratio = 1vm.min_slab_ratio = 5vm.stat_interval = 1vm.mmap_min_addr = 4096vm.numa_zoneList_order = defaultvm.scan_unevictable_pages = 0vm.memory_failure_early_kill = 0vm.memory_failure_recovery = 1

编辑：并附上第一条OOM消息……经过仔细检查,它说明了一些东西显然也是为了吃掉整个交换空间.

Feb 21 17:12:49 host kernel: MysqLd invoked oom-killer: gfp_mask=0x201da,order=0,oom_adj=0Feb 21 17:12:51 host kernel: MysqLd cpuset=/ mems_allowed=0Feb 21 17:12:51 host kernel: PID: 2777,comm: MysqLd Not tainted 2.6.32-71.29.1.el6.x86_64 #1Feb 21 17:12:51 host kernel: Call Trace:Feb 21 17:12:51 host kernel: [<ffffffff810c2e01>] ? cpuset_print_task_mems_allowed+0x91/0xb0Feb 21 17:12:51 host kernel: [<ffffffff8110f1bb>] oom_kill_process+0xcb/0x2e0Feb 21 17:12:51 host kernel: [<ffffffff8110f780>] ? select_bad_process+0xd0/0x110Feb 21 17:12:51 host kernel: [<ffffffff8110f818>] __out_of_memory+0x58/0xc0Feb 21 17:12:51 host kernel: [<ffffffff8110fa19>] out_of_memory+0x199/0x210Feb 21 17:12:51 host kernel: [<ffffffff8111ebe2>] __alloc_pages_nodemask+0x832/0x850Feb 21 17:12:51 host kernel: [<ffffffff81150cba>] alloc_pages_current+0x9a/0x100Feb 21 17:12:51 host kernel: [<ffffffff8110c617>] __page_cache_alloc+0x87/0x90Feb 21 17:12:51 host kernel: [<ffffffff8112136b>] __do_page_cache_readahead+0xdb/0x210Feb 21 17:12:51 host kernel: [<ffffffff811214c1>] ra_submit+0x21/0x30Feb 21 17:12:51 host kernel: [<ffffffff8110e1c1>] filemap_fault+0x4b1/0x510Feb 21 17:12:51 host kernel: [<ffffffff81135604>] __do_fault+0x54/0x500Feb 21 17:12:51 host kernel: [<ffffffff81135ba7>] handle_pte_fault+0xf7/0xad0Feb 21 17:12:51 host kernel: [<ffffffff8103cd18>] ? pvclock_clocksource_read+0x58/0xd0Feb 21 17:12:51 host kernel: [<ffffffff8100f951>] ? xen_clocksource_read+0x21/0x30Feb 21 17:12:51 host kernel: [<ffffffff8100fa39>] ? xen_clocksource_get_cycles+0x9/0x10Feb 21 17:12:51 host kernel: [<ffffffff8100c949>] ? __raw_callee_save_xen_pmd_val+0x11/0x1eFeb 21 17:12:51 host kernel: [<ffffffff8113676d>] handle_mm_fault+0x1ed/0x2b0Feb 21 17:12:51 host kernel: [<ffffffff814ce503>] do_page_fault+0x123/0x3a0Feb 21 17:12:51 host kernel: [<ffffffff814cbf75>] page_fault+0x25/0x30Feb 21 17:12:51 host kernel: Mem-Info:Feb 21 17:12:51 host kernel: Node 0 DMA per-cpu:Feb 21 17:12:51 host kernel: cpu    0: hi:    0,btch:   1 usd:   0Feb 21 17:12:51 host kernel: cpu    1: hi:    0,btch:   1 usd:   0Feb 21 17:12:51 host kernel: cpu    2: hi:    0,btch:   1 usd:   0Feb 21 17:12:51 host kernel: cpu    3: hi:    0,btch:   1 usd:   0Feb 21 17:12:51 host kernel: Node 0 DMA32 per-cpu:Feb 21 17:12:51 host kernel: cpu    0: hi:  186,btch:  31 usd:  47Feb 21 17:12:51 host kernel: cpu    1: hi:  186,btch:  31 usd:   0Feb 21 17:12:51 host kernel: cpu    2: hi:  186,btch:  31 usd:   0Feb 21 17:12:51 host kernel: cpu    3: hi:  186,btch:  31 usd: 174Feb 21 17:12:51 host kernel: active_anon:74201 inactive_anon:74249 isolated_anon:0Feb 21 17:12:51 host kernel: active_file:120 inactive_file:276 isolated_file:0Feb 21 17:12:51 host kernel: unevictable:0 dirty:0 writeback:2 unstable:0Feb 21 17:12:51 host kernel: free:1600 slab_reclaimable:2713 slab_unreclaimable:19139Feb 21 17:12:51 host kernel: mapped:177 shmem:84 pagetables:12939 bounce:0Feb 21 17:12:51 host kernel: Node 0 DMA free:3024kB min:64kB low:80kB high:96kB active_anon:5384kB inactive_anon:5460kB active_file:36kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14368kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:116kB kernel_stack:32kB pagetables:140kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8 all_unreclaimable? noFeb 21 17:12:51 host kernel: lowmem_reserve[]: 0 741 741 741Feb 21 17:12:51 host kernel: Node 0 DMA32 free:3376kB min:3448kB low:4308kB high:5172kB active_anon:291420kB inactive_anon:291536kB active_file:444kB inactive_file:1092kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:759520kB mlocked:0kB dirty:0kB writeback:8kB mapped:692kB shmem:336kB slab_reclaimable:10836kB slab_unreclaimable:76440kB kernel_stack:2520kB pagetables:51616kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2560 all_unreclaimable? yesFeb 21 17:12:51 host kernel: lowmem_reserve[]: 0 0 0 0Feb 21 17:12:51 host kernel: Node 0 DMA: 5*4kB 4*8kB 2*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3028kBFeb 21 17:12:51 host kernel: Node 0 DMA32: 191*4kB 63*8kB 9*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3396kBFeb 21 17:12:51 host kernel: 4685 total pagecache pagesFeb 21 17:12:51 host kernel: 4131 pages in swap cacheFeb 21 17:12:51 host kernel: Swap cache stats: add 166650,delete 162519,find 1524867/1527901Feb 21 17:12:51 host kernel: Free swap  = 0kBFeb 21 17:12:51 host kernel: Total swap = 523256kBFeb 21 17:12:51 host kernel: 196607 pages RAMFeb 21 17:12:51 host kernel: 6737 pages reservedFeb 21 17:12:51 host kernel: 33612 pages sharedFeb 21 17:12:51 host kernel: 180803 pages non-sharedFeb 21 17:12:51 host kernel: Out of memory: kill process 2053 (MysqLd_safe) score 891049 or a childFeb 21 17:12:51 host kernel: Killed process 2266 (MysqLd) vsz:1540232kB,anon-RSS:4692kB,file-RSS:128kB

解决方法 我只是看了oom日志转储,我质疑该图的准确性.注意第一个’Node 0 DMA32’行.它说免费：3376kB,最小：3448kB,低：4308kB.每当自由值低于低值时,kswapd应该开始交换事物,直到该值重新高于高值.每当自由下降到min以下时,系统基本上会冻结,直到内核将其恢复到最小值以上.该消息还表明交换完全被用于称为Free swap = 0kB的地方.
所以基本上kswapd被触发了,但是swap已经满了,所以它无法做任何事情,并且pages_free值仍然低于pages_min值,所以唯一的选择是开始杀死东西,直到它可以让pages_free备份.
你绝对没有记忆.

@L_301_2@对其工作方式有一个非常好的解释.请参阅底部的“实施”部分.

总结

以上是内存溢出为你收集整理的linux – OOM杀手用大量(？)的空闲内存杀死东西全部内容，希望文章能够帮你解决linux – OOM杀手用大量(？)的空闲内存杀死东西所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。