kubernetes pod java oom

Question

三台机器 node1 node2 node3
同样的resources限制启动java应用

java -Xrs -server -verbosegc -Xmx700m -Xms200m -XX:PermSize=64m -XX:MaxPermSize=128m -jar xxx.jar
###########
  limits:
    cpu: 1
    memory: 1200Mi
  requests:
    cpu: 0.5
    memory: 1200Mi

只有node3能够正常启动pod
node1 node2 启动几秒钟后就会提示 oomkilled

dmesg日志

[1296839.655308] Task in /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/41a1d0ed6c78e572cc9a91224a0995449196c4a3ab12554dbd36e1d851813215 killedas a result of limit of /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/41a1d0ed6c78e572cc9a91224a0995449196c4a3ab12554dbd36e1d851813215
[1296839.655310] memory: usage 716800kB, limit 716800kB, failcnt 5188
[1296839.655311] memory+swap: usage 716800kB, limit 716800kB, failcnt 0
[1296839.655312] kmem: usage 2020kB, limit 9007199254740988kB, failcnt 0
[1296839.655313] Memory cgroup stats for /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/41a1d0ed6c78e572cc9a91224a0995449196c4a3ab12554dbd36e1d851813215: cache:0KB rss:714780KB rss_huge:661504KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:714764KB inactive_file:0KB active_file:0KB unevictable:0KB
[1296839.655322] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[1296839.655380] [30340]     0 30340  1127294   181455     421        0           981 java
[1296839.655384] Memory cgroup out of memory: Kill process 30381 (java) score 1994 or sacrifice child
[1296839.656556] Killed process 30340 (java), UID 0, total-vm:4509176kB, anon-rss:714296kB, file-rss:11524kB, shmem-rss:0kB
[1296844.150872] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=981
[1296844.150876] java cpuset=32bbde2479ce6841aca922f3cf99756ac2fcc9d9b14597afa45b4c12fc450329 mems_allowed=0
[1296844.150878] CPU: 3 PID: 30532 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
[1296844.150880] Hardware name: ASUS All Series/B85M-E R2.0, BIOS 2106 02/13/2015
[1296844.150881] Call Trace:
[1296844.150887]  [<ffffffffb1d792c2>] dump_stack+0x19/0x1b
[1296844.150890]  [<ffffffffb1d73c64>] dump_header+0x90/0x229
[1296844.150893]  [<ffffffffb1896f4e>] ? ep_poll_callback+0xee/0x210
[1296844.150897]  [<ffffffffb17bfd74>] oom_kill_process+0x254/0x3e0
[1296844.150899]  [<ffffffffb17314d1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[1296844.150902]  [<ffffffffb183c666>] mem_cgroup_oom_synchronize+0x546/0x570
[1296844.150904]  [<ffffffffb183bae0>] ? mem_cgroup_charge_common+0xc0/0xc0
[1296844.150906]  [<ffffffffb17c0614>] pagefault_out_of_memory+0x14/0x90
[1296844.150908]  [<ffffffffb1d72212>] mm_fault_error+0x6a/0x157
[1296844.150911]  [<ffffffffb1d878d1>] __do_page_fault+0x491/0x500
[1296844.150914]  [<ffffffffb1d87975>] do_page_fault+0x35/0x90
[1296844.150916]  [<ffffffffb1d83778>] page_fault+0x28/0x30
[1296844.150918] Task in /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/32bbde2479ce6841aca922f3cf99756ac2fcc9d9b14597afa45b4c12fc450329 killedas a result of limit of /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/32bbde2479ce6841aca922f3cf99756ac2fcc9d9b14597afa45b4c12fc450329
[1296844.150920] memory: usage 716800kB, limit 716800kB, failcnt 5324
[1296844.150921] memory+swap: usage 716800kB, limit 716800kB, failcnt 0
[1296844.150922] kmem: usage 1896kB, limit 9007199254740988kB, failcnt 0
[1296844.150923] Memory cgroup stats for /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/32bbde2479ce6841aca922f3cf99756ac2fcc9d9b14597afa45b4c12fc450329: cache:0KB rss:714904KB rss_huge:661504KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:714888KB inactive_file:0KB active_file:0KB unevictable:0KB
[1296844.150932] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[1296844.150988] [30507]     0 30507  1127294   181439     420        0           981 java
[1296844.150990] Memory cgroup out of memory: Kill process 30546 (java) score 1994 or sacrifice child
[1296844.152206] Killed process 30507 (java), UID 0, total-vm:4509176kB, anon-rss:714232kB, file-rss:11524kB, shmem-rss:0kB
[1296863.347355] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=981
[1296863.347366] java cpuset=171e8c0d560cae55cfba5917af3e2fbfc610694417a58af77376bf2842249264 mems_allowed=0
[1296863.347368] CPU: 1 PID: 30850 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
[1296863.347370] Hardware name: ASUS All Series/B85M-E R2.0, BIOS 2106 02/13/2015
[1296863.347371] Call Trace:
[1296863.347376]  [<ffffffffb1d792c2>] dump_stack+0x19/0x1b
[1296863.347380]  [<ffffffffb1d73c64>] dump_header+0x90/0x229
[1296863.347383]  [<ffffffffb1896f4e>] ? ep_poll_callback+0xee/0x210
[1296863.347387]  [<ffffffffb17bfd74>] oom_kill_process+0x254/0x3e0
[1296863.347389]  [<ffffffffb17314d1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[1296863.347392]  [<ffffffffb183c666>] mem_cgroup_oom_synchronize+0x546/0x570
[1296863.347394]  [<ffffffffb183bae0>] ? mem_cgroup_charge_common+0xc0/0xc0
[1296863.347396]  [<ffffffffb17c0614>] pagefault_out_of_memory+0x14/0x90
[1296863.347399]  [<ffffffffb1d72212>] mm_fault_error+0x6a/0x157
[1296863.347402]  [<ffffffffb1d878d1>] __do_page_fault+0x491/0x500
[1296863.347404]  [<ffffffffb1d87975>] do_page_fault+0x35/0x90
[1296863.347406]  [<ffffffffb1d83778>] page_fault+0x28/0x30
[1296863.347408] Task in /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/171e8c0d560cae55cfba5917af3e2fbfc610694417a58af77376bf2842249264 killedas a result of limit of /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/171e8c0d560cae55cfba5917af3e2fbfc610694417a58af77376bf2842249264
[1296863.347410] memory: usage 716800kB, limit 716800kB, failcnt 3684
[1296863.347411] memory+swap: usage 716800kB, limit 716800kB, failcnt 0
[1296863.347413] kmem: usage 2060kB, limit 9007199254740988kB, failcnt 0
[1296863.347413] Memory cgroup stats for /kubepods/burstable/pode10af940-fa16-11e9-a15a-40167e24734d/171e8c0d560cae55cfba5917af3e2fbfc610694417a58af77376bf2842249264: cache:0KB rss:714740KB rss_huge:661504KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:714712KB inactive_file:0KB active_file:0KB unevictable:0KB
[1296863.347422] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[1296863.347474] [30824]     0 30824  1127294   181410     420        0           981 java
[1296863.347477] Memory cgroup out of memory: Kill process 30864 (java) score 1994 or sacrifice child
[1296863.348691] Killed process 30824 (java), UID 0, total-vm:4509176kB, anon-rss:714136kB, file-rss:11504kB, shmem-rss:0kB

实在不明白 为何只有node3不会报oomkilled 每台机器的资源都是充足的

---

更新下问题
我取消了容器内存限制 进入pod里面top查看java实际占用了3.5g

-xmx1000m 参数似乎没有生效

node3能够正常启动是因为里面的java老老实实的只用了1000m内的内存
问题现在变成了 同样的启动参数 只有一台遵循了 1000m的限制