lxc和virtualbox和物理机的简单性能测试和对比

说明

测试各种虚拟化系统下的虚拟机性能。

测试使用sysbench。

CPU采用如下指令测试。

sysbench --test=cpu --num-threads=2 --cpu-max-prime=50000 run

文件IO采用如下指令测试。

sysbench --test=fileio --file-total-size=10G prepare
sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

内存采用如下指令测试。

sysbench --test=memory --num-threads=2 --memory-access-mode=seq run
sysbench --test=memory --num-threads=2 --memory-access-mode=rnd run

线程采用如下指令。

sysbench --test=threads --num-threads=2 run
sysbench --test=mutex --num-threads=2 --mutex-locks=1000000 run

裸硬盘测试采用如下指令。

hdparm -tT <dev>

物理机上有三个文件系统，ext4/xfs/btrfs，前两者仅做fileio测试以对比性能。

另外做两个特殊文件系统对比，aufs带复制和aufs无复制。前者在只读层上准备好测试文件，而后进行随机读写测试。其中就附带了文件复制开销。后者在aufs建立后初始化测试文件，因此消除了文件复制开销。

所有测试都是测试数次，取最高者（因为低者可能受到各种干扰）。一般是2-3次。

物理机是一台DELL Intel 64位桌面系统，支持硬件虚拟化，有4G内存。系统采用debian jessie，测试于2014年1月17日-20日执行，内核3.12.6-2 (2013-12-29) x86_64。 虚拟机lxc是使用lxc切分的一台虚拟机，没有做资源限制。

虚拟机vbox是使用virtualbox切分的一台虚拟机，分配了所有CPU，打开了硬件虚拟化，分配了1G内存。

文件系统

ext4

Operations performed: 21311 Read, 14207 Write, 45440 Other = 80958
Total Read 332.98Mb Written 221.98Mb
Total transferred 554.97Mb (1.8499Mb/sec)
 118.39 Requests/sec executed

Test execution summary:
 total time: 300.0044s
 total number of events: 35518
 total time taken by event execution: 168.4761
 per-request statistics:
 min: 0.00ms
 avg: 4.74ms
 max: 118.67ms
 approx. 95 percentile: 12.48ms

Threads fairness:
 events (avg/stddev): 35518.0000/0.00
 execution time (avg/stddev): 168.4761/0.00

xfs

Operations performed: 20789 Read, 13859 Write, 44288 Other = 78936
Total Read 324.83Mb Written 216.55Mb
Total transferred 541.38M (1.8046Mb/sec)
 115.49 Requests/sec executed

Test execution summary:
 total time: 300.0018s
 total number of events: 34648
 total time taken by event execution: 172.0475
 per-request statistics:
 min: 0.00ms
 avg: 4.97ms
 max: 96.11ms
 approx. 95 percentile: 12.30ms

Threads fairness:
 events (avg/stddev): 34648.0000/0.00
 execution time (avg/stddev): 172.0475/0.00

btrfs

Operations performed: 6180 Read, 4120 Write, 13105 Other = 23405 Total
 Read 96.562Mb Written 64.375Mb
 Total transferred 160.94M (549.23Kb/sec)
 34.33 Requests/sec executed

Test execution summary:
 total time: 300.0556s
 total number of events: 10300
 total time taken by event execution: 65.8914
 per-request statistics:
 min: 0.00ms
 avg: 6.40ms
 max: 337.28ms
 approx. 95 percentile: 17.01ms

Threads fairness:
 events (avg/stddev): 10300.0000/0.00
 execution time (avg/stddev): 65.8914/0.00

aufs透明

Operations performed: 5340 Read, 3560 Write, 11279 Other = 20179
Total Read 83.438Mb Written 55.625Mb
Total transferred 139.06M (474.65Kb/sec)
 29.67 Requests/sec executed

Test execution summary:
 total time: 300.0084s
 total number of events: 8900

Test execution summary:
 total time: 300.0084s
 total number of events: 8900
 total time taken by event execution: 32.3634
 per-request statistics:
 min: 0.00ms
 avg: 3.64ms
 max: 1037.04ms
 approx. 95 percentile: 0.02ms

Threads fairness:
 events (avg/stddev): 8900.0000/0.00
 execution time (avg/stddev): 32.3634/0.00

aufs非透明

Operations performed: 20320 Read, 13546 Write, 43264 Other = 77130
Total Read 317.5Mb Written 211.66Mb
Total transferred 529.16Mb (1.7638Mb/sec)
 112.88 Requests/sec executed

Test execution summary:
 total time: 300.0054s
 total number of events: 33866
 total time taken by event execution: 170.7252
 per-request statistics:
 min: 0.00ms
 avg: 5.04ms
 max: 143.86ms
 approx. 95 percentile: 12.62ms

Threads fairness:
 events (avg/stddev): 33866.0000/0.00
 execution time (avg/stddev): 170.7252/0.00

物理机

hdparm

Timing cached reads: 11980 MB in 2.00 seconds = 5992.56 MB/sec
 Timing buffered disk reads: 366 MB in 3.01 seconds = 121.52 MB/sec

cpu

Test execution summary:
 total time: 51.4463s
 total number of events: 10000
 total time taken by event execution: 102.8828
 per-request statistics:
 min: 9.93ms
 avg: 10.29ms
 max: 36.11ms
 approx. 95 percentile: 11.29ms

Threads fairness:
 events (avg/stddev): 5000.0000/30.00
 execution time (avg/stddev): 51.4414/0.00

memory

Operations performed: 104857600 (4545662.48 ops/sec)

102400.00 MB transferred (4439.12 MB/sec)

Test execution summary:
 total time: 23.0676s
 total number of events: 104857600
 total time taken by event execution: 34.1991
 per-request statistics:
 min: 0.00ms
 avg: 0.00ms
 max: 18.05ms
 approx. 95 percentile: 0.00ms

Threads fairness:
 events (avg/stddev): 52428800.0000/69790.00
 execution time (avg/stddev): 17.0995/0.01

Operations performed: 104857600 (5407739.22 ops/sec)

102400.00 MB transferred (5281.00 MB/sec)

Test execution summary:
 total time: 19.3903s
 total number of events: 104857600
 total time taken by event execution: 26.8579
 per-request statistics:
 min: 0.00ms
 avg: 0.00ms
 max: 22.46ms
 approx. 95 percentile: 0.00ms

Threads fairness:
 events (avg/stddev): 52428800.0000/7211.00
 execution time (avg/stddev): 13.4289/0.14

threads

Test execution summary:
 total time: 1.0112s
 total number of events: 10000
 total time taken by event execution: 2.0210
 per-request statistics:
 min: 0.15ms
 avg: 0.20ms
 max: 11.19ms
 approx. 95 percentile: 0.24ms

Threads fairness:
 events (avg/stddev): 5000.0000/3.00
 execution time (avg/stddev): 1.0105/0.00

Test execution summary:
 total time: 0.1665s
 total number of events: 2
 total time taken by event execution: 0.3238
 per-request statistics:
 min: 157.39ms
 avg: 161.90ms
 max: 166.41ms
 approx. 95 percentile: 10000000.00ms

Threads fairness:
 events (avg/stddev): 1.0000/0.00
 execution time (avg/stddev): 0.1619/0.00

nginx

Concurrency Level: 10000
 Time taken for tests: 5.745 seconds
 Complete requests: 100000
 Failed requests: 0
 Write errors: 0
 Total transferred: 172100000 bytes
 HTML transferred: 160000000 bytes
 Requests per second: 17404.93 [\#/sec] (mean)
 Time per request: 574.550 [ms] (mean)
 Time per request: 0.057 [ms] (mean, across all concurrent requests)
 Transfer rate: 29251.85 [Kbytes/sec] received

lxc

cpu

Test execution summary:
 total time: 51.4368s
 total number of events: 10000
 total time taken by event execution: 102.8619
 per-request statistics:
 min: 9.92ms
 avg: 10.29ms
 max: 35.08ms
 approx. 95 percentile: 11.68ms

Threads fairness:
 events (avg/stddev): 5000.0000/5.00
 execution time (avg/stddev): 51.4310/0.00

fileio

Operations performed: 5548 Read, 3698 Write, 11776 Other = 21022
Total Read 86.688Mb Written 57.781Mb
Total transferred 144.47Mb (493.07Kb/sec)
 30.82 Requests/sec executed

Test execution summary:
 total time: 300.0294s
 total number of events: 9246
 total time taken by event execution: 84.4687
 per-request statistics:
 min: 0.01ms
 avg: 9.14ms
 max: 394.13ms
 approx. 95 percentile: 36.20ms

Threads fairness:
 events (avg/stddev): 9246.0000/0.00
 execution time (avg/stddev): 84.4687/0.00

memory

Operations performed: 104857600 (4456398.83 ops/sec)

102400.00 MB transferred (4351.95 MB/sec)

Test execution summary:
 total time: 23.5297s
 total number of events: 104857600
 total time taken by event execution: 34.8417
 per-request statistics:
 min: 0.00ms
 avg: 0.00ms
 max: 20.22ms
 approx. 95 percentile: 0.00ms

Threads fairness:
 events (avg/stddev): 52428800.0000/155952.00
 execution time (avg/stddev): 17.4208/0.06

Operations performed: 104857600 (5327923.43 ops/sec)

102400.00 MB transferred (5203.05 MB/sec)

Test execution summary:
 total time: 19.6808s
 total number of events: 104857600
 total time taken by event execution: 27.3010
 per-request statistics:
 min: 0.00ms
 avg: 0.00ms
 max: 16.48ms
 approx. 95 percentile: 0.00ms

Threads fairness:
 events (avg/stddev): 52428800.0000/297738.00
 execution time (avg/stddev): 13.6505/0.09

threads

Test execution summary:
 total time: 1.2490s
 total number of events: 10000
 total time taken by event execution: 2.4954
 per-request statistics:
 min: 0.21ms
 avg: 0.25ms
 max: 7.39ms
 approx. 95 percentile: 0.28ms

Threads fairness:
 events (avg/stddev): 5000.0000/7.00
 execution time (avg/stddev): 1.2477/0.00

Test execution summary:
 total time: 0.1222s
 total number of events: 2
 total time taken by event execution: 0.2275
 per-request statistics:
 min: 107.53ms
 avg: 113.77ms
 max: 120.02ms
 approx. 95 percentile: 10000000.00ms

Threads fairness:
 events (avg/stddev): 1.0000/0.00
 execution time (avg/stddev): 0.1138/0.01

nginx

Concurrency Level: 10000
 Time taken for tests: 16.976 seconds
 Complete requests: 100000
 Failed requests: 0
 Write errors: 0
 Total transferred: 1551500000 bytes
 HTML transferred: 1539400000 bytes
 Requests per second: 5890.69 [#/sec] (mean)
 Time per request: 1697.594 [ms] (mean)
 Time per request: 0.170 [ms] (mean, across all concurrent requests)
 Transfer rate: 89252.02 [Kbytes/sec] received

vbox

hdparm

Timing cached reads: 10122 MB in 1.99 seconds = 5088.70 MB/sec
 Timing buffered disk reads: 300 MB in 3.00 seconds = 99.87 MB/sec

cpu

Test execution summary:
 total time: 54.0469s
 total number of events: 10000
 total time taken by event execution: 108.0595
 per-request statistics:
 min: 9.03ms
 avg: 10.81ms
 max: 61.39ms
 approx. 95 percentile: 14.87ms

Threads fairness:
 events (avg/stddev): 5000.0000/87.00
 execution time (avg/stddev): 54.0297/0.00

fileio

Operations performed: 68153 Read, 45435 Write, 145280 Other = 258868
Total Read 1.0399Gb Written 709.92Mb
Total transferred 1.7332Gb (5.916Mb/sec)
 378.62 Requests/sec executed

Test execution summary:
 total time: 300.0045s
 total number of events: 113588
 total time taken by event execution: 177.2314
 per-request statistics:
 min: 0.01ms
 avg: 1.56ms
 max: 579.66ms
 approx. 95 percentile: 13.10ms

Threads fairness:
 events (avg/stddev): 113588.0000/0.00
 execution time (avg/stddev): 177.2314/0.00

memory

一直报错，测试不出来。

threads

Test execution summary:
 total time: 16.0377s
 total number of events: 10000
 total time taken by event execution: 32.0421
 per-request statistics:
 min: 1.19ms
 avg: 3.20ms
 max: 38.39ms
 approx. 95 percentile: 5.74ms

Threads fairness:
 events (avg/stddev): 5000.0000/15.00
 execution time (avg/stddev): 16.0210/0.00

Test execution summary:
 total time: 0.3023s
 total number of events: 2
 total time taken by event execution: 0.5990
 per-request statistics:
 min: 297.52ms
 avg: 299.50ms
 max: 301.47ms
 approx. 95 percentile: 10000000.00ms

Threads fairness:
 events (avg/stddev): 1.0000/0.00
 execution time (avg/stddev): 0.2995/0.00

nginx

Concurrency Level: 5000
 Time taken for tests: 41.758 seconds
 Complete requests: 50000
 Failed requests: 0
 Write errors: 0
 Keep-Alive requests: 0
 Total transferred: 890350000 bytes
 HTML transferred: 884250000 bytes
 Requests per second: 1197.38 [#/sec] (mean)
 Time per request: 4175.799 [ms] (mean)
 Time per request: 0.835 [ms] (mean, across all concurrent requests)
 Transfer rate: 20821.94 [Kbytes/sec] received

vbox on lvm

在物理机上开辟一个lvm卷，然后用vmdk引用物理卷的功能挂到vbox上使用，格式化为ext4文件格式。

hdparm

Timing cached reads: 10034 MB in 1.99 seconds = 5044.30 MB/sec
 Timing buffered disk reads: 270 MB in 3.01 seconds = 89.73 MB/sec

fileio

Operations performed: 22765 Read, 15176 Write, 48512 Other = 86453
Total Read 355.7Mb Written 237.12Mb Total transferred 592.83Mb (1.976Mb/sec)
 126.47 Requests/sec executed

Test execution summary:
 total time: 300.0080s
 total number of events: 37941
 total time taken by event execution: 286.5402
 per-request statistics:
 min: 0.01ms
 avg: 7.55ms
 max: 336.67ms
 approx. 95 percentile: 20.49ms

Threads fairness:
 events (avg/stddev): 37941.0000/0.00
 execution time (avg/stddev): 286.5402/0.00

vbox on vdi

基本同vbox on lvm，不过使用vdi作为存储。和vbox测试相比，内部系统换为主系统同样的debian。

hdparm

Timing cached reads: 9152 MB in 1.99 seconds = 4598.80 MB/sec
 Timing buffered disk reads: 352 MB in 3.00 seconds = 117.16 MB/sec

fileio

Operations performed: 151020 Read, 100680 Write, 322060 Other = 573760
Total Read 2.3044Gb Written 1.5363Gb
Total transferred 3.8406Gb (13.109Mb/sec)
 839.00 Requests/sec executed

Test execution summary:
 total time: 300.0011s
 total number of events: 251700
 total time taken by event execution: 60.6296
 per-request statistics:
 min: 0.01ms
 avg: 0.24ms
 max: 106.94ms
 approx. 95 percentile: 0.26ms

Threads fairness:
 events (avg/stddev): 251700.0000/0.00
 execution time (avg/stddev): 60.6296/0.00

分析

计算

从CPU上说，vbox的消耗大约是5%，而lxc的消耗基本是0%。

内存测试上，vbox无法测量。lxc的性能和物理机十分相近，两者的差异在1.5%-2%之间。

在threads测试和mutex测试上，vbox显示出远远低于lxc的性能。这些基本都是内核陷入类的事务，也是预期vbox会发生性能下降的地方。

IO

从硬件设备IO上来分析，lxc和主机是共用一套物理设备的。vbox的裸设备IO比物理机低了15-18%。但是文件系统IO则表现出完全相反的景象。

lxc底层使用的是btrfs，因此性能和物理机上的btrfs性能十分相近，误差在10%以内。这一性能比物理机上的ext4/xfs低了70%以上。这个表现出btrfs的性能和ext4/xfs性能的差异(注：怎么会差这么多？)。

如果使用aufs的话，则要视复制特性而定。如果引发复制，性能会跌到和btrfs相近。而不引发复制的话，则和aufs下面的文件系统相近(误差在5%以内)。
从上面可以看出，对于lxc这种小消耗的虚拟方案，与其在意虚拟机的性能消耗，不如更在意文件系统的性能消耗。但是比较倒霉的是，lxc是不能在虚拟机内自行配置文件系统的，需要从主机内分配挂载。

• ext4：适用于大部分情况，iops很高，小文件下吞吐量很不错。大部分虚拟化场景都是小系统简单服务的时候适合。
• xfs：适用于大设备内部的大虚拟机。每个虚机的服务复杂，或者是有大文件。
• btrfs：这个东西性能很低，但是写时复制的能力对快速clone很重要。适合用于产生一堆临时生成的环境，用于程序员测试或者测试工程师搭环境，测试完了就删除的。
• aufs：和btrfs情况差不多，快速clone不错。不过安全起见，clone后原image不可以启动实例或者修改image。在没有COW写入时性能很高，这点比btrfs好。而在COW时瞬时开销很高。

---

by shell909090
under CC-BY-SA 3.0