总之网站上说得很好啦:http://msgpack.org/,又快,生成的文件又小等等,总之就是很牛逼的救世主咯

因为忍受不了 pickle 的速度,所以尝试切换一下。

import pickle
import msgpack
import gc

a = list(range(10000*10000))

def dump1():
    with open('1', 'wb') as f:
        pickle.dump(a, f)

def dump2():
    with open('2', 'wb') as f:
        msgpack.dump(a, f)

def dump3():
    gc.disable()
    with open('2', 'wb') as f:
        msgpack.dump(a, f)
    gc.enable()

然后测速,有 dump3 的原因是官方网站说关闭gc有奇效:

CPython's GC starts when growing allocated object. This means unpacking may cause useless GC. You can use gc.disable() when unpacking large message.

恩,先看测试结果(测试了10次):

dump1: 1 loop, best of 3: 3.98 s per loop
dump2: 1 loop, best of 10: 5.22 s per loop
dump3: 1 loop, best of 10: 5.13 s per loop

恩,彻底打脸了。关闭gc也没有奇效。

然后看dump出来的文件 (=,=b):

-rw-r--r--  1 chendx  staff   477M  8 11 18:43 1
-rw-r--r--  1 chendx  staff   477M  8 11 18:49 2

恩,浪费我的时间……


然后不死心

def load1():
    with open('1', 'rb') as f:
        pickle.load(f)

def load2():
    with open('2', 'rb') as f:
        msgpack.load(f)

结果(还是10次)

load1: 1 loop, best of 10: 6.78 s per loop
load2: 1 loop, best of 10: 5.91 s per loop

恩,真是鸡肋啊


winterdawn
108 声望1 粉丝