使用mnesia在节点间共享内存

概述

有很多场景需要在一系列节点间共享内存数据. 如, 有一系列水平对等的网关, 可以在任意网关节点上拿到所有网关的特定内存信息.
一般的做法是使用zookeeper, etcd等提供了分布式一致性保证的服务.
使用zookeeper, etcd做节点间的数据同步当然没有问题. 但是:

• erlang内置数据类型需要额外的序列化/反序列化处理. 如pid.
• 不想引入一个复杂系统.

我最终使用了 gossip protocol 共享数据. 因为它非常简单可控, 能解决上面的痛点. 也可以实现节点间的最终一致性.
erlang原生的mnesia看起来也很适合上述场景. 在最初做选型的时候, 对mnesia的实现没有透彻了解, 这里探讨一下使用mnesia的可行性, 以及mnesia是如何实现的:

• 分布式事务是如何实现的?
• 有新节点加入时, 数据是如何同步的?
• 有没有主节点概念? 网络分区后如何恢复?
• 提供什么级别的一致性保证?

使用mnesia在节点间共享数据

~/platform/launcher(master*) » iex --sname t1
Erlang/OTP 21 [erts-10.3.5.6] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(t1@ubuntu)1> alias :mnesia, as: Mnesia
:mnesia
iex(t1@ubuntu)2> Mnesia.start()
:ok
iex(t1@ubuntu)3> Mnesia.create_table(Person, [attributes: [:id, :name, :job]])                
{:atomic, :ok}
iex(t1@ubuntu)4> Mnesia.dirty_write({Person, 1, "Seymour Skinner", "Principal"})
:ok
iex(t1@ubuntu)5> Mnesia.dirty_read({Person, 1})
[{Person, 1, "Seymour Skinner", "Principal"}]
iex(t1@ubuntu)6> Mnesia.table_info(Person, :all)
[
  access_mode: :read_write,
  active_replicas: [:t1@ubuntu],
  all_nodes: [:t1@ubuntu],
  arity: 4,
  attributes: [:id, :name, :job],
  checkpoints: [],
  commit_work: [],
  cookie: {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu},
  cstruct: {:cstruct, Person, :set, [:t1@ubuntu], [], [], [], 0, :read_write,
   false, [], [], false, Person, [:id, :name, :job], [], [], [],
   {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu}, {{2, 0}, []}},
  disc_copies: [],
  disc_only_copies: [],
  external_copies: [],
  frag_properties: [],
  index: [],
  index_info: {:index, :set, []},
  load_by_force: false,
  load_node: :t1@ubuntu,
  load_order: 0,
  load_reason: {:dumper, :create_table},
  local_content: false,
  majority: false,
  master_nodes: [],
  memory: 321,
  ram_copies: [:t1@ubuntu],
  record_name: Person,
  record_validation: {Person, 4, :set},
  size: 1,
  snmp: [],
  storage_properties: [],
  storage_type: :ram_copies,
  subscribers: [],
  type: :set,
  user_properties: [],
  version: {{2, 0}, []},
  where_to_commit: [t1@ubuntu: :ram_copies],
  where_to_read: :t1@ubuntu, 
  where_to_wlock: {[:t1@ubuntu], false},
  where_to_write: [:t1@ubuntu],
  wild_pattern: {Person, :_, :_, :_}
]

启动t2

~/platform/launcher(master*) » iex --sname t2
Erlang/OTP 21 [erts-10.3.5.6] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(t2@ubuntu)1> alias :mnesia, as: Mnesia
:mnesia
iex(t2@ubuntu)2> Mnesia.start()
:ok

copy table至t2并在t2验证.

iex(t1@ubuntu)9> Mnesia.add_table_copy(Person, :t2@ubuntu, :ram_copies)
{:atomic, :ok}
iex(t1@ubuntu)10> Mnesia.change_config(:extra_db_nodes, [:t2@ubuntu]) 
iex(t2@ubuntu)4> Mnesia.dirty_read({Person, 1})
[{Person, 1, "Seymour Skinner", "Principal"}]

t2的写入也在t1可读

iex(t2@ubuntu)5> Mnesia.dirty_write({Person, 2, "Homer Simpson", "Safety Inspector"})
:ok
iex(t1@ubuntu)11> Mnesia.dirty_read({Person, 2})
[{Person, 2, "Homer Simpson", "Safety Inspector"}]

若t2重启, 需要t1重新change_config, t2才会重新从t1同步数据.

~/platform/launcher(master*) » iex --sname t2
Erlang/OTP 21 [erts-10.3.5.6] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(t2@ubuntu)1> alias :mnesia, as: Mnesia
:mnesia
iex(t2@ubuntu)2> Mnesia.start()
:ok
iex(t2@ubuntu)3> Mnesia.dirty_read({Person, 1})
** (exit) {:aborted, {:no_exists, [Person, 1]}}
    (mnesia) mnesia.erl:355: :mnesia.abort/1
iex(t1@ubuntu)12> Mnesia.change_config(:extra_db_nodes, [:t2@ubuntu])   
{:ok, [:t2@ubuntu]}
iex(t2@ubuntu)3> Mnesia.dirty_read({Person, 1})
[{Person, 1, "Seymour Skinner", "Principal"}]
iex(t2@ubuntu)4> Mnesia.dirty_read({Person, 2})
[{Person, 2, "Homer Simpson", "Safety Inspector"}]

若有新节点t3, t1/t2执行change_config即可. 并不需要add_table_copy, 不add_table_copy的node, 无法写入. 在关闭t1, t2节点后. 节点t3写入失败. 如果重新启动t1节点, 并在t3 change_config, 可以将schema数据拷贝回t1. 重新可以提交. 但之前的数据全部丢失了.

iex(t3@ubuntu)6> Mnesia.dirty_write({Person, 4, "Person 4", "Safety Inspector"})
** (exit) {:aborted, {:no_exists, Person}}
    (mnesia) mnesia.erl:355: :mnesia.abort/1
    (mnesia) mnesia_tm.erl:1061: :mnesia_tm.dirty/2
iex(t3@ubuntu)6> Mnesia.table_info(Person, :all)                                
[
  access_mode: :read_write,
  active_replicas: [],
  all_nodes: [:t2@ubuntu, :t1@ubuntu],
  arity: 4,
  attributes: [:id, :name, :job],
  checkpoints: [],
  commit_work: [],
  cookie: {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu},
  cstruct: {:cstruct, Person, :set, [:t2@ubuntu, :t1@ubuntu], [], [], [], 0,
   :read_write, false, [], [], false, Person, [:id, :name, :job], [], [], [],
   {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu},
   {{3, 0}, {:t1@ubuntu, {1593, 853838, 540527}}}},
  disc_copies: [],
  disc_only_copies: [],
  external_copies: [],
  frag_properties: [],
  index: [],
  index_info: {:index, :set, []},
  load_by_force: false,
  load_node: :unknown,
  load_order: 0,
  load_reason: :unknown,
  local_content: false,
  majority: false,
  master_nodes: [],
  memory: 0,
  ram_copies: [:t2@ubuntu, :t1@ubuntu],
  record_name: Person,
  record_validation: {Person, 4, :set},
  size: 0,
  snmp: [],
  storage_properties: [],
  storage_type: :unknown,
  subscribers: [],
  type: :set,
  user_properties: [],
  version: {{3, 0}, {:t1@ubuntu, {1593, 853838, 540527}}},
  where_to_commit: [],
  where_to_read: :nowhere,
  where_to_wlock: {[], false},
  where_to_write: [],
  wild_pattern: {Person, :_, :_, :_}
]
iex(t3@ubuntu)7> Mnesia.change_config(:extra_db_nodes, [:t1@ubuntu])   
{:ok, [:t1@ubuntu]}
iex(t3@ubuntu)8> Mnesia.table_info(Person, :all)                       
[
  access_mode: :read_write,
  active_replicas: [:t1@ubuntu],
  all_nodes: [:t2@ubuntu, :t1@ubuntu],
  arity: 4,
  attributes: [:id, :name, :job],
  checkpoints: [],
  commit_work: [],
  cookie: {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu},
  cstruct: {:cstruct, Person, :set, [:t2@ubuntu, :t1@ubuntu], [], [], [], 0,
   :read_write, false, [], [], false, Person, [:id, :name, :job], [], [], [],
   {{1593853684922256987, -576460752303423391, 1}, :t1@ubuntu},
   {{3, 0}, {:t1@ubuntu, {1593, 853838, 540527}}}},
  disc_copies: [],
  disc_only_copies: [],
  external_copies: [],
  frag_properties: [],
  index: [],
  index_info: {:index, :set, []},
  load_by_force: false,
  load_node: :unknown,
  load_order: 0,
  load_reason: :unknown,
  local_content: false,
  majority: false,
  master_nodes: [],
  memory: 0,
  ram_copies: [:t2@ubuntu, :t1@ubuntu],
  record_name: Person,
  record_validation: {Person, 4, :set},
  size: 0,
  snmp: [],
  storage_properties: [],
  storage_type: :unknown,
  subscribers: [],
  type: :set,
  user_properties: [],
  version: {{3, 0}, {:t1@ubuntu, {1593, 853838, 540527}}},
  where_to_commit: [t1@ubuntu: :ram_copies],
  where_to_read: :t1@ubuntu,
  where_to_wlock: {[:t1@ubuntu], false},
  where_to_write: [:t1@ubuntu],
  wild_pattern: {Person, :_, :_, :_}
]

在关闭t1, t3节点后, t2节点仍然可以写入成功.

推测

• mnesia使用多主节点lock and commit.
• 有新节点加入时, 通过change_config主动从特定节点上复制数据.
• 无一致性保证, 有脑裂问题. (写入时没有要求多于半数节点存活)

总结

mnesia 不适合有一致性要求的场景.