hive并发多版本错误

起因

最近公司增加了多台hive客户端来访问hive元数据，半夜的时候hive报错出现如下信息：

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
    ... 12 more
Caused by: MetaException(message:Metastore contains multiple versions)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
    at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
    ... 17 more

如报错提示，hive metastore出现了多个版本，去hive的元数据库查看表VERSION，发现多了一条记录

select * from VERSION;

1    1.1.0    Set by MetaStore hadoop@10.252.97.244
2    1.1.0    Set by MetaStore hadoop@10.252.97.244 #这条是多出的

解决过程

查资料

google了一些资料，网上也有人提过，如HIVE-9543，网上大家说的解决方法有如下

设置datanucleus.autoCreateSchema=false
此配置官网介绍：
    Default Value: true
    Added In: Hive 0.7.0
    Removed In: Hive 2.0.0 with HIVE-6113, replaced by datanucleus.schema.autoCreateAll
    Creates necessary schema on a startup if one does not exist. Set this to false, after creating it once.
#意思就是这个参数再hive元数据初始化的时候用到，之后就可以设置为false禁用

设置此参数为false后，继续观察，错误还是会再次出现

查看日志及报错

#查看hive运行日志发现多版本出现的时候，有如下日志
Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version
#意思是说在metastore中未找到版本信息，hive.metastore.schema.verification未禁用，因此记录下版本信息，也就是往版本表中插入一条记录
#接着结合之前的报错，看看报错类如下：
Caused by: MetaException(message:Metastore contains multiple versions)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)

查看源代码

通过查看源代码发现，相关代码如下

##同步方法checkSchema()
private synchronized void checkSchema() throws MetaException {
    // recheck if it got verified by another thread while we were waiting
    if (isSchemaVerified.get()) {
      return;
    }

    //获取hive配置，也就是hive.metastore.schema.verification的值
    boolean strictValidation =
      HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
    // read the schema version stored in metastore db
    //读取metastore的版本信息
    String schemaVer = getMetaStoreSchemaVersion();
    if (schemaVer == null) {
      //如果版本信息未找到，这个时候strictValidation为true的时候直接抛出异常
      if (strictValidation) {
        throw new MetaException("Version information not found in metastore. ");
      } else {
        //否则，调用方法插入版本信息，也就是之前日志所说的
        LOG.warn("Version information not found in metastore. "
            + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
            " is not enabled so recording the schema version " +
            MetaStoreSchemaInfo.getHiveSchemaVersion());
        setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
          "Set by MetaStore " + USER + "@" + HOSTNAME);
      }
    }
    
    
##setMetaStoreSchemaVersion方法如下
public void setMetaStoreSchemaVersion(String schemaVersion, String comment) throws MetaException {
    MVersionTable mSchemaVer;
    boolean commited = false;
    //此参数控制了记录version信息
    boolean recordVersion =
      HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION_RECORD_VERSION);
    //参数为false，则返回，不记录版本信息，否则将插入版本信息
    if (!recordVersion) {
      LOG.warn("setMetaStoreSchemaVersion called but recording version is disabled: " +
        "version = " + schemaVersion + ", comment = " + comment);
      return;
    }

    try {
      mSchemaVer = getMSchemaVersion();
    } catch (NoSuchObjectException e) {
      // if the version doesn't exist, then create it
      mSchemaVer = new MVersionTable();
    }
    
##查看HiveConf中METASTORE_SCHEMA_VERIFICATION_RECORD_VERSION可知hive.metastore.schema.verification.record.version默认为true,则允许记录版本信息    

解决方案

通过以上的源码查看，解决方案已经出来了，其实方法有几种，我选取的做法是将hive.metastore.schema.verification.record.version设置为fals
当然你也可以关闭版本校验

遗留问题

1. 看网上说的hive多版本问题似乎是并发、网络引起的，源代码中为什么没有获取到metastore的版本schema信息，这是一个问题，还有待源码探究
2. 几个参数都可以起到在代码流程中阻断记录版本信息的操作，哪种是无风险的，还有待深究

感受

1. 发现问题第一时间上谷歌查看似乎是我现在的习惯，总想着网上有现成的解决方案，但是有些东西拿过来还是自己要思考一下
2. 大数据组件繁多，对于源码的探究永无止境，不能只停留于表面，要花时间去一点点积累