Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4605

HiveProducer write metrics less when one host running some kylin servers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v3.0.0-alpha, v3.0.0-beta
    • None
    • Metrics
    • None

    Description

      Fails to write metrics to file becase file lease:

      hdfs://xxxx/kday_date=2020-01-14/bigdata-kylin-shuyi3-00.gz01.diditaxi.com-part-0000 due to org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to APPEND_FILE /user/prod_kylin/bigdata_kylin/hive/bigdata_kylin/hive_metrics_query_prod3/kday_date=2020-01-14/bigdata-kylin-shuyi3-00.gz01.diditaxi.com-part-0000 for DFSClient_NONMAPREDUCE_-1428991477_29 on 100.69.76.32 because this file lease is currently owned by DFSClient_NONMAPREDUCE_-312505276_29 on 100.69.76.32
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2700)
      at org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:118)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2735)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:842)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:493)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
      at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:886)
      at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:828)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1903)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2717)

      kylin-4026已解决这个问题。启动时加了个时间戳,然后outputstream一直不关,写一天,代替append。
      由于同一台机器的进程不可能同毫秒初始化HiveProducer,所以必然不会冲突

      Attachments

        Issue Links

          Activity

            People

              xiaoge chuxiao
              xiaoge chuxiao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: