Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-13202

Impala workloads can exceed Kudu client's rpc_max_message_size limit

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • Backend
    • None
    • ghx-label-5

    Description

      The way Impala integrates with KRPC is porting the KRPC codes into the Impala code base. Flags and methods of KRPC are defined as GLOBAL in the impalad executable. libkudu_client.so also compiles from the same KRPC codes and have duplicate flags and methods defined as HIDDEN.

      To be specifit, both the impalad executable and libkudu_client.so have the symbol for kudu::rpc::InboundTransfer::ReceiveBuffer() 

      $ readelf -s --wide be/build/latest/service/impalad | grep ReceiveBuffer
 11118: 00000000022f5c88  1936 FUNC    GLOBAL DEFAULT   13 _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
 81380: 00000000022f5c88  1936 FUNC    GLOBAL DEFAULT   13 _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE

$ readelf -s --wide toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so | grep ReceiveBuffer
  1601: 0000000000086e4a   108 FUNC    LOCAL  DEFAULT   12 _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE.cold
 11905: 00000000001fec60  2076 FUNC    LOCAL  HIDDEN    12 _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE

$ c++filt _ZN4kudu3rpc15InboundTransfer13ReceiveBufferEPNS_6SocketEPNS_10faststringE
kudu::rpc::InboundTransfer::ReceiveBuffer(kudu::Socket*, kudu::faststring*)

      KRPC flags like rpc_max_message_size are also defined in both the impalad executable and libkudu_client.so:

      $ readelf -s --wide be/build/latest/service/impalad | grep FLAGS_rpc_max_message_size
 14380: 0000000006006738     8 OBJECT  GLOBAL DEFAULT   30 _ZN5fLI6426FLAGS_rpc_max_message_sizeE
 80396: 0000000006006741     1 OBJECT  GLOBAL DEFAULT   30 _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
 81399: 0000000006006741     1 OBJECT  GLOBAL DEFAULT   30 _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
117873: 0000000006006738     8 OBJECT  GLOBAL DEFAULT   30 _ZN5fLI6426FLAGS_rpc_max_message_sizeE

$ readelf -s --wide toolchain/toolchain-packages-gcc10.4.0/kudu-e742f86f6d/debug/lib/libkudu_client.so | grep FLAGS_rpc_max_message_size
 11882: 00000000008d61e1     1 OBJECT  LOCAL  HIDDEN    27 _ZN3fLB44FLAGS_rpc_max_message_size_enable_validationE
 11906: 00000000008d61d8     8 OBJECT  LOCAL  DEFAULT   27 _ZN5fLI6426FLAGS_rpc_max_message_sizeE

$ c++filt _ZN5fLI6426FLAGS_rpc_max_message_sizeE
fLI64::FLAGS_rpc_max_message_size

      libkudu_client.so uses its own methods and flags. The flags are HIDDEN so can't be modified by Impala codes. E.g. IMPALA-4874 bumps FLAGS_rpc_max_message_size to 2GB in RpcMgr::Init(), but the HIDDEN variable FLAGS_rpc_max_message_size used in libkudu_client.so is still the default value 50MB (52428800). We've seen error messages like this in the master branch:

      I0708 10:23:31.784974  2943 meta_cache.cc:294] c243bda4702a5ab9:0ba93d2400000001] tablet 0c8f3446538449ee9d3df5056afe775e: replica e0e1db54dab74f208e37ea1b975595e5 (127.0.0.1:31202) has failed: Network error: TS failed: RPC frame had a length of 53477464, but we only support messages up to 52428800 bytes long.

      CC joemcdonnell wzhou aserbin 

      Attachments

        1. data.parquet
          2.39 MB
          Quanlong Huang

        Issue Links

          relates to

          Sub-task - The sub-task of the issue IMPALA-4874 Increase maximum KRPC message size

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KUDU-3595 Add a way to set Kudu client's rpc_max_message_size via KuduClientBuilder

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: