Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49489

'TTransportException: MaxMessageSize reached' occurs if get a table with a large number of partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • SQL
    • None

    Description

      org.apache.thrift.transport.TTransportException: MaxMessageSize reached
      at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) 
      at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) 
      at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) 
      at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) 
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) 
      at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63) 
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) 
      at org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329) 
      at org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273) 
      at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461) 
      at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454) 
      at org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388) 
      at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269) 
      at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248) 
      at org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110) 
      at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270) 
      at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205) 
      at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) 
      ...
      

      Attachments

        1. Internal fix.png
          452 kB
          Yuming Wang

        Issue Links

          Activity

            People

              Unassigned Unassigned
              yumwang Yuming Wang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: