Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9453

S3 build failed with many strange symptoms

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Won't Fix
    • Impala 3.4.0
    • Not Applicable
    • None

    Description

      There were a lot of incorrect results:

      uery_test/test_mt_dop.py:49: in test_mt_dop     self.run_test_case('QueryTest/mt-dop', new_vector) common/impala_test_suite.py:690: in run_test_case     self.__verify_results_and_errors(vector, test_section, result, use_db) common/impala_test_suite.py:523: in __verify_results_and_errors     replace_filenames_with_placeholder) common/test_result_verifier.py:456: in verify_raw_results     VERIFIER_MAP[verifier](expected, actual) common/test_result_verifier.py:278: in verify_query_result_is_equal     assert expected_results == actual_results E   assert Comparing QueryTestResults (expected vs actual): E     7300 != 6990
      
      Stacktrace
      
      query_test/test_mt_dop.py:49: in test_mt_dop
          self.run_test_case('QueryTest/mt-dop', new_vector)
      common/impala_test_suite.py:690: in run_test_case
          self.__verify_results_and_errors(vector, test_section, result, use_db)
      common/impala_test_suite.py:523: in __verify_results_and_errors
          replace_filenames_with_placeholder)
      common/test_result_verifier.py:456: in verify_raw_results
          VERIFIER_MAP[verifier](expected, actual)
      common/test_result_verifier.py:278: in verify_query_result_is_equal
          assert expected_results == actual_results
      E   assert Comparing QueryTestResults (expected vs actual):
      E     7300 != 6990
      
      Standard Error
      
      ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
      7300 != 6990
      
      

      The impalads eventually crashed:

      F0302 00:50:55.607841   483 parquet-page-reader.cc:67] e24eb0839fa75423:8ac0bf7300000002] Check failed: col_end < file_desc.file_length (7010 vs. 7010) 
      *** Check failure stack trace: ***
          @          0x4f7277c  google::LogMessage::Fail()
          @          0x4f74021  google::LogMessage::SendToLog()
          @          0x4f72156  google::LogMessage::Flush()
          @          0x4f7571d  google::LogMessageFatal::~LogMessageFatal()
          @          0x2e3a520  impala::ParquetPageReader::InitColumnChunk()
          @          0x2e37dee  impala::ParquetColumnChunkReader::InitColumnChunk()
          @          0x2cd8000  impala::BaseScalarColumnReader::Reset()
          @          0x2c91239  impala::HdfsParquetScanner::InitScalarColumns()
          @          0x2c8775a  impala::HdfsParquetScanner::NextRowGroup()
          @          0x2c85c2a  impala::HdfsParquetScanner::GetNextInternal()
          @          0x2c8403e  impala::HdfsParquetScanner::ProcessSplit()
          @          0x28b826b  impala::HdfsScanNode::ProcessSplit()
          @          0x28b7440  impala::HdfsScanNode::ScannerThread()
          @          0x28b679d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x28b8d91  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x20aa1e9  boost::function0<>::operator()()
          @          0x266a84a  impala::Thread::SuperviseThread()
          @          0x2672ace  boost::_bi::list5<>::operator()<>()
          @          0x26729f2  boost::_bi::bind_t<>::operator()()
          @          0x26729b5  boost::detail::thread_data<>::run()
          @          0x3e98b19  thread_proxy
          @     0x7f8c2ccefe24  start_thread
          @     0x7f8c2985b34c  __clone
      
      F0302 00:50:41.466794 32643 parquet-page-reader.cc:67] dd48a46583bea9c8:e3be641000000002] Check failed: col_end < file_desc.file_length (7010 vs. 7010) 
      *** Check failure stack trace: ***
          @          0x4f7277c  google::LogMessage::Fail()
          @          0x4f74021  google::LogMessage::SendToLog()
          @          0x4f72156  google::LogMessage::Flush()
          @          0x4f7571d  google::LogMessageFatal::~LogMessageFatal()
          @          0x2e3a520  impala::ParquetPageReader::InitColumnChunk()
          @          0x2e37dee  impala::ParquetColumnChunkReader::InitColumnChunk()
          @          0x2cd8000  impala::BaseScalarColumnReader::Reset()
          @          0x2c91239  impala::HdfsParquetScanner::InitScalarColumns()
          @          0x2c8775a  impala::HdfsParquetScanner::NextRowGroup()
          @          0x2c85c2a  impala::HdfsParquetScanner::GetNextInternal()
          @          0x2c8403e  impala::HdfsParquetScanner::ProcessSplit()
          @          0x28b826b  impala::HdfsScanNode::ProcessSplit()
          @          0x28b7440  impala::HdfsScanNode::ScannerThread()
          @          0x28b679d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x28b8d91  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x20aa1e9  boost::function0<>::operator()()
          @          0x266a84a  impala::Thread::SuperviseThread()
          @          0x2672ace  boost::_bi::list5<>::operator()<>()
          @          0x26729f2  boost::_bi::bind_t<>::operator()()
          @          0x26729b5  boost::detail::thread_data<>::run()
          @          0x3e98b19  thread_proxy
          @     0x7f16a5766e24  start_thread
          @     0x7f16a22d234c  __clone
      Wrote minidump to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/ee_tests/minidumps/impalad/49436f97-da0e-47c9-76b372a5-6b3fb146.dmp
      

      If it makes a differnence, this was using CDP, so that version of the S3 connector.

      Attachments

        1. impalad_node1.ERROR
          33 kB
          Fang-Yu Rao
        2. impalad_node2.ERROR
          23 kB
          Fang-Yu Rao
        3. impalad.ERROR
          529 kB
          Fang-Yu Rao

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: