Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3824

Use WriteToBigQuery in Python mobile gaming examples.

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.5.0
    • examples-python
    • None

    Description

      python -m apache_beam.examples.complete.game.hourly_team_score --project=$PROJECT --dataset=beam_release_2_4_0 --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv

      The pipeline fails with:

      INFO:root:finish <DoOperation WriteTeamScoreSums/WriteToBigQuery output_tags=['out'], receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]>
      Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
      "_main_", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
      exec code in run_globals
      File "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", line 276, in <
      module>
      run()
      File "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", line 270, in r
      un
      write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 389, in _exit_
      self.run().wait_until_finish()
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 369, in run
      self.to_runner_api(), self.runner, self._options).run(False)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 382, in run
      return self.runner.run_pipeline(self)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 129, in run_pip
      eline
      return runner.run_pipeline(pipeline)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 215, in ru
      n_pipeline
      return self.run_via_runner_api(pipeline.to_runner_api())
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 218, in ru
      n_via_runner_api
      return self.run_stages(*self.create_stages(pipeline_proto))
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 837, in ru
      n_stages
      pcoll_buffers, safe_coders).process_bundle.metrics
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 938, in ru
      n_stage
      self._progress_frequency).process_bundle(data_input, data_output)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1110, in p
      rocess_bundle
      result_future = self._controller.control_handler.push(process_bundle)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1003, in p
      ush
      response = self.worker.do_instruction(request)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 185, in do_instruc
      tion
      request.instruction_id)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 202, in process_bu
      ndle
      processor.process_bundle(instruction_id)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 298, in proc
      ess_bundle
      op.finish()
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py", line 389, in finish
      self.dofn_runner.finish()
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 517, in finish
      self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 508, in _invoke_bundle_method
      self._reraise_augmented(exn)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 537, in _reraise_augmented
      six.raise_from(new_exn, original_traceback)
      File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/six.py", line 737, in raise_from
      raise value
      RuntimeError: Could not successfully insert rows to BigQuery table [google.com:clouddfe:beam_release_2_4_0.leader_board]. Errors: [<InsertEr
      rorsValueListEntry
      errors: [<ErrorProto
      debugInfo: u''
      location: u'processing_time'
      message: u'no such field.'
      reason: u'invalid'>]
      index: 0>, <InsertErrorsValueListEntry
      errors: [<ErrorProto
      debugInfo: u''
      location: u'processing_time'
      message: u'no such field.'
      reason: u'invalid'>]
      index: 1>, <InsertErrorsValueListEntry
      errors: [<ErrorProto
      debugInfo: u''
      location: u'processing_time'
      message: u'no such field.'
      reason: u'invalid'>]
      index: 2>, <InsertErrorsValueListEntry
      errors: [<ErrorProto
      debugInfo: u''
      location: u'processing_time'
      message: u'no such field.'
      reason: u'invalid'>]
      index: 3>, <InsertErrorsValueListEntry
      errors: [<ErrorProto
      debugInfo: u''
      location: u'processing_time'
      message: u'no such field.'
      reason: u'invalid'>]
      ...

      Attachments

        Activity

          People

            altay Ahmet Altay
            tvalentyn Valentyn Tymofieiev
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h