Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1684

Inconsistent usage of store func.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.8.0
    • impl
    • None
    • A custom StoreFuncInterface used to store data at the reducer.
      (Output of a group )

    • Reviewed

    Description

      Pig seems to be using multiple instances of StoreFuncInterface in the reducer inconsistently.
      Some hadoop api calls are made to one instance and others made to other : which makes state management very inconsistent and is requiring hacks on our part to deal with it.

      The call snippet below should hopefully indicate the issue.
      The format is :

      Instance.toString() method_call.

      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 getOutputFormat()
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 getOutputCommitter
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 setupTask
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 init
      com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 getOutputFormat()
      com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 getRecordWriter
      com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 init
      com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 putNext()
      ...
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 needsTaskCommit
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 commitTask
      com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 finish()

      As is obvious, two instances are used for different purposes - one to get the record writer and do the actual write, and another to call the OutputCommitter and its methods.
      Since they are from different instances (StoreFuncInterface), the output committer is unable to gracefully commit and cleanup.

      I am not attaching the StoreFunc, but any user defined StoreFunc will exhibit this behavior.

      Attachments

        1. PIG-1684.1.patch
          18 kB
          Thejas Nair
        2. javadoc.patch
          0.6 kB
          Thejas Nair

        Activity

          People

            thejas Thejas Nair
            mridulm@yahoo-inc.com Mridul Muralidharan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: