Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4246

HBaseStorage should implement getShipFiles

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • None
    • None
    • Reviewed

    Description

      HBaseStorage.initializeHBaseClassLoaderResources() uses TableMapReduceUtil APIs to add dependency jars. That sets the tmpjars setting which makes JobClient ship the jars to hdfs and use that path in distributed cache. That bypasses the optimizations in PIG-2672 and PIG-3861 which avoid shipping the jars to hdfs. Instead it should implement the getShipFiles() API introduced in PIG-4141 so that PIG-2672 or PIG-3861 avoid shipping the same jar multiple times to hdfs for a job.

      Attachments

        1. PIG-4246-1.patch
          9 kB
          Rohini Palaniswamy

        Activity

          People

            rohini Rohini Palaniswamy
            rohini Rohini Palaniswamy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: