Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-1192

Slider Agent tarball should be uploaded to HDFS to avoid client side dependency

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • None
    • Slider 1.0.0
    • None
    • None

    Description

      Trying to use Slider client from a Hive process that is shipped with slider client JAR

       
Caused by: java.io.FileNotFoundException: File /grid/5/sershe/tez-autobuild/dist/hive/lib/slider-agent.tar.gz does not exist
   at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)
   at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
   at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
   at org.apache.slider.providers.ProviderUtils.addAgentTar(ProviderUtils.java:128)
   at org.apache.slider.providers.agent.AgentClientProvider.prepareAMAndConfigForLaunch(AgentClientProvider.java:270)

      I can see the gz file in the slider lib directory. However, using slider client one should not even be required to have slider on the box (Hive already has the jars); nor is there a good way to find where it is, if installed (given that jar is shipped and we are not aware of the slider installation).

      The gz file should be included with client jars (perhaps as a resource?)

      Update: setting slider.libdir and/or SLIDER_HOME actually doesn't work at all.
      The code in addAgentTar basically looks for it in the directory of the client jar, no option to override it at all.
      That makes slider client absolutely unusable

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. SLIDER-777 Provide slider dependencies as a self contained versioned tarball

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is required by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-15688 LlapServiceDriver - an option to start the cluster immediately

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              gsaha Gour Saha
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: