Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8237

Enabling preads always fetches hedged reads metrics

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Backend
    • None
    • ghx-label-4

    Description

      In HdfsFileReader if preads are enabled, we assume that hedged reads are enabled as well, so whenever we close a file we make a libhdfs call to collect a few hedged read metrics from the underlying FileSystem object. However, as part of IMPALA-5212 we may want to enable preads even when hedged reads are disabled, so making the call to libhdfs to fetch hedged read metrics will be a waste.

      Digging through the HDFS code, it seems the HDFS client triggers hedged reads only if dfs.client.hedged.read.threadpool.size is greater than 0. We can use the same check in HdfsFileReader to trigger the fetch of hedged read metrics. The issue is that currently libhdfs does not provide a good way of getting the value of dfs.client.hedged.read.threadpool.size, it provides a method called hdfsConfGetInt, but that method simply calls new Configuration() and fetches the value of  dfs.client.hedged.read.threadpool.size from it. The issue is that calling new Configuration simply loads the current hdfs-site.xml, core-site.xml, etc. which does not take into account the scenario where the default configuration has been modified for specific filesystem objects - e.g. using hdfsBuilder to set non-default configuration parameters (see HDFS-14301 for more details).

      Attachments

        Issue Links

          Activity

            People

              stakiar Sahil Takiar
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: