Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11583

Parent spans are not initialized to NullScope for every DFSPacket

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.1
    • 2.7.4
    • tracing
    • None
    • Reviewed

    Description

      The issue was found while working with PHOENIX-3752.

      Each packet received by the run() method of DataStreamer class, uses the parents field of the DFSPacket to create a new dataStreamer span, which in turn creates a writeTo span as its child span. The parents field is initialized when the packet is added to the dataQueue and the value is initialized from the ThreadLocal. This is how HTrace handles spans.
      A TraceScope is created and initialized to NullScope before the loop which runs till the point when the stream is closed.

      Consider the following scenario, when the dataQueue contains multiple packets, only the first of which has a tracing enabled. The scope is initialized to the dataStreamer scope and a writeTo span is created as its child, which gets closed once the packet is sent out to a remote datanode. Before writeTo span is started, the dataStreamer scope is detached. So calling the close method on it doesn't do anything at the end of loop.

      The second iteration will be using the stale value of the scope variable with a DFSPacket on which tracing is not enabled. This results in generation of an orphan writeTo spans which are being delivered to the SpanReceiver as registered in the TraceFramework. This may result in unlimited number of spans being generated and sent out to receiver.

      Attachments

        1. HDFS-11583-branch-2.7.003.patch
          2 kB
          Masatake Iwasaki
        2. HDFS-11583-branch-2.7.002.patch
          2 kB
          Masatake Iwasaki
        3. HDFS-11583-branch-2.7.001.patch
          2 kB
          Masatake Iwasaki

        Activity

          People

            iwasakims Masatake Iwasaki
            karanmehta93 Karan Mehta
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: