Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-2497

Failed post results in tunnel buffers being returned to freelist prematurely

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.2.0
    • Core
    • None

    Description

      When a post fails to an origin server either the server died or the server returned a response without reading all of the post data, in either case, TS will destroy buffers too early. This normally does not result in a crash because the MIOBuffers are returned to the freelist and only with sufficient load will the race happen causing a crash. Additionally, even if a crash doesn't happen you might have data corruption across post requests from the buffers being used after being returned to the freelist.

      Thanks to Thomas Jackson for help reproducing and resolving this bug.

      An example stack trace, while we've seen other crashes in write_avail too.

      #0 0x00000000004eff14 in IOBufferBlock::read_avail (this=0x0) at ../iocore/eventsystem/I_IOBuffer.h:362
      #1 0x000000000050d151 in MIOBuffer::append_block_internal (this=0x2aab38001130, b=0x2aab0c037200) at ../iocore/eventsystem/P_IOBuffer.h:946
      #2 0x000000000050d39b in MIOBuffer::append_block (this=0x2aab38001130, asize_index=15) at ../iocore/eventsystem/P_IOBuffer.h:986
      #3 0x000000000050d49b in MIOBuffer::add_block (this=0x2aab38001130) at ../iocore/eventsystem/P_IOBuffer.h:994
      #4 0x000000000055cee2 in MIOBuffer::check_add_block (this=0x2aab38001130) at ../iocore/eventsystem/P_IOBuffer.h:1002
      #5 0x000000000055d115 in MIOBuffer::write_avail (this=0x2aab38001130) at ../iocore/eventsystem/P_IOBuffer.h:1048
      #6 0x00000000006c18f3 in read_from_net (nh=0x2aaafca0d208, vc=0x2aab1c009140, thread=0x2aaafca0a010) at UnixNetVConnection.cc:234
      #7 0x00000000006c37bf in UnixNetVConnection::net_read_io (this=0x2aab1c009140, nh=0x2aaafca0d208, lthread=0x2aaafca0a010) at UnixNetVConnection.cc:816
      #8 0x00000000006be392 in NetHandler::mainNetEvent (this=0x2aaafca0d208, event=5, e=0x271d8e0) at UnixNet.cc:380
      #9 0x00000000004f05c4 in Continuation::handleEvent (this=0x2aaafca0d208, event=5, data=0x271d8e0) at ../iocore/eventsystem/I_Continuation.h:146
      #10 0x00000000006e361e in EThread::process_event (this=0x2aaafca0a010, e=0x271d8e0, calling_code=5) at UnixEThread.cc:142
      #11 0x00000000006e3b13 in EThread::execute (this=0x2aaafca0a010) at UnixEThread.cc:264
      #12 0x00000000006e290b in spawn_thread_internal (a=0x2716400) at Thread.cc:88
      #13 0x0000003372c077e1 in start_thread () from /lib64/libpthread.so.0
      #14 0x00000033728e68ed in clone () from /lib64/libc.so.6

      Attachments

        1. TS-2497.patch
          1 kB
          Brian Geffon
        2. repro.js
          1 kB
          Brian Geffon
        3. origin-server.js
          0.7 kB
          Feifei Cai
        4. client.js
          1 kB
          Feifei Cai

        Issue Links

          Activity

            People

              briang Brian Geffon
              briang Brian Geffon
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: