Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-380

Avro Container File format change: add block size to block descriptor

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.3.0
    • doc, java, spec
    • None
    • Incompatible change, Reviewed

    Description

      The new file format in AVRO-160 limits a few use cases that I have found to be important.

      A block currently contains a count of the number of records, the block data, and a sync marker.
      This change would add the block size, in bytes, along side the number of records.

      This allows efficient access to a block's data without the need to decode the data into individual Datums, which is useful for various use cases.

      Attachments

        1. AVRO-380.patch
          11 kB
          Scott Carey
        2. AVRO-380.patch
          13 kB
          Scott Carey

        Activity

          People

            scott_carey Scott Carey
            scott_carey Scott Carey
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: