Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7438

Counters, statistics, and metrics [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • Spark

    Description

      Hive makes use of MapReduce counters for statistics and possibly for other purposes. For Hive on Spark, we should achieve the same functionality using Spark's accumulators.

      Hive also collects metrics from MapReduce jobs traditionally. Spark job very likely publishes a different set of metrics, which, if made available, would help user to get insights into their spark jobs. Thus, we should obtain the metrics and make them available as we do for MapReduce.

      This task therefore includes:

      1. identify Hive's existing functionality w.r.t. counters, statistics, and metrics;
      2. design and implement the same functionality in Spark.

      Please refer to the design document for more information. https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark#HiveonSpark-CountersandMetrics

      Attachments

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. HIVE-7706 ClassCastException trying to CTAS table

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-7551 expand spark accumulator to support hive counter [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-7552 Collect spark job statistic through spark metrics [Spark Branch]

          • Major - Major loss of function.
          • Resolved
          incorporates

          Task - A task that needs to be done. HIVE-7893 Find a way to get a job identifier when submitting a spark job [Spark Branch]

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Test - A new unit, integration or system test. HIVE-7772 Add tests for order/sort/distribute/cluster by query [Spark Branch]

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              chengxiang li Chengxiang Li
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: