Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11617

incorrect AgeOfLastAppliedOp and AgeOfLastShippedOp in replication Metrics when no new replication OP

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.98.2
    • 0.99.0, 0.98.6
    • Replication
    • None
    • Reviewed

    Description

      AgeOfLastAppliedOp in MetricsSink.java is to indicate the time an edit sat in the 'replication queue' before it got replicated(aka applied)

        /**
         * Set the age of the last applied operation
         *
         * @param timestamp The timestamp of the last operation applied.
         * @return the age that was set
         */
        public long setAgeOfLastAppliedOp(long timestamp) {
          lastTimestampForAge = timestamp;
          long age = System.currentTimeMillis() - lastTimestampForAge;
          rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
          return age;
        } 
      

      In the following scenario:
      1) at 7:00am a sink op is applied, and the SINK_AGE_OF_LAST_APPLIED_OP is
      set for example 100ms;
      2) and then NO new Sink op occur.
      3) when a refreshAgeOfLastAppliedOp() is invoked at 8:00am. Instead of
      return the 100ms, the AgeOfLastAppliedOp become 1hour + 100ms,

      It was because that refreshAgeOfLastAppliedOp() get invoked periodically by getStats().

      proposed fix:

      --- hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
      +++ hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
      @@ -35,6 +35,7 @@ public class MetricsSink {
       
         private MetricsReplicationSource rms;
         private long lastTimestampForAge = System.currentTimeMillis();
      +  private long age = 0;
       
         public MetricsSink() {
           rms = CompatibilitySingletonFactory.getInstance(MetricsReplicationSource.class);
      @@ -47,8 +48,12 @@ public class MetricsSink {
          * @return the age that was set
          */
         public long setAgeOfLastAppliedOp(long timestamp) {
      -    lastTimestampForAge = timestamp;
      -    long age = System.currentTimeMillis() - lastTimestampForAge;
      +    if (lastTimestampForAge != timestamp) {
      +      lastTimestampForAge = timestamp;
      +      this.age = System.currentTimeMillis() - lastTimestampForAge;
      +    } else {
      +      this.age = 0;
      +    }
           rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
           return age;
         }
      

      detail discussion in dev@hbase

      Attachments

        Issue Links

          Activity

            People

              nidmhbase Demai Ni
              nidmhbase Demai Ni
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: