Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5444 Ozone Upgrades vNext
  3. HDDS-5516

Duplicate metrics registered while running checkScmHA upon scm startup

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • None

    Description

      Running non-rolling upgrade for scm, log reports:

      // code placeholder
      2021-07-30 07:17:00,932 [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.ozone.upgrade.UpgradeFinalizer: Running pre-finalized state validations for unfinalized layout features.
      2021-07-30 07:17:00,934 [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.ozone.upgrade.UpgradeFinalizer: Executing pre finalize state validation ScmHAUnfinalizedStateValidationAction
      2021-07-30 07:17:00,934 [Listener at 0.0.0.0/9860] INFO org.apache.hadoop.ozone.upgrade.AbstractLayoutVersionManager: Initializing Layout version manager with metadata layout = INITIAL_VERSION (version = 0), software layout = SCM_HA (version = 2)
      2021-07-30 07:17:00,934 [Listener at 0.0.0.0/9860] WARN org.apache.hadoop.metrics2.util.MBeans: Error creating MBean object name: Hadoop:service=LayoutVersionManager,name=HDDSLayoutVersionManager
      org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=LayoutVersionManager,name=HDDSLayoutVersionManager already exists!
              at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
              at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
              at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:165)
              at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:97)
              at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:73)
              at org.apache.hadoop.ozone.upgrade.AbstractLayoutVersionManager.init(AbstractLayoutVersionManager.java:75)
              at org.apache.hadoop.hdds.upgrade.HDDSLayoutVersionManager.<init>(HDDSLayoutVersionManager.java:50)
              at org.apache.hadoop.hdds.scm.server.upgrade.ScmHAUnfinalizedStateValidationAction.checkScmHA(ScmHAUnfinalizedStateValidationAction.java:61)
              at org.apache.hadoop.hdds.scm.server.upgrade.ScmHAUnfinalizedStateValidationAction.execute(ScmHAUnfinalizedStateValidationAction.java:48)
              at org.apache.hadoop.hdds.scm.server.upgrade.ScmHAUnfinalizedStateValidationAction.execute(ScmHAUnfinalizedStateValidationAction.java:41)
              at org.apache.hadoop.ozone.upgrade.BasicUpgradeFinalizer.runValidationAction(BasicUpgradeFinalizer.java:264)
              at org.apache.hadoop.ozone.upgrade.BasicUpgradeFinalizer.runPrefinalizeStateActions(BasicUpgradeFinalizer.java:243)
              at org.apache.hadoop.hdds.scm.server.upgrade.SCMUpgradeFinalizer.runPrefinalizeStateActions(SCMUpgradeFinalizer.java:133)
              at org.apache.hadoop.hdds.scm.server.upgrade.SCMUpgradeFinalizer.runPrefinalizeStateActions(SCMUpgradeFinalizer.java:39)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManager.start(StorageContainerManager.java:1242)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:169)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:142)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:71)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:47)
              at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
              at picocli.CommandLine.access$1100(CommandLine.java:145)
              at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
              at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
              at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
              at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2152)
              at picocli.CommandLine.parseWithHandlers(CommandLine.java:2530)
              at picocli.CommandLine.parseWithHandler(CommandLine.java:2465)
              at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:96)
              at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87)
              at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:60)
      Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=LayoutVersionManager,name=HDDSLayoutVersionManager already exists!
              at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:131)
              ... 29 more
      

      Here we create a new HDDSLayoutVersionManager object upon each call of checkScmHA(), each will try to register metrics. But we already have a HDDSLayoutVersionManager created for a running scm, so we got duplicate registers.

      Attachments

        Issue Links

          Activity

            People

              markgui Mark Gui
              markgui Mark Gui
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: