Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-751

Namenode constantly using up 100% CPU

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • 0.8.0
    • None
    • None
    • None

    Description

      Trying to figure out the exact cause, but CPU load of the namenode is constantly 100% without any file exchanges.

      .log file of the namenode showing the following exception about every 20 seconds.

      2006-11-27 16:02:07,199 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020 call error: java.io.IOException: java.util.NoSuchElementException
      java.io.IOException: java.util.NoSuchElementException
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1029)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
      at java.util.AbstractCollection.toArray(AbstractCollection.java:176)
      at org.apache.hadoop.dfs.DatanodeDescriptor.getBlocks(DatanodeDescriptor.java:96)
      at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1446)
      at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:506)
      at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:387)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:518)

      On some of the datanodes, .log file is showing

      2006-11-27 16:03:57,542 WARN org.apache.hadoop.dfs.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.NoSuchElementException
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1029)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
      at java.util.AbstractCollection.toArray(AbstractCollection.java:176)
      at org.apache.hadoop.dfs.DatanodeDescriptor.getBlocks(DatanodeDescriptor.java:96)
      at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1446)
      at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:506)
      at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:387)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:518)

      at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)

      Attachments

        Issue Links

          Activity

            People

              sameerp Sameer Paranjpye
              knoguchi Koji Noguchi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: