Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-1124

IgniteCacheAtomicReplicatedNodeRestartSelfTest hangs because of an assertion

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • sprint-7
    • ignite-1.4
    • cache
    • None

    Description

      This test hangs almost always causing the execution timeout of the whole Ignite Cache Restarts suite.

      The test start hanging after the following error appears in the log:

      java.lang.AssertionError: null
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1011)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1008)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:309)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:47)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:291)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.checkComplete(GridDhtAtomicUpdateFuture.java:407)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,298][INFO ][disco-event-worker-#166979%replicated.IgniteCacheAtomicReplicatedNodeRestartSelfTest5%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=3028cca9-7f58-4355-81b1-017e948ba003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47504], discPort=47504, order=326, intOrder=166, loc=false, ver=1.4.1#19700101-sha1:00000000, isClient=false]
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onNodeLeft(GridDhtAtomicUpdateFuture.java:152)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,298][INFO ][disco-event-worker-#166979%replicated.IgniteCacheAtomicReplicatedNodeRestartSelfTest5%][GridDiscoveryManager] Topology snapshot [ver=330, server nodes=4, client nodes=0, CPUs=4, heap=2.8GB]
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$3.onEvent(GridCacheMvccManager.java:197)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,299][INFO ][restart-worker-1][TcpDiscoverySpi] Successfully bound to TCP port [port=47504, localHost=/127.0.0.1]
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1740)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1931)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1779)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] [13:22:38,298][ERROR][disco-event-worker-#155953%replicated.IgniteCacheAtomicReplicatedNodeRestartSelfTest1%][GridDiscoveryManager] Unexpected exception in discovery worker thread (ignored).
[13:22:38]W:		 [org.apache.ignite:ignite-core] java.lang.AssertionError: null
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1011)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1008)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:309)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:47)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:291)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.checkComplete(GridDhtAtomicUpdateFuture.java:407)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onNodeLeft(GridDhtAtomicUpdateFuture.java:152)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$3.onEvent(GridCacheMvccManager.java:197)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1740)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1931)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,299][INFO ][restart-worker-0][GridCacheProcessor] Stopped cache: TEST_CACHE
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1779)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,299][INFO ][restart-worker-0][GridCacheProcessor] Stopped cache: ignite-marshaller-sys-cache
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,299][INFO ][restart-worker-0][GridCacheProcessor] Stopped cache: ignite-sys-cache
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,299][INFO ][restart-worker-0][GridCacheProcessor] Stopped cache: ignite-atomics-sys-cache
[13:22:38]W:		 [org.apache.ignite:ignite-core] [13:22:38,298][ERROR][disco-event-worker-#155953%replicated.IgniteCacheAtomicReplicatedNodeRestartSelfTest1%][GridDiscoveryManager] Runtime error caught during grid runnable execution: DiscoveryWorker [nodeSegFired=false]
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,300][INFO ][restart-worker-0][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1436955758178, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=7680a819e41-3028cca9-7f58-4355-81b1-017e948ba003, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1, pendingUndeploy=false, undeployed=true, usage=0]
[13:22:38]W:		 [org.apache.ignite:ignite-core] java.lang.AssertionError: null
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1011)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,300][INFO ][restart-worker-0][GridDeploymentLocalStore] Class locally undeployed: class org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1008)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,300][INFO ][restart-worker-0][GridDeploymentLocalStore] Class locally undeployed: class org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:309)
[13:22:38] :		 [org.apache.ignite:ignite-core] [13:22:38,300][INFO ][restart-worker-0][GridDeploymentLocalStore] Class locally undeployed: class org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:47)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:291)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.checkComplete(GridDhtAtomicUpdateFuture.java:407)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onNodeLeft(GridDhtAtomicUpdateFuture.java:152)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$3.onEvent(GridCacheMvccManager.java:197)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1740)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1931)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1779)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] Exception in thread "disco-event-worker-#155953%replicated.IgniteCacheAtomicReplicatedNodeRestartSelfTest1%" java.lang.AssertionError: null
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1011)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$2.apply(GridNearAtomicUpdateFuture.java:1008)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:309)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onDone(GridDhtAtomicUpdateFuture.java:47)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:291)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.checkComplete(GridDhtAtomicUpdateFuture.java:407)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture.onNodeLeft(GridDhtAtomicUpdateFuture.java:152)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$3.onEvent(GridCacheMvccManager.java:197)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1740)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1931)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1779)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
[13:22:38]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)

      Attachments

        1. ignite-1124.patch
          106 kB
          Semen Boikov

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-1345 Cache stability issues in load/failover tests

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              dmagda Denis A. Magda
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: