Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10561

MVCC: Fix MVCC cache rebalance.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.8
    • mvcc

    Description

      Unexpected transaction state may be observed on some nodes after rebalance. Reproducer: GridCacheRebalancingSyncSelfTest#testComplexRebalancing

      Stacktrace:

      javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]

	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
	at org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
	at org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
	at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:933)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:152)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:150)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1059)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:584)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:383)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:309)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:100)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:299)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1568)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1196)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092)
	at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	... 1 more
Caused by: class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1544033736224:446552:1, txVer=1544033736224:800063:1073741823, localNodeId=e60be4d5-b1d7-4c09-8b88-9b8964400002]
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:317)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:304)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.isVisible(MvccUtils.java:271)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.isVisible(MvccUtils.java:210)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils$GetVisibleState.apply(MvccUtils.java:906)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils$GetVisibleState.apply(MvccUtils.java:890)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.invoke(MvccUtils.java:618)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.getVisibleState(MvccUtils.java:362)
	at org.apache.ignite.internal.processors.cache.tree.mvcc.search.MvccSnapshotSearchRow.apply(MvccSnapshotSearchRow.java:88)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.fillFromBuffer0(BPlusTree.java:5446)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5316)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5241)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetCursor.notFound(BPlusTree.java:2935)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5651)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5636)
	at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
	at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8700(BPlusTree.java:90)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5325)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.iterate(BPlusTree.java:5475)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.access$5000(BPlusTree.java:5401)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1032)
	... 25 more

       

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-11088 Flacky LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge

          • Major - Major loss of function.
          • Closed
          Dependent

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-10693 MVCC TX: Random server restart tests became failed

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-10604 MVCC: mvcc history can be missed during remove operation.

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-10560 MVCC: Data loss after rebalance

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-11088 Flacky LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-10769 MVCC: CacheMvccContinuousQueryClientTest.testNodeJoinsRestartQuery fails sometimes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-11623 MVCC: testRebalancingDuringLoad_N tests are flacky in master

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-9949 MVCC TX: Scan query can return wrong result on unstable topology

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #5799

          Activity

            People

              amashenkov Andrey Mashenkov
              rkondakov Roman Kondakov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m