[HBASE-20662] Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0-alpha-1, 2.0.0
Fix Version/s: 3.0.0-alpha-1, 2.0.6, 2.1.5
Component/s: None
Labels:
- Quota
- space

Hadoop Flags:

Reviewed

Description

Steps to reproduce

Create a table and set quota with SpaceViolationPolicy.DISABLE having limit say 2MB
Now put rows until space quota is violated and table gets disabled
Next, increase space quota with limit say 4MB on the table
Now try putting a row into the table

 private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws Exception {
    SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
    Put put = new Put(Bytes.toBytes("to_reject"));
    put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1), Bytes.toBytes("to"),
      Bytes.toBytes("reject"));

    // Do puts until we violate space policy
    final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);

    // Now, increase limit
    setQuotaLimit(tn, policy, 4L);

    // Put some row now: should not violate as quota limit increased
    verifyNoViolation(policy, tn, put);
  }

Expected
We should be able to put data as long as newly set quota limit is not reached

Actual
We fail to put any new row even after increasing limit

Root cause
Increasing quota on a violated table triggers the table to be enabled, but since the table is already in violation, the system does not allow it to be enabled (may be thinking that a user is trying to enable it)

Relevant exception trace

2018-05-31 00:34:27,563 INFO  [regionserver/root1-ThinkPad-T440p:0.Chore.1] client.HBaseAdmin$14(844): Started enable of testSetQuotaAndThenIncreaseQuotaWithDisable0
2018-05-31 00:34:27,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525] ipc.CallRunner(142): callId: 11 service: MasterService methodName: EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568, exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a violated space quota.
2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1] quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will remain in violation.
org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a violated space quota.
	at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
	at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
	at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
	at org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:348)
	at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3061)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3053)
	at org.apache.hadoop.hbase.client.HBaseAdmin.enableTableAsync(HBaseAdmin.java:839)
	at org.apache.hadoop.hbase.client.HBaseAdmin.enableTable(HBaseAdmin.java:833)
	at org.apache.hadoop.hbase.quotas.policies.DisableTableViolationPolicyEnforcement.disable(DisableTableViolationPolicyEnforcement.java:62)
	at org.apache.hadoop.hbase.quotas.RegionServerSpaceQuotaManager.disableViolationPolicyEnforcement(RegionServerSpaceQuotaManager.java:208)
	at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:110)
	at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a violated space quota.
	at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
	at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
	at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
	at org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
	at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
	at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
	at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
	at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	... 1 more

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HBASE-20662.master.001.patch
01/Jun/18 19:10
10 kB
Nihal Jain
HBASE-20662.master.002.patch
09/Jun/18 22:47
72 kB
Nihal Jain
HBASE-20662.master.003.patch
01/Jul/18 09:17
21 kB
Nihal Jain
HBASE-20662.master.004.patch
11/Oct/18 20:16
21 kB
Nihal Jain
HBASE-20662.master.004.patch
08/Jan/19 18:08
21 kB
Nihal Jain
HBASE-20662.master.005.patch
08/Jan/19 18:47
21 kB
Nihal Jain
HBASE-20662.master.006.patch
10/Jan/19 21:08
30 kB
Nihal Jain
HBASE-20662.master.007.patch
13/Jan/19 20:25
28 kB
Nihal Jain
HBASE-20662.master.008.patch
24/Jan/19 19:57
27 kB
Nihal Jain
HBASE-20662.master.008.patch
25/Jan/19 08:47
27 kB
Nihal Jain
HBASE-20662.master.009.patch
25/Jan/19 09:33
27 kB
Nihal Jain
screenshot.png
25/Jan/19 17:15
172 kB
Nihal Jain
HBASE-20662.master.009.patch
25/Jan/19 17:55
27 kB
Nihal Jain
HBASE-20662.master.010.patch
04/Feb/19 13:06
28 kB
Nihal Jain
HBASE-20662.branch-2.1.001.patch
27/Mar/19 08:07
28 kB
Nihal Jain

Issue Links

is duplicated by

HBASE-21891 New space quota policy doesn't take effect if quota policy is changed after violation

Resolved

is related to

HBASE-20820 Removing SpaceQuota from a namespace does not remove it completely

Open

relates to

HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Namespace level

Resolved

HBASE-22129 Rewrite TestSpaceQuotas as parameterized tests

Open

HBASE-22914 Backport HBASE-20662 to branch-2.2

Resolved

links to

Review Board (master)

(1 links to)

Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates