Major Description
Very occasionally one of the linkRoute tests in the system_tests_multi_tenancy.py ctests will fail:
38: FAIL: test_22_two_router_link_route_targeted_no_tenant (system_tests_multi_tenancy.RouterTest)
38: ----------------------------------------------------------------------
38: Traceback (most recent call last):
38: File "/home/kgiusti/work/dispatch/qpid-dispatch/test /system_tests_multi_tenancy.py", line 290, in test_22_two_router_link_route_targeted_no_tenant
38: self.assertIsNone(test.error)
38: AssertionError: 'Timeout Expired: n_sent=0 n_rcvd=0 n_settled=0' is not None
Examining the router logs it appears as if the attempt to use the link route is rejected with a "no route" error.
I believe the problem is that each test uses the same link route configured address. Each test starts by establishing the route container connection then polls the routers waiting for the link route address to propagate. Once the poll succeeds the test starts.
IIUC what is happening is that at the end of each test the link route address may still exist in the route tables - it has not been totally removed. This causes the next test's poll to immediately succeed before the test's link route has been properly propagated. This results in the failure if the new link route doesn't establish itself before traffic is sent.
The fix is to have each test wait for all link route addresses to be removed from the routing tables before the test ends.