validatesCrossLanguageRunnerGoUsingJava failing on beam_PostCommit_XVR_Direct

This appears to be a flake, but it has failed 5 of the last 6 runs and 12 of the last 28. This has been flaking at around 50% of runs for longer than we have build history. I suspect https://github.com/apache/beam/pull/16819 is responsible.

begining of history:
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2441/
(skiped a bunch)
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2474/
(skiped a bunch)
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2501/
(skiped a bunch)
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2531/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2534/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2541/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2542/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2545/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2546/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2550/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2554/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2556/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2557/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2558/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/2559/