DTC Transactions do not respect rollbacks consistently leading to duplicated messages

In certain circumstances, rollback does not work properly, as a result duplicate messages are generated from normal DTC operations. The circumstances under which this issues occurs are:

• DTC transaction initiated, but fails to commit;
• Another DTC transaction initiated and successfully commits;
• When a transaction is roll backed and committed on retry;

As a result the number of messages sent is larger to the number of messages received by the target broker. Some of the messages are not roll backed when it fails and they get sent again when committed on retry.

This issue has been raised in Jira as issues 413 and 472:
https://issues.apache.org/jira/browse/AMQNET-413
https://issues.apache.org/jira/browse/AMQNET-472

Analysing the code we realised that these two issues above are similar, and they have the same cause. We have therefore developed a patch based on feedback and unit tests provided by those two issues, where we fix the problem and we pass all existing unit tests. The changes modify only code related to DTC, and it was developed in .NET 2.0 as per the requirements for 413/472

The fix implements the solution suggested in the patch for issue 472 where some lock conditions were missing. In addition we have added a condition where we will not lock the transaction whenever a distributed transaction occurs. The DtcWaitHandle.WaitOne causes DTC transactions involving a database or XA transaction to fail unit testing.

The patch was extensively tested with over a 1000 messages (1K, 1.5K, 10K, 100K, 1M) using two brokers across different servers to guarantee distributed transactions. Testing also included different message sizes, infrastructure setup, databases, and message content verification to validate its integrity and ensure messages were not duplicated.