maven-javadoc-plugin not working correctly together with maven-toolchains-plugin

We are using maven-javadoc-plugin in conjunction with maven-toolchains-plugin and we are facing an unpleasant problem.
We have defined toolchains.xml file containing jdk versions needed for the build.
For our build automation server we are using jenkins Config File Provider plug-in to supply this toolchains.xml file to the build via -t command line argument e.g:

mvn clean deploy -t <path generated by the Config File Provider plug-in>.

We have maven multi-module build and some modules use maven-javadoc-plugin to generate javadocs for our API. Sometimes we have API modules that refer API classes located in another API module. maven-javadoc-plugin tries to link all of these but it fails every time with this error: "[ERROR] No toolchain found for type jdk".
I've looked to the source repository of maven-javadoc-plugin on github and I noticed that it forks maven process to build dependent javadocs without passing info from toolchains.xml to the forked maven which is needed to run the javadoc build. Instead the forked process always expects that toolchains.xml is located to ~/.m2/toolchains.xml and this doesn't work on jenkins.