[Python] CMake cannot find package configuration file in CMAKE_MODULE_PATH

Hi, I got exactly the same issue as described here:

https://github.com/apache/arrow/pull/14586

This happens to some people when they trying to install pyarrow from pip under windows. Since only opening an MR at github might not cause any attention so I open an issue here.