[Python][CI] AMD64 Conda Python 3.7 Pandas 0.24 cron job failing in to_pandas extension dtype test

See https://github.com/apache/arrow/pull/9096/checks?check_run_id=1689113764#step:8:5354

I suppose an error in the test specifically for pandas 0.24 (since we have quite some pandas version checks in that test)

This is happening only in the cron job, other (nightly) builds only have pandas 0.23, latest or master