UDF|TMUDF returns various errors regarding dlopen cached_libs file

In the latest version, we started to see various random errors returned from queries with UDF or TMUDF. The errors sometimes are 4316 error indicating failures to create the cached_libs directory, sometimes are 11245|11248 errors complaining about 'file too short' with the cached file. Whenever the errors occurred, the actual file appeared to exist in the cached_libs directory on every node. 

 It is a regression introduced  presumably because of the UDF|TMUDF cached_libs concept. This problem doesn't always happen. Sometimes the same sequence of statements failed previously would work fine the next time. It's also not clear if this only happens on a multi-node cluster, as the errors were seen on nap035, which happens to be a 4-node cluster.

Here are a few examples of these failures seen from the SQL regression run:

===========================================================================
SQL>select * from myview v, mytable t
+>where (qa_udf_integer(v.a), qa_udf_integer(t.b)) >
+>all (select qa_udf_integer(a), qa_udf_integer(b) from mytable2
+>where qa_udf_integer(a) = qa_udf_integer(b));

• • • ERROR[4316] An error was detected while extracting a udr routine library to local cache : Error 0 creating directory :/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/MYTEST/ .

===========================================================================

SQL>select * from
UDF(MYTMFUNC(TABLE(select * from trafodion.zzz_single_stream_features_namespace.mytable),
'TEST_CMPTIME_RUNTIME_OUT_TABLE_DEFINED_USING_PASSTHRU', '3')) t1,
UDF(MYTMFUNC(TABLE(select * from trafodion.zzz_single_stream_features_namespace_myns.mytable),
'TEST_CMPTIME_RUNTIME_OUT_TABLE_DEFINED_USING_PASSTHRU', '3')) t2,
UDF(MYTMFUNC(TABLE(select * from trafodion.zzz_single_stream_features_namespace_myns.mytable_too),
'TEST_CMPTIME_RUNTIME_OUT_TABLE_DEFINED_USING_PASSTHRU', '3')) t3
where t1.a = t2.a and t2.a = t3.a order by 1;

• • • ERROR[11245] An error occurred loading library 'qaTmudfTest_212414198362332378.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/ZZZ_SINGLE_STREAM_FEATURES_NAMESPACE_MYNS'. [2019-01-14 00:50:22]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/ZZZ_SINGLE_STREAM_FEATURES_NAMESPACE_MYNS/qaTmudfTest_212414198362332378.so: file too short. [2019-01-14 00:50:22]

===========================================================================

SQL>select * from myview v, mytable t
where (qa_udf_integer(v.a), qa_udf_integer(t.b)) >
all (select qa_udf_integer(a), qa_udf_integer(b) from mytable2
where qa_udf_integer(a) = qa_udf_integer(b));

• • • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]
    • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]
    • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]
    • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]
    • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]
    • ERROR[11245] An error occurred loading library 'qaUdfTest_212414083371782103.so' from path '/opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL'. [2019-01-12 15:20:07]
    • ERROR[11248] A call to dlopen returned errors 0 and 0. Details:
      /opt/trafodion/esgyndb/udr/DB__ROOT/cached_libs/UDF_UDF_GENERAL/qaUdfTest_212414083371782103.so: file too short. [2019-01-12 15:20:07]