2021-03-03T23:53:32.1060336Z 2021-03-03 23:52:30,669 ERROR org.apache.flink.runtime.blob.BlobClient [] - Failed to fetch BLOB cb48df78410d3d7319d32477aa68c5f5/p-496c3ccd82a70ee27f7f85532569e6871337a1ec-2a5bd98b6e2e355e86797bd2b1c5e1f5 from c08db7158bcf/172.27.0.2:42041 and store it under /__w/1/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-localDir-nm-0_0/usercache/hadoop/appcache/application_1614815513529_0001/blobStore-8a708ff7-4c43-4373-aede-0e9f511adddf/incoming/temp-00000000 No retries left.
2021-03-03T23:53:32.1061435Z java.io.IOException: Could not connect to BlobServer at address c08db7158bcf/172.27.0.2:42041
2021-03-03T23:53:32.1061906Z java.io.IOException: Could not connect to BlobServer at address c08db7158bcf/172.27.0.2:42041
2021-03-03T23:53:32.1062642Z at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:102) ~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1063501Z at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:137) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1064585Z at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:166) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1065487Z at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:187) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1066542Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:251) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1067717Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:228) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1068930Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:199) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1070084Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:333) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1071083Z at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:991) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1071907Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1072818Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1073289Z at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
2021-03-03T23:53:32.1073662Z Caused by: java.net.ConnectException: Connection refused (Connection refused)
2021-03-03T23:53:32.1074137Z at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
2021-03-03T23:53:32.1074650Z at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
2021-03-03T23:53:32.1075276Z at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
2021-03-03T23:53:32.1075796Z at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
2021-03-03T23:53:32.1076271Z at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
2021-03-03T23:53:32.1076663Z at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
2021-03-03T23:53:32.1077389Z at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:96) ~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1077794Z ... 11 more
2021-03-03T23:53:32.1079748Z 2021-03-03 23:52:30,680 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: Custom File Source (1/1)#0 (07ae013da887de04146706553020c491) switched from DEPLOYING to FAILED with failure cause: java.io.IOException: Failed to fetch BLOB cb48df78410d3d7319d32477aa68c5f5/p-496c3ccd82a70ee27f7f85532569e6871337a1ec-2a5bd98b6e2e355e86797bd2b1c5e1f5 from c08db7158bcf/172.27.0.2:42041 and store it under /__w/1/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-localDir-nm-0_0/usercache/hadoop/appcache/application_1614815513529_0001/blobStore-8a708ff7-4c43-4373-aede-0e9f511adddf/incoming/temp-00000000
2021-03-03T23:53:32.1081142Z at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:167)
2021-03-03T23:53:32.1081646Z at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:166)
2021-03-03T23:53:32.1082149Z at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:187)
2021-03-03T23:53:32.1082760Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:251)
2021-03-03T23:53:32.1083471Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:228)
2021-03-03T23:53:32.1084309Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:199)
2021-03-03T23:53:32.1085010Z at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:333)
2021-03-03T23:53:32.1085615Z at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:991)
2021-03-03T23:53:32.1086053Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
2021-03-03T23:53:32.1086449Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
Bug
Major