Skip to content

Commit 5b40803

Browse files
committed
Fix flaky FileIOTest.testMatchWatchForNewFiles test under CI filesystems
addresses #19480 The `FileIOTest.testMatchWatchForNewFiles` test occasionally flakes in the CI environment because the `updOptions` configuration in `CopyFilesFn` does not preserve file attributes when overwriting existing files. In some CI filesystems, this causes the copied file to register a `lastModifiedMillis` timestamp of `0`. Thus, when `ExtractFilenameAndLastUpdateFn` parses this file, it throws a `RuntimeException` at `FileIO.java:800`, failing the pipeline run. This PR adds `StandardCopyOption.COPY_ATTRIBUTES` to preserve the file's original timestamps, avoiding the exception.
1 parent c8e45e7 commit 5b40803

File tree

1 file changed

+5
-3
lines changed

1 file changed

+5
-3
lines changed

sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -249,16 +249,18 @@ public void processElement(ProcessContext context, @StateId("count") ValueState<
249249
context.output(Objects.requireNonNull(context.element()).getValue());
250250

251251
CopyOption[] cpOptions = {StandardCopyOption.COPY_ATTRIBUTES};
252-
CopyOption[] updOptions = {StandardCopyOption.REPLACE_EXISTING};
252+
CopyOption[] updOptions = {
253+
StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.COPY_ATTRIBUTES
254+
};
253255
final Path sourcePath = Paths.get(sourcePathStr);
254256
final Path watchPath = Paths.get(watchPathStr);
255257

256258
if (0 == current) {
257-
Thread.sleep(100);
259+
Thread.sleep(500);
258260
Files.copy(sourcePath.resolve("first"), watchPath.resolve("first"), updOptions);
259261
Files.copy(sourcePath.resolve("second"), watchPath.resolve("second"), cpOptions);
260262
} else if (1 == current) {
261-
Thread.sleep(100);
263+
Thread.sleep(500);
262264
Files.copy(sourcePath.resolve("first"), watchPath.resolve("first"), updOptions);
263265
Files.copy(sourcePath.resolve("second"), watchPath.resolve("second"), updOptions);
264266
Files.copy(sourcePath.resolve("third"), watchPath.resolve("third"), cpOptions);

0 commit comments

Comments
 (0)