File HWM#

HWM classes

What is File HWM?#

Sometimes it’s necessary to read/download only new files from a source folder.

For example, there is a folder with files:

$ hdfs dfs -ls /path

2MB 2023-09-09 10:13 /path/my/file123
4Mb 2023-09-09 10:15 /path/my/file234

When new file is being added to this folder:

$ hdfs dfs -ls /path

2MB 2023-09-09 10:13 /path/my/file123
4Mb 2023-09-09 10:15 /path/my/file234
5Mb 2023-09-09 10:20 /path/my/file345  # new one

To download only new files, if is required to somehow track them, and then filter using the information from a previous run.

This technique is called High WaterMark or HWM for short. It is used by different strategies to implement some complex logic of filtering source data.

Supported types#

There are a several ways to track HWM value:

  • Save the entire file list, and then select only files not present in this list (file_list)

  • Save max modified time of all files, and then select only files with modified_time higher than this value

  • If file name contains some incrementing value, e.g. id or datetime, parse it and save max value of all files, then select only files with higher value

  • and so on

Currently the only HWM type implemented for files is file_list. Other ones can be implemented on-demand