Detect HWM Store decorator

@etl_entities.hwm_store.hwm_store_detect.detect_hwm_store

Detect HWM store by config object.

Note

This decorator could use only HWM Stores which were registered using register_hwm_store_class.

Parameters:

keystr

The name of the section in the config that stores information about hwm

Warning

DO NOT use dot . in config keys

Examples

Config

conf/config.yaml

# if HWM store can be created with no args
hwm_store: yaml  # this is HWM Store class alias

or

conf/config.yaml

# named constructor args
hwm_store:
    yaml:
        path: /some/path
        encoding: utf-8

Config could be nested:

conf/config.yaml

myetl:
    env:
        hwm_store: yaml

Using decorator:

pipelines/my_pipeline.py

import hydra
from omegaconf import DictConfig

from etl_entities.hwm_store import detect_hwm_store

@hydra.main(
    # path to config dir and file name (without extension)
    config_path="../conf",
    config_name="config",
)
@detect_hwm_store(
    # key=... is a full key name of config item, delimited by dot ``.``
    key="myetl.env.hwm_store",
)
def main(config: DictConfig):
    # inside this function YAMLHWMStore is used
    pass