kedro.contrib.config.templated_config.TemplatedConfigLoader

class kedro.contrib.config.templated_config.TemplatedConfigLoader(conf_paths, *, globals_pattern=None, globals_dict=None)[source]

Bases: kedro.config.config.ConfigLoader

Extension of the ConfigLoader class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.

The easiest way to use this class is by incorporating it into the KedroContext. This can be done by extending the KedroContext and overwriting the config_loader method, making it return a TemplatedConfigLoader object instead of a ConfigLoader object.

For this method to work, the context_path variable in .kedro.yml needs to be pointing at this newly created class. The run.py script has an extension of the KedroContext by default, called the ProjectContext.

Example:

from kedro.context import KedroContext, load_context
from kedro.contrib.config import TemplatedConfigLoader


class MyNewContext(KedroContext):

    def _create_config_loader(self, conf_paths: Iterable[str]) -> TemplatedConfigLoader:
        return TemplatedConfigLoader(
            conf_paths,
            globals_pattern="*globals.yml",
            globals_dict={"param1": "CSVLocalDataSet"}
        )

my_context = load_context(Path.cwd(), env=env)
my_context.run(tags, runner, node_names, from_nodes, to_nodes)

The contents of the dictionary resulting from the globals_pattern get merged with the globals_dict. In case of conflicts, the keys in globals_dict take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, a ValueError will be raised.

Global parameters can be namespaced as well. An example could work as follows:

globals.yml

bucket: "my_s3_bucket"

environment: "dev"

datasets:
    csv: "CSVS3DataSet"
    spark: "SparkDataSet"

folders:
    raw: "01_raw"
    int: "02_intermediate"
    pri: "03_primary"
    fea: "04_features"

catalog.yml

raw_boat_data:
    type: "${datasets.spark}"
    filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
    file_format: parquet

raw_car_data:
    type: "${datasets.csv}"
    filepath: "data/${environment}/${folders.raw}/cars.csv"
    bucket_name: "${bucket}"
    file_format: "${car_file_format|parquet}"

This uses jmespath in the background. For more information see: https://github.com/jmespath/jmespath.py and http://jmespath.org/.

Methods

TemplatedConfigLoader.__init__(conf_paths, *) Instantiate a TemplatedConfigLoader.
TemplatedConfigLoader.get(*patterns) Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.
__init__(conf_paths, *, globals_pattern=None, globals_dict=None)[source]

Instantiate a TemplatedConfigLoader.

Parameters:
  • conf_paths (Union[str, Iterable[str]]) – Non-empty path or list of paths to configuration directories.
  • globals_pattern (Optional[str]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.
  • globals_dict (Optional[Dict[str, Any]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the globals_dict keys take precedence.
get(*patterns)[source]

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

Parameters:patterns (str) – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
Return type:Dict[str, Any]
Returns:A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details).
Raises:ValueError – malformed config found.