kedro.config.TemplatedConfigLoader¶
-
class
kedro.config.
TemplatedConfigLoader
(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶ Extension of the
ConfigLoader
class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.The easiest way to use this class is by registering it into the
KedroContext
using hooks. This can be done by updating the hook implementation register_config_loader in hooks.py, making it return aTemplatedConfigLoader
object instead of aConfigLoader
object.Example:
from kedro.config import TemplatedConfigLoader class ProjectHooks: @hook_impl def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader: return TemplatedConfigLoader( conf_paths, globals_pattern="*globals.yml", globals_dict={"param1": "pandas.CSVDataSet"} )
The contents of the dictionary resulting from the globals_pattern get merged with the
globals_dict
. In case of conflicts, the keys inglobals_dict
take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, aValueError
will be raised.Global parameters can be namespaced as well. An example could work as follows:
globals.yml
bucket: "my_s3_bucket" environment: "dev" datasets: csv: "pandas.CSVDataSet" spark: "spark.SparkDataSet" folders: raw: "01_raw" int: "02_intermediate" pri: "03_primary" fea: "04_feature"
catalog.yml
raw_boat_data: type: "${datasets.spark}" filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv" file_format: parquet raw_car_data: type: "${datasets.csv}" filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"
This uses
jmespath
in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.Methods
get
(*patterns)Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.-
__init__
(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶ Instantiate a
TemplatedConfigLoader
.- Parameters
conf_paths (
Union
[str
,Iterable
[str
]]) – Non-empty path or list of paths to configuration directories.globals_pattern (
Optional
[str
]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.globals_dict (
Optional
[Dict
[str
,Any
]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, theglobals_dict
keys take precedence.
-
get
(*patterns)[source]¶ Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.- Parameters
*patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
- Return type
Dict
[str
,Any
]- Returns
A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details).
- Raises
ValueError – malformed config found.
-