kedro.config.ConfigLoader¶

class kedro.config.ConfigLoader(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local')[source]¶

Recursively scan directories (config paths) contained in conf_source for configuration files with a yaml, yml, json, ini, pickle, xml or properties extension, load them, and return them in the form of a config dictionary.

The first processed config path is the base directory inside conf_source. The optional env argument can be used to specify a subdirectory of conf_source to process as a config path after base.

When the same top-level key appears in any 2 config files located in the same (sub)directory, a ValueError is raised.

When the same key appears in any 2 config files located in different (sub)directories, the last processed config path takes precedence and overrides this key.

For example, if your conf_source looks like this:

.
`-- conf
    |-- README.md
    |-- base
    |   |-- catalog.yml
    |   |-- logging.yml
    |   `-- experiment1
    |       `-- parameters.yml
    `-- local
        |-- catalog.yml
        |-- db.ini
        |-- experiment1
        |   |-- parameters.yml
        |   `-- model_parameters.yml
        `-- experiment2
            `-- parameters.yml

You can access the different configurations as follows:

import logging.config
from kedro.config import ConfigLoader
from kedro.framework.project import settings

conf_path = str(project_path / settings.CONF_SOURCE)
conf_loader = ConfigLoader(conf_source=conf_path, env="local")

conf_logging = conf_loader["logging"]
logging.config.dictConfig(conf_logging)  # set logging conf

conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]

Attributes

conf_paths

Property method to return deduplicated configuration paths.

Methods

`clear`()
`copy`()
`fromkeys`(iterable[, value])
`get`(*patterns)	Retrieve the corresponding layout by the string key.
`items`()
`keys`()
`pop`(k[,d])	If key is not found, d is returned if given, otherwise KeyError is raised.
`popitem`()	as a 2-tuple; but raise KeyError if D is empty.
`setdefault`(k[,d])
`update`([E, ]**F)	If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
`values`()

__init__(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local')[source]¶

Instantiates a ConfigLoader.

Parameters

conf_source (str) – Path to use as root directory for loading configuration.
env (Optional[str]) – Environment that will take precedence over base.
runtime_params (Optional[Dict[str, Any]]) – Extra parameters passed to a Kedro run.
config_patterns (Optional[Dict[str, List[str]]]) – Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in CONFIG_LOADER_ARGS in settings.py.
base_env (str) – Name of the base environment. Defaults to “base”. This is used in the conf_paths property method to construct the configuration paths.
default_run_env (str) – Name of the base environment. Defaults to “local”. This is used in the conf_paths property method to construct the configuration paths. Can be overriden by supplying the env argument.

clear() → None. Remove all items from D.¶

property conf_paths¶: Property method to return deduplicated configuration paths.

copy()¶

classmethod fromkeys(iterable, value=None)¶

get(*patterns)[source]¶

Retrieve the corresponding layout by the string key.

When there isn’t an exact match, all the existing keys in the layout map will be treated as a regex and map against the input key again. The first match will be returned, based on the key insertion order. Return None if there isn’t any match found.

Parameters: key – the string key as the query for the layout.
Return type: Dict[str, Any]
Returns: Corresponding layout based on the query.

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

pop(k[, d]) → v, remove specified key and return the corresponding value.¶: If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair¶: as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D¶

update([E, ]**F) → None. Update D from mapping/iterable E and F.¶: If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D’s values¶