kedro.config.ConfigLoader

class kedro.config.ConfigLoader(conf_paths)

Bases: object

Recursively scan the directories specified in conf_paths for configuration files with a yaml, yml, json, ini, pickle, xml or properties extension, load them, and return them in the form of a config dictionary.

When the same top-level key appears in any 2 config files located in the same conf_path (sub)directory, a ValueError is raised.

When the same key appears in any 2 config files located in different conf_path directories, the last processed config path takes precedence and overrides this key.

For example, if your conf_path looks like this:

.
`-- conf
    |-- README.md
    |-- base
    |   |-- catalog.yml
    |   |-- logging.yml
    |   `-- experiment1
    |       `-- parameters.yml
    `-- local
        |-- catalog.yml
        |-- db.ini
        |-- experiment1
        |   |-- parameters.yml
        |   `-- model_parameters.yml
        `-- experiment2
            `-- parameters.yml

You can access the different configurations as follows:

import logging.config
from kedro.config import ConfigLoader

conf_paths = ['conf/base', 'conf/local']
conf_loader = ConfigLoader(conf_paths)

conf_logging = conf_loader.get('logging*')
logging.config.dictConfig(conf_logging)  # set logging conf

conf_catalog = conf_loader.get('catalog*', 'catalog*/**')
conf_params = conf_loader.get('**/parameters.yml')

Attributes

Methods

ConfigLoader.get(*patterns)

Recursively scan for configuration files, load and merge them, and return them in the form of a config dictionary.

__init__(conf_paths)

Instantiate a ConfigLoader.

Parameters

conf_paths (Union[str, Iterable[str]]) – Non-empty path or list of paths to configuration directories.

Raises

ValueError – If conf_paths is empty.

get(*patterns)

Recursively scan for configuration files, load and merge them, and return them in the form of a config dictionary.

Parameters

patterns (str) – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.

Raises

• ValueError – If 2 or more configuration files inside the same config path (or its subdirectories) contain the same top-level key.

• MissingConfigException – If no configuration files exist within a specified config path.

Returns

A Python dictionary with the combined

configuration from all configuration files. Note: any keys that start with _ will be ignored.

Return type

Dict[str, Any]