Kedro architecture overview¶
At a high level, Kedro consists of five main parts:
As a data pipeline developer, you will interact with a Kedro project, which consists of:
conf/directory, which contains configuration for the project, such as data catalog configuration, parameters, etc.
srcdirectory, which contains the source code for the project, including:
pipelinesdirectory, which contains the source code for your pipelines.
settings.pyfile contains the settings for the project, such as library component registration, custom hooks registration, etc.
hooks.py, which contains custom Hooks implementations in the project, including both registration hooks and extension hooks.
cli.pyfile contains project specific CLI commands (e.g.,
kedro test, etc.).
pipeline_registry.pyfile defines the project pipelines, i.e. pipelines that can be run using
kedro run --pipeline.
__main__.pyfile serves as the main entry point of the project in package mode.
pyproject.tomlidentifies the project root by providing project metadata, including:
package_name: A valid Python package name for your project package
project_name: A human readable name for your project
project_version: Kedro version with which the project was generated
Kedro library consists of independent units, each responsible for one aspect of computation in a data pipeline:
Config Loaderprovides utility to parse and load configuration defined in a Kedro project.
Pipelineprovides a collection of abstractions to model data pipelines.
Runnerprovides an abstraction for different execution strategy of a data pipeline.
I/Oprovides a collection of abstractions to handle I/O in a project, including
Kedro framework serves as the interface between a Kedro project and Kedro library components. The major building blocks of the Kedro framework include:
Sessionis responsible for managing the lifecycle of a Kedro run.
Contextholds the configuration and Kedro’s main functionality, and also serves as the main entry point for interactions with core library components.
Hooksdefines all hook specifications available to extend Kedro.
CLIdefines built-in Kedro CLI commands and utilities to load custom CLI commands from plugins.