Welcome to Kedro’s API docs and tutorials!

Introduction

• Introduction
  • What is Kedro?
  • Learning about Kedro
  • Assumptions

Getting Started

• Installation prerequisites
  • macOS / Linux
  • Windows
  • Python virtual environments
• Installation guide
• Creating a new project
  • Create a new project interactively
  • Create a new project from a configuration file
  • Starting with an existing project
  • Install project dependencies
• A “Hello World” example
  • Project directory structure
  • Project source code
  • Project components
  • Data
  • Example pipeline
  • Configuration
  • Running the example
  • Summary

Tutorial

• Typical Kedro workflow
  • Development workflow
  • Git workflow
• Kedro Spaceflights tutorial
  • Creating the tutorial project
• Setting up the data
  • Adding your datasets to data
  • Reference all datasets
  • Creating custom datasets
• Creating a pipeline
  • Node basics
  • Assemble nodes into a pipeline
  • Persisting pre-processed data
  • Creating a master table
  • Working with multiple pipelines
  • Partial pipeline runs
  • Using decorators for nodes and pipelines
  • Kedro runners
• Packaging a project
  • Add documentation to your project
  • Package your project
  • Manage project dependencies
  • Extend your project
  • What is next?

User Guide

• Setting up Visual Studio Code
  • Advanced: For those using venv / virtualenv
  • Setting up tasks
  • Debugging
• Setting up PyCharm
  • Set up Run configurations
  • Debugging
  • Advanced: Remote SSH interpreter
• Configuration
  • Local and base configuration
  • Loading
  • Additional configuration environments
  • Templating configuration
  • Parameters
  • Credentials
  • Configuring kedro run arguments
• The Data Catalog
  • Using the Data Catalog within Kedro configuration
  • Specifying the location of the dataset
  • Using the Data Catalog with the YAML API
  • Adding parameters
  • Feeding in credentials
  • Loading multiple datasets that have similar configuration
  • Transcoding datasets
  • Transforming datasets
  • Versioning datasets and ML models
  • Using the Data Catalog with the Code API
  • Configuring a Data Catalog
  • Loading datasets
  • Saving data
• Nodes
  • Creating a pipeline node
  • Tagging nodes
  • Running nodes
• Pipelines
  • Building pipelines
  • Developing modular pipelines
  • Connecting existing pipelines
  • Using a modular pipeline twice
  • Bad pipelines
  • Running pipelines
  • Running pipelines with IO
  • Outputting to a file
  • Partial pipelines
• Logging
  • Configure logging
  • Use logging
  • Logging for anyconfig
• Advanced IO
  • Error handling
  • AbstractDataSet
  • Versioning
  • Partitioned dataset
• Working with PySpark
  • Initialising a SparkSession
  • Creating a SparkDataSet
  • Working with PySpark and Kedro pipelines
• Developing Kedro plugins
  • Overview
  • Initialisation
  • global and project commands
  • Working with click
  • Contributing process
  • Example of a simple plugin
  • Supported plugins
  • Community-developed plugins
• Working with IPython and Jupyter Notebooks / Lab
  • Startup script
  • Working with context
  • Adding global variables
  • Working with IPython
  • Working from Jupyter
  • Extras
• Working with Databricks
  • Databricks Connect (recommended)
  • GitHub workflow with Databricks
• Journal
  • Overview
  • Steps to manually reproduce your code and run the previous pipeline
• Creating a new dataset
  • Scenario
  • Project setup
  • Problem
  • The anatomy of a dataset
  • Implement the _load method with fsspec
  • Implement the _save method with fsspec
  • Implement the _describe method
  • Bringing it all together
  • Integrating with PartitionedDataSet
  • Adding Versioning
  • Thread-safety consideration
  • Handling credentials and different filesystems
  • Contribute your dataset to Kedro

Resources

• Frequently asked questions
  • What is Kedro?
  • What are the primary advantages of Kedro?
  • How does Kedro compare to other projects?
  • How can I find out more about Kedro?
  • What is data engineering convention?
  • What version of Python does Kedro use?
  • How do I upgrade Kedro?
  • What best practice should I follow to avoid leaking confidential data?
  • What is the philosophy behind Kedro?
  • Where do I store my custom editor configuration?
  • How do I look up an API function?
  • How do I build documentation for my project?
  • How do I build documentation about Kedro?
• Kedro architecture overview
  • Building blocks
• Guide to CLI commands
  • Autocomplete
  • Global Kedro commands
  • Project-specific Kedro commands
  • Using Python
• Linting your Kedro project
• Images and icons
  • White background
  • Black background

API Docs

kedro

Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Indices and tables

• Index
• Module Index
• Search Page