kedro.io.CachedDataSet¶
-
class
kedro.io.
CachedDataSet
(dataset, version=None, copy_mode=None)[source]¶ Bases:
kedro.io.core.AbstractDataSet
CachedDataSet
is a dataset wrapper which caches in memory the data saved, so that the user avoids io operations with slow storage media.You can also specify a
CachedDataSet
in catalog.yml:test_ds: type: CachedDataSet versioned: true dataset: type: pandas.CSVDataSet filepath: example.csv
Please note that if your dataset is versioned, this should be indicated in the wrapper class as shown above.
Attributes
Methods
Checks whether a data set’s output already exists by calling the provided _exists() method.
CachedDataSet.from_config
(name, config[, …])Create a data set instance using the configuration provided.
Loads data by delegation to the provided load method.
Release any cached data.
CachedDataSet.save
(data)Saves data by delegation to the provided save method.
-
__init__
(dataset, version=None, copy_mode=None)[source]¶ Creates a new instance of
CachedDataSet
pointing to the provided Python object.- Parameters
dataset (
Union
[AbstractDataSet
,Dict
]) – A Kedro DataSet object or a dictionary to cache.version (
Optional
[Version
]) – If specified, should be an instance ofkedro.io.core.Version
. If itsload
attribute is None, the latest version will be loaded. If itssave
attribute is None, save version will be autogenerated.copy_mode (
Optional
[str
]) – The copy mode used to copy the data. Possible values are: “deepcopy”, “copy” and “assign”. If not provided, it is inferred based on the data type.
- Raises
ValueError – If the provided dataset is not a valid dict/YAML representation of a dataset or an actual dataset.
-
exists
()¶ Checks whether a data set’s output already exists by calling the provided _exists() method.
- Return type
bool
- Returns
Flag indicating whether the output already exists.
- Raises
DataSetError – when underlying exists method raises error.
-
classmethod
from_config
(name, config, load_version=None, save_version=None)¶ Create a data set instance using the configuration provided.
- Parameters
name (
str
) – Data set name.config (
Dict
[str
,Any
]) – Data set config dictionary.load_version (
Optional
[str
]) – Version string to be used forload
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.save_version (
Optional
[str
]) – Version string to be used forsave
operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
- Return type
AbstractDataSet
- Returns
An instance of an
AbstractDataSet
subclass.- Raises
DataSetError – When the function fails to create the data set from its config.
-
load
()¶ Loads data by delegation to the provided load method.
- Return type
Any
- Returns
Data returned by the provided load method.
- Raises
DataSetError – When underlying load method raises error.
-
release
()¶ Release any cached data.
- Raises
DataSetError – when underlying release method raises error.
- Return type
None
-
save
(data)¶ Saves data by delegation to the provided save method.
- Parameters
data (
Any
) – the value to be saved by provided save method.- Raises
DataSetError – when underlying save method raises error.
- Return type
None
-