Description
I want resource(s) to be added/updated to catalog record in automated way when processing of data finished so that the dataset catalog record will be updated by new metadata about resource(s) created/updated without human assistance.
Preconditions
- at least one catalog record is created in catalog
- pipeline is defined to update dataset resources
- source data is available
How to do it:
- pipeline is executed
- pipeline finished processing without error (to be more precise: L-Catalog DPU invoking update of resources in the catalog finished successfully; these two things (successful run of L-Catalog DPU and of the whole pipeline) are typically equivalent, as L-Catalog DPU will be typically the last DPU on the pipeline)
- new resource(s) and metadata about resource(s) are stored in ODN/storage when such resource(s) are not available
- metadata about resource(s) are updated when such resource(s) are already defined
- pipeline finished with error
- in this case, resource was not updated (as long as the L-Catalog DPU did not finish successfully or was not even executed, because there was an error before). The pipeline must be rerun and the resource is created/update if and only if L-Catalog DPU finishes successfully)
Note
50