Description
I want resource(s) to be added/updated to catalog record in automated way when processing of data finished so that the dataset catalog record will be updated by new metadata about resource(s) created/updated without human assistance.
Preconditions
- at least one catalog record is created in catalog
- pipeline is defined to update dataset resources
- source data is available
Acceptance criteria
- pipeline is executed
- pipeline finished processing without error (to be more precise: L-Catalog DPU invoking update of resources in the catalog finished successfully; these two things (successful run of L-Catalog DPU and of the whole pipeline) are typically equivalent, as L-Catalog DPU will be typically the last DPU on the pipeline)
- new resource(s) and metadata about resource(s) are stored in ODN/storage when such resource(s) are not available
- metadata about resource(s) are updated when such resource(s) are already defined
- pipeline finished with error
- in this case, resource was not updated (as long as the L-Catalog DPU did not finish successfully or was not even executed, because there was an error before). The pipeline must be rerun and the resource is created/update if and only if L-Catalog DPU finishes successfully)
Requirements
F-ODN/uv-dpu-1.1,
F-ODN/uv-dpu-1.1.X
- F-ODN/uv-dpu-1.2.
- F-ODN/uv-dpu-1.2.X
- F-ODN/uv-dpu-1.3
- F-ODN/uv-dpu-1.4
- F-ODN/uv-dpu-1.4.X
F-ODN/ic-2.1
- F-ODN/ic-2.2
- F-ODN/ic-2.4
- F-ODN/ic-2.4.1
- F-ODN/ic-2.4.2
- F-ODN/ic-2.7
- F-ODN/ic-2.5
- F-ODN/ic-2.6
Notes
- define, how to maintain last modification time of resource - Last modification time is produces by loaders on the pipelines
- define what to do when pipeline finished with an error.