¶ Description
Search for all the data files required to create a final file.
¶ Parameters
¶ Parameters tab
Parameters:
- Developer mode: activate/deactivate code tab.
- Add button: Add custom parameters
- Trash button: Delete selected parameter(s)
- column with direction of data flow (input/output)
- column with pipeline name
¶ Description tab
See dedicated page for more information.
¶ Configuration tab
See dedicated page for more information.
¶ About
The dependencies action button traces the complete data lineage for a final output file. It identifies all the files and transformation steps (pipelines) involved, from the original data sources to the final result, by analyzing the direction of data flow between them.
¶ Input
This action requires an input table that defines file dependencies in terms of source (input) and destination (output) relationships.
Expected columns in the input file:
Direction: Specifies the role of each file in the data flow. Accepted values areinputoroutput.fileName: Name of the file or pipeline involved in the transformation.
Accepted format:
- CSV file, typically read with a
readCSVnode. - The column headers must match exactly:
Direction,fileName.
¶ Output
The output is a visual and tabular representation of dependencies:
- It shows the transformation path from source files to the final output file.
- You can view the result directly in the downstream nodes or logs.
¶ Parameters
| Parameter | Description | Example value |
|---|---|---|
column with direction of data flow (input/output) |
Name of the column that contains input or output. |
Direction |
column with pipeline name |
Name of the column with file/pipeline names. | fileName |
¶ How to Run
-
Prepare the input file with two columns:
Direction(values:input,output)fileName(e.g.,data.csv,pipeline_X)
Example:
Direction,fileName input,data_raw.csv output,cleaned.csv input,cleaned.csv output,final_report.csv -
Add a readCSV node to load the file.
-
Connect the output of the readCSV node to the top-left pin of the dependencies node (Input Pin 1).
-
Configure the parameters in the dependencies node:
- column with direction of data flow (input/output): Direction
- column with pipeline name: fileName
-
Run the pipeline (not just preview mode) to generate the transformation path.
-
Check the output using a downstream RunToFinishLine, or a view/log node.
