Scripting

HxMap allows the user to extend the workflow by adding custom user scripts to be run after HxMap processes finish.

Run Script after Ingest

When creating Ingest Jobs, HxMap will look for a script in the same directory as the workflow manager executable (bin directory) that can be used to customize the Ingest workflow. The script needs to be named “hxmap-ingest-external.sh” or “hxmap-ingest-external.bat” depending on the platform used.

If such script is found, HxMap will add external jobs and execute them after the ingest jobs have completed. For cluster runs, HxMap will uses the HTCondor SUBDAG functionality to run the external jobs.

For local runs, HxMap will just treat the script just like any other command line job to run after Ingest.

• During Ingest Job submission

  • External DAG preparer script Job will be added to the HxMap Ingest DAG as a normal Job with Dependency on the last "Ingest" sub task job

  • A "SUBDAG EXTERNAL" entry to support the script running and creating an External DAG is added to the DAG for any number of jobs to be run after "Ingest".

  • The job name of the external script is "Ingest DAG" name with the "_external_prepare" appended

  • External DAG name will be "Ingest DAG" name with the "_external" appended

  • Here is an Example DAG file that has a custom user script:

    CODE

    JOBSTATE_LOG 141228093432_ingest_jobstate.log...
    
    JOB 141228093432_ingest_external_prepare 141228093432_ingest_external_prepare.sub
    
    SUBDAG EXTERNAL 141228093432_ingest_external 141228093432_ingest_external.dag
    
    PARENT 141228093432_ingest_external_prepare CHILD 141228093432_ingest_external

• During Ingest Job Run

  • the ""hxmap-ingest-external" script is responsible for creating the External DAG that is referenced as "SUBDAG EXTERNAL"

  • HxMap Job management module copies the external scripts to the HxMap HTCondor Temp library just like it does for the “hxmap_condor_runner”

  • HTCondor will run the External DAG just like any other DAG once the "prepare" script job has run and created “*_ingest_external.dag”

  • the external script is expected to create at list a single JOB in the DAG file so that the overall Ingest job does not fail

The following parameter are passed to the “hxmap-ingest-external” script:

Example for a script structure

The linked script is called as a dependency to the last Ingest job. Inside the script, any additional jobs that need to be executed have to be captured in the SUB DAG.

List below shows an example outline of how the script could be set up:

1. parse/validate provided parameter (see list above)

2. configure environment (e.g. set up shared drives)

3. use sensor type information to trigger data type specific tasks

4. use session parameter to extract structure of processing tasks (e.g. determine flight lines inside the session)

5. create DAG with all the processing jobs (e.g. for each of the determined flight lines)

6. Clean Up (e.g. unmount drives)

7. exit script