run_05_overlap_geo2rdr "FileNotFoundError" because of missing date in`secondarys`: fix by checking for "junk" in run_02*.o?

[falkamelung]

For the problem date no entry is produced in secondarys causing the run_05 "FileNotFoundError" . It turns out that run run_05*.o contains the strings "junk after document element". Eliminating files containing this string may solve this issue.
For

minsarApp.bash -1.714:-1.682,101.25:101.283 iMissDataKerinci --flight-dir desc --start-date 20240701 --end-date 20240930 --no-mintpy --miaplpy

I get the error:

---------------------------------------------------------------------------------------------------------------------------------------------------------
| run_05_ove...r_0.job | 4     | 0/2000    | 2/3000    | 1/1       | 1/4     | Submitted: 3087296                                                     |
---------------------------------------------------------------------------------------------------------------------------------------------------------
Jobs submitted: 3087296
iMissDataKerinciSenD91, run_05_overlap_geo2rdr, 1 jobs : 1 COMPLETED , 0 RUNNING , 0 PENDING , 0 WAITING   .
check_job_outputs.py  /scratch/05861/tg851601/iMissDataKerinciSenD91/run_files/run_05_overlap_geo2rdr_0.job
checking *.e, *.o from /scratch/05861/tg851601/iMissDataKerinciSenD91/run_files/run_05_overlap_geo2rdr_0.job
Error: "FileNotFoundError" found in /scratch/05861/tg851601/iMissDataKerinciSenD91/run_files/run_05_overlap_geo2rdr_0_20240819_1.e
Error: "Traceback" found in /scratch/05861/tg851601/iMissDataKerinciSenD91/run_files/run_05_overlap_geo2rdr_0_20240819_1.e
For known issues see https://github.com/geodesymiami/minsar/tree/master/docs/known_issues.md
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/code/minsar/minsar/scripts/check_job_outputs.py", line 287, in <module>
    main()
  File "/work2/05861/tg851601/stampede2/code/minsar/minsar/scripts/check_job_outputs.py", line 240, in main
    raise RuntimeError('Error in run_file: ' + run_file_base)

It turns out that

lsd | grep SAFE
S1A_IW_SLC__1SSV_20240807T225739_20240807T225739_055113_06B737_3D13.SAFE/
S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/
S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/
S1A_IW_SLC__1SSV_20240912T225740_20240912T225740_055638_06CAF7_05AE.SAFE/
S1A_IW_SLC__1SSV_20240924T225740_20240924T225740_055813_06D1E1_0DC9.SAFE/

while we have:

get_reference_date and  ls -1 secondarys/
20240807
--- missing file:  20240819 caused the error
20240831
20240912
20240924

A good file produces

cat run_02_unpack_secondary_slc_0_20240831_2.o
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: Sentinel1_TOPS
['--dirname', '/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE', '--swaths', '1 2 3', '--orbitdir', '/work2/05861/tg851601/stampede2/insarlab/S1orbits', '--outdir', '/scratch/05861/tg851601/iMissDataKerinciSenD91/secondarys/20240831', '--auxdir', '/work2/05861/tg851601/stampede2/insarlab/S1aux', '--bbox', '-1.71 -1.68 98.25 104.28', '--pol', 'vv']
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/annotation/s1a-iw1-slc-vv-20240831t225739-20240831t225742-055463-06c40b-002.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE for subswath IW1
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/annotation/s1a-iw2-slc-vv-20240831t225739-20240831t225742-055463-06c40b-001.xml']
Input TIFF files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/measurement/s1a-iw2-slc-vv-20240831t225739-20240831t225742-055463-06c40b-001.tiff']
Manifest files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/manifest.safe']
Setting IPF version to :  003.80
Extracting orbit from Orbit File:  /work2/05861/tg851601/stampede2/insarlab/S1orbits/S1A_OPER_AUX_POEORB_OPOD_20240920T070634_V20240830T225942_20240901T005942.EOF
Number of Bursts before cropping:  1
Number of Bursts after cropping:  1
Updating burst number from 1 to 1
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE/annotation/s1a-iw3-slc-vv-20240831t225739-20240831t225742-055463-06c40b-003.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240831T225739_20240831T225739_055463_06C40B_0BC6.SAFE for subswath IW3
(minsar) //login3/scratch/05861/tg851601/iMissDataKerinciSenD91/run_files/stdout_run_02_unpack_secondary_slc[1064] cat run_02_unpack_secondary_slc_0_20240819_1.o
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: Sentinel1_TOPS
['--dirname', '/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE', '--swaths', '1 2 3', '--orbitdir', '/work2/05861/tg851601/stampede2/insarlab/S1orbits', '--outdir', '/scratch/05861/tg851601/iMissDataKerinciSenD91/secondarys/20240819', '--auxdir', '/work2/05861/tg851601/stampede2/insarlab/S1aux', '--bbox', '-1.71 -1.68 98.25 104.28', '--pol', 'vv']
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw1-slc-vv-20240819t225739-20240819t225742-055288-06bd96-003.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE for subswath IW1
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw2-slc-vv-20240819t225739-20240819t225742-055288-06bd96-001.xml']
Input TIFF files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/measurement/s1a-iw2-slc-vv-20240819t225739-20240819t225742-055288-06bd96-001.tiff']
Manifest files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/manifest.safe']
Setting IPF version to :  003.80
junk after document element: line 121729, column 5
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw3-slc-vv-20240819t225739-20240819t225742-055288-06bd96-002.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE for subswath IW3

While the failed file produces

cat run_02_unpack_secondary_slc_0_20240819_1.o
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: Sentinel1_TOPS
['--dirname', '/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE', '--swaths', '1 2 3', '--orbitdir', '/work2/05861/tg851601/stampede2/insarlab/S1orbits', '--outdir', '/scratch/05861/tg851601/iMissDataKerinciSenD91/secondarys/20240819', '--auxdir', '/work2/05861/tg851601/stampede2/insarlab/S1aux', '--bbox', '-1.71 -1.68 98.25 104.28', '--pol', 'vv']
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw1-slc-vv-20240819t225739-20240819t225742-055288-06bd96-003.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE for subswath IW1
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw2-slc-vv-20240819t225739-20240819t225742-055288-06bd96-001.xml']
Input TIFF files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/measurement/s1a-iw2-slc-vv-20240819t225739-20240819t225742-055288-06bd96-001.tiff']
Manifest files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/manifest.safe']
Setting IPF version to :  003.80
junk after document element: line 121729, column 5
Input XML files:  ['/scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE/annotation/s1a-iw3-slc-vv-20240819t225739-20240819t225742-055288-06bd96-002.xml']
No tiff file found in directory: /scratch/05861/tg851601/iMissDataKerinciSenD91/SLC/S1A_IW_SLC__1SSV_20240819T225739_20240819T225739_055288_06BD96_4C56.SAFE for subswath IW3

Note the text "junk after document element" in the run_5*.o file

[falkamelung]

Same for Chiles (there is 'junk' in the run_02_unpack)

#### /scratch/09580/gdisilvestro/ChilesSenD142/run_files/run_05_overlap_geo2rdr_1_20241209_143.e
#########################
Traceback (most recent call last):
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 251, in <module>
    main(args.start,args.end)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 242, in main
    cfgParser.runCmd()
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 54, in runCmd
    func_modules.main(self.funcParams[section])
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/geo2rdr.py", line 235, in main
    secondary = ut.loadProduct(os.path.join(inps.secondary, 'IW{0}.xml'.format(swath)))
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/s1a_isce_utils.py", line 26, in loadProduct
    obj = pm.loadProduct(xmlname)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/lib/python3.10/site-packages/isce/components/iscesys/Component/ProductManager.py", line 25, in loadProduct
    self.load(filename)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/lib/python3.10/site-packages/isce/components/iscesys/Component/Configurable.py", line 1407, in load
    tmpProp, tmpFact, tmpMisc = FP.parse(filename)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/lib/python3.10/site-packages/isce/components/iscesys/Parsers/XmlParser.py", line 41, in parse
    root = ET.parse(filename)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/lib/python3.10/xml/etree/ElementTree.py", line 1222, in parse
    tree.parse(source, parser)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/lib/python3.10/xml/etree/ElementTree.py", line 569, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/09580/gdisilvestro/ChilesSenD142/secondarys/20241209/IW3.xml'
#########################
#### /scratch/09580/gdisilvestro/ChilesSenD142/run_files/run_05_overlap_geo2rdr_error_matches.e
#########################
Error: "FileNotFoundError" found in /scratch/09580/gdisilvestro/ChilesSenD142/run_files/run_05_overlap_geo2rdr_1_20241209_143.e
Error: "Traceback" found in /scratch/09580/gdisilvestro/ChilesSenD142/run_files/run_05_overlap_geo2rdr_1_20241209_143.e
(base) //login3/scratch/09580/gdisilvestro/ChilesSenD142[526] cd run_files/stdout_run_02_unpack_secondary_slc/
(base)

//login3/scratch/09580/gdisilvestro/ChilesSenD142/run_files/stdout_run_02_unpack_secondary_slc[527] grep junk run_02_unpack_secondary_slc_0_*
run_02_unpack_secondary_slc_0_20241209_363.o:junk after document element: line 15454, column 0

Is this a temporary problem?

I then ran

minsarApp.bash 0.649:0.837,-78.075:-77.79 q1Chiles --start-date 20241015 --end-date 20241231 --flight-dir desc --no-mintpy —miaplpy

and it worked:

(minsar) //login3/scratch/05861/tg851601/q1ChilesSenD142[1240] ll secondarys/20241209
total 68
drwxr-xr-x 2 tg851601 G-820134  4096 May  7 20:12 IW3
-rw-r--r-- 1 tg851601 G-820134 69033 May  7 20:12 IW3.xml

[disilvestro]

Same command:

minsarApp.bash 0.649:0.837,-78.075:-77.79 q1Chiles --start-date 20241015 --end-date 20241231 --flight-dir desc --no-mintpy

Get the Error:

(base) //login4/scratch/09580/gdisilvestro/q1ChilesSenD142[586] cat out_run_01_unpack_topo_reference.e 
#########################
#### /scratch/09580/gdisilvestro/q1ChilesSenD142/run_files/run_01_unpack_topo_reference_0_reference_1.e 
#########################
Traceback (most recent call last):
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 251, in <module>
    main(args.start,args.end)
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 242, in main
    cfgParser.runCmd()
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/SentinelWrapper.py", line 54, in runCmd
    func_modules.main(self.funcParams[section])
  File "/work2/09580/gdisilvestro/stampede3/code/minsar/tools/miniforge3/envs/minsar/share/isce2/topsStack/topo.py", line 112, in main
    bbox = [np.min(boxes[:,0]), np.max(boxes[:,1]), np.min(boxes[:,2]), np.max(boxes[:,3])]
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
#########################
#### /scratch/09580/gdisilvestro/q1ChilesSenD142/run_files/run_01_unpack_topo_reference_error_matches.e 
#########################
Error: "Traceback" found in /scratch/09580/gdisilvestro/q1ChilesSenD142/run_files/run_01_unpack_topo_reference_0_reference_1.e