Batch command to extract position of wildfire photos

[markz]

Hello,
I would appreciate help defining a DOS Batch command to extract a text string from a series of XML files, and place those strings as comma separated value on separate lines in a single .CSV file:

PhotoPositions.CSV
Filename1.XML,LatitudeDeg,LongitudeDeg,AltitudeMeters
Filename2.XML,LatitudeDeg,LongitudeDeg,AltitudeMeters
Filename3.XML,LatitudeDeg,LongitudeDeg,AltitudeMeters
etc.

The string desired is the number values in the following portion of the XML file (entire file below)
-<ImagePos>
-<LatLonAlt>
<LatitudeDeg>38.2104710594137</LatitudeDeg>
<LongitudeDeg>-120.526212270026</LongitudeDeg>
<AltitudeMeters>620.315856933594</AltitudeMeters>

This is to create an input file of photo positions for Pix4D. Each thermal infrared photo (possibly hundreds) has an associated XML file from which extracted strings need to become one line in a .CSV.

This will be used to stitch together ortho-mosaics from many infrared photos of wildfires (video frame-grabs), and then trace fire perimeters for Incident Command teams.

Thanks for any help, or to point me to a source of info,
Mark Zaller
Wildfire Pilot
markz@courtneyaviation.com

sample XML file created automatically with each infrared photo:
<!DOCTYPE Vehicle>
<Vehicle>
<VehicleId>4294967295</VehicleId>
<GimbalState>
<PanTiltRollAngles>
<Pan>0.000872665</Pan>
<Tilt>-1.39626</Tilt>
<Roll>0</Roll>
</PanTiltRollAngles>
<GpsData>
<Time>
<Year>2016</Year>
<Month>6</Month>
<Day>28</Day>
<Hour>19</Hour>
<Minute>18</Minute>
<Seconds>24.67</Seconds>
</Time>
<NumSatsTracked>9</NumSatsTracked>
<Status>59392</Status>
<PDOP>1.6</PDOP>
<GimbalPos>
<LatLonAlt>
<LatitudeDeg>38.1709177777778</LatitudeDeg>
<LongitudeDeg>-120.549282222222</LongitudeDeg>
<AltitudeMeters>1711.23999023438</AltitudeMeters>
<AltitudeType>1</AltitudeType>
</LatLonAlt>
</GimbalPos>
<GroundSpeed>
<VectorNED>
<North>7.73000001907349</North>
<East>48.1199989318848</East>
<Down>0.280000001192093</Down>
</VectorNED>
</GroundSpeed>
</GpsData>
<CameraEulerAngles>
<Euler>
<Roll>0.201364621520042</Roll>
<Pitch>-1.45399701595306</Pitch>
<Yaw>1.21607315540314</Yaw>
</Euler>
</CameraEulerAngles>
<GimbalBodyEulerAngles>
<Euler>
<Roll>0.0233999993652105</Roll>
<Pitch>-0.0599999986588955</Pitch>
<Yaw>1.41390001773834</Yaw>
</Euler>
</GimbalBodyEulerAngles>
<GimbalModeSettings>
<GimbalMode>17</GimbalMode>
</GimbalModeSettings>
<GimbalVersion>
<SerialNumber>663</SerialNumber>
<FirmwareMajor>2</FirmwareMajor>
<FirmwareMinor>2</FirmwareMinor>
<FirmwareSub>2</FirmwareSub>
<FirmwareReleased>1</FirmwareReleased>
<FirmwareDate>10/29/2013</FirmwareDate>
<HardwareRevision>65</HardwareRevision>
<HardwareConfig>33980419</HardwareConfig>
<HardwareDate>11/25/2014</HardwareDate>
<CameraATypeId>10</CameraATypeId>
<CameraBTypeId>21</CameraBTypeId>
<CameraCTypeId>0</CameraCTypeId>
<CameraDTypeId>0</CameraDTypeId>
<FirmwarePatch>1</FirmwarePatch>
<FirmwareDesc></FirmwareDesc>
<LensMajor>0</LensMajor>
<LensMinor>0</LensMinor>
<LensSub>0</LensSub>
<LensReleased>0</LensReleased>
<LensDate></LensDate>
<LensPatch>0</LensPatch>
<LensDesc></LensDesc>
</GimbalVersion>
<SPOI>
<LatLonAlt>
<LatitudeDeg>-90</LatitudeDeg>
<LongitudeDeg>-180</LongitudeDeg>
<AltitudeMeters>-30000</AltitudeMeters>
<AltitudeType>0</AltitudeType>
</LatLonAlt>
</SPOI>
<ImagePos>
<LatLonAlt>
<LatitudeDeg>38.1713978007073</LatitudeDeg>
<LongitudeDeg>-120.547633701498</LongitudeDeg>
<AltitudeMeters>402.219909667969</AltitudeMeters>
<AltitudeType>1</AltitudeType>
</LatLonAlt>
</ImagePos>
<CorrectedImagePos>
<LatLonAlt>
<LatitudeDeg>-90</LatitudeDeg>
<LongitudeDeg>-180</LongitudeDeg>
<AltitudeMeters>-30000</AltitudeMeters>
<AltitudeType>0</AltitudeType>
</LatLonAlt>
</CorrectedImagePos>
<NavFilterHealth>
<NavFilterMode>2</NavFilterMode>
<NavFilterPosition>1</NavFilterPosition>
<NavFilterVelocity>1</NavFilterVelocity>
<NavFilterHeading>1</NavFilterHeading>
<NavFilterAttitude>1</NavFilterAttitude>
<NavFilterGyroBias>1</NavFilterGyroBias>
<NavFilterAccBias>1</NavFilterAccBias>
<NavFilterMagBias>1</NavFilterMagBias>
</NavFilterHealth>
<BoardTemp>36</BoardTemp>
<DeployState>10</DeployState>
</GimbalState>
<GimbalRates>
<GimbalTime>2420517</GimbalTime>
<AxisRates>
<Pan>-0.00837758</Pan>
<Tilt>0.0174533</Tilt>
<Roll>0</Roll>
</AxisRates>
<MountRates>
<Pan>0.0106465</Pan>
<Tilt>0.00610865</Tilt>
<Roll>0.0612611</Roll>
</MountRates>
<GimbalModeSettings>
<GimbalMode>17</GimbalMode>
</GimbalModeSettings>
</GimbalRates>
<CameraState>
<ActiveCamera>1</ActiveCamera>
<HorizontalFOV>0.1839</HorizontalFOV>
<VerticalFOV>0.1379</VerticalFOV>
<ZoomRatio>1</ZoomRatio>
<MaxFOV0>0.971974</MaxFOV0>
<MinFOV0>0.0338594</MinFOV0>
<MaxFOV1>0.183958</MaxFOV1>
<MinFOV1>0.183958</MinFOV1>
<MaxFOV2>0.418879</MaxFOV2>
<MinFOV2>0.0232711</MinFOV2>
<MaxFOV3>0.418879</MaxFOV3>
<MinFOV3>0.0232711</MinFOV3>
<FpaTemp>26</FpaTemp>
<CaseTemp>-128</CaseTemp>
<SceneIntesity>0</SceneIntesity>
</CameraState>
<Laser00 Wavelength="0" Classification="0" MinAgl="0" LaserType="0" LaserProtocol="0" PowerLevel="0" Id="0" BeamDivergence="0" PowerLimited="0" ArmedState="0" LaserPower="0"/>
<Laser11 Wavelength="0" Classification="0" MinAgl="0" LaserType="0" LaserProtocol="0" PowerLevel="0" Id="1" BeamDivergence="0" PowerLimited="0" ArmedState="0" LaserPower="0"/>
<Laser22 Wavelength="0" Classification="0" MinAgl="0" LaserType="0" LaserProtocol="0" PowerLevel="0" Id="2" BeamDivergence="0" PowerLimited="0" ArmedState="0" LaserPower="0"/>
<Laser33 Wavelength="0" Classification="0" MinAgl="0" LaserType="0" LaserProtocol="0" PowerLevel="0" Id="3" BeamDivergence="0" PowerLimited="0" ArmedState="0" LaserPower="0"/>
</Vehicle>

[Aacini]

You missed to show us an example of the output file.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set "i=0"
(for /F "tokens=1-3 delims=:<>" %%a in ('findstr "ImagePos LatitudeDeg LongitudeDeg AltitudeMeters" *.xml') do (
   if !i! gtr 0 (
      set /A i+=1
      set "value[!i!]=%%c"
      if !i! equ 4 (
         echo !filename!,!value[2]!,!value[3]!,!value[4]!
         set "i=0"
      )
   ) else if "%%b" equ "ImagePos" (
      set "i=1"
      set "filename=%%a"
   )
)) > PhotoPositions.CSV

This is the output:

Code: Select all

Filename1.xml,38.1713978007073,-120.547633701498,402.219909667969

EDIT - Slightly simpler version:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set "output="
(for /F "tokens=1-3 delims=:<>" %%a in ('findstr "ImagePos LatitudeDeg LongitudeDeg AltitudeMeters" *.xml') do (
   if defined output (
      set "output=!output!,%%c"
      if "%%b" equ "AltitudeMeters" (
         echo !output!
         set "output="
      )
   ) else if "%%b" equ "ImagePos" (
      set "output=%%a"
   )
)) > PhotoPositions.CSV

Antonio

[Compo]

You could use powershell and run it from a batch file.

RunPSfile.cmd

Code: Select all

@PowerShell -ExecutionPolicy Bypass -File "LatLonAlt.ps1">%~dp0PhotoPositions.CSV

LatLonAlt.ps1

Code: Select all

$Path = "C:\Users\Mark\WildFire"
$Files = Get-ChildItem $Path -Filter *.xml
ForEach ($File in $Files) {
   [XML] $Content = Get-Content $File.FullName
   $NodeList = $Content.SelectNodes("/Vehicle/GimbalState/ImagePos/LatLonAlt")
   ForEach ($SelNode in $NodeList) {
      $Lat = $SelNode.SelectSingleNode("LatitudeDeg").get_InnerXml()
      $Long = $SelNode.SelectSingleNode("LongitudeDeg").get_InnerXml()
      $Alt = $SelNode.SelectSingleNode("AltitudeMeters").get_InnerXml()
      Write-Host "$File,$Lat,$Long,$Alt"
   }
}

Just edit the first line with your real parent directory path

[foxidrive]

I've used the characteristics of the xml file in my code below, as mentioned in this description of how batch code can be simplified: viewtopic.php?f=3&t=6108

Code: Select all

@echo off
setlocal enabledelayedexpansion
(for %%a in (*.xml) do (
   for /f "tokens=1,2 delims=<>" %%c in ('findstr "LatitudeDeg LongitudeDeg AltitudeMeters" "%%a" ^|find "." ' ) do (
   set %%c=%%d
   )
echo "%%a",!LatitudeDeg!,!LongitudeDeg!,!AltitudeMeters!
))>"file.csv"

[markz]

Thank-you everyone, very much. I'm new to PowerScript, but I got your scripts to work, somewhat. Here are notes on the three different versions:

Summary:
#1 from Aacini in Mexico City: Both the first and simplified produced errors, and no PhotoPositions.CSV file

#2 from Compo: This worked the closest. Running the script displayed the right looking content to the screen, but not to a file. Please help me understand how to redirect what scrolls by on the screen to a filename.csv.

#3 from foxdrive: produced errors and a file. The resulting file had filename.xml, 3 commas, but no data

I tried attaching a ZIP of the XML files, but I'm not sure that worked. Let me know how to make a samples of the files available if XMLfiles.zip is not attached.

Aacini asked about the output file. That is "PhotoPositions.CSV" in my original message.

Details:
#1 Error from the non simple version
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:1 char:7
+ @echo off
+ ~~~
Unexpected token 'off' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:7 char:18
+ set /A i+=1
+ ~
Missing closing ')' in expression.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:8 char:27
+ set "value[!i!]=%%c"
+ ~
Missing closing ')' in expression.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:10 char:58
+ echo !filename!,!value[2]!,!value[3]!,!value[4]!
+ ~
Missing closing ')' in expression.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:13 char:4
+ ) else if "%%b" equ "ImagePos" (
+ ~
Unexpected token ')' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:13 char:13
+ ) else if "%%b" equ "ImagePos" (
+ ~
Missing '(' after 'if' in if statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:13 char:20
+ ) else if "%%b" equ "ImagePos" (
+ ~~~
Unexpected token 'equ' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:14 char:16
+ set "i=1"
+ ~
Missing closing ')' in expression.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:16 char:4
+ )
+ ~
Unexpected token ')' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:17 char:1
+ )) > PhotoPositions.CSV
+ ~
Unexpected token ')' in expression or statement.
Not all parse errors were reported. Correct the reported errors and try again.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : UnexpectedToken


#2 Beautifully displayed every filename.xml,lat,long,alt on the screen (20 lines in my test), but couldn't get any data into a filename.csv


#3 Errors while run from the directory where the data resided since I was not sure how to put in a path:
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:1 char:7
+ @echo off
+ ~~~
Unexpected token 'off' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:6 char:5
+ )
+ ~
Missing closing ')' in expression.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:8 char:2
+ ))>"file.csv"
+ ~
Unexpected token ')' in expression or statement.
At C:\Users\admin\Documents\batch dos xml\dos1not simple.ps1:1 char:1
+ @echo off
+ ~~~~~
The splatting operator '@' cannot be used to reference variables in an expression. '@echo' can be used only as an
argument to a command. To reference variables in an expression use '$echo'.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : UnexpectedToken

results: file.csv (missing data)
"16_06_28_12_26_51_119.xml",,,
"16_06_28_12_26_54_088.xml",,,
"16_06_28_12_26_57_758.xml",,,
"16_06_28_12_27_01_628.xml",,,
"16_06_28_12_27_05_234.xml",,,
"16_06_28_12_27_08_035.xml",,,
"16_06_28_12_27_10_337.xml",,,
"16_06_28_12_27_13_340.xml",,,
"16_06_28_12_27_16_477.xml",,,
"16_06_28_12_27_20_012.xml",,,
"16_06_28_12_27_22_849.xml",,,
"16_06_28_12_27_25_886.xml",,,
"16_06_28_12_27_28_455.xml",,,
"16_06_28_12_27_30_924.xml",,,
"16_06_28_12_27_33_827.xml",,,
"16_06_28_12_27_37_896.xml",,,
"16_06_28_12_27_41_902.xml",,,
"16_06_28_12_27_44_305.xml",,,
"16_06_28_12_27_47_840.xml",,,
"16_06_28_12_27_52_344.xml",,,


Thanks again,
-MarkZ

Mod Edit: Removed email address

[markz]

The script from Compo mostly works, the others had errors. Compo's script displayed exactly the right data, but wouldn't save a file, so I cut and pasted. It worked perfectly with Pix4D. Is there a way to save to a filename.csv?

Thank-you very much to everyone. This could be helpful for wildfire fighting. If you're curious where this will be employed here is a youtube I made from a wildfire several years ago. https://www.youtube.com/watch?v=X5VG-h8eIzA

Thanks again,
-MarkZ

[Aacini]

@markz,
Some notes about your reply:

#1 from Aacini in Mexico City: Both the first and simplified produced errors, and no PhotoPositions.CSV file

This is part of my answer:

This is the output:

Code: Select all

Filename1.xml,38.1713978007073,-120.547633701498,402.219909667969

This means that I run my code and it produced the PhotoPositions.CSV file with the output shown, so the point is: how did you run my code? You should created a folder with just two files: my code in a Batch file and your data example in a file called Filename1.xml; this should create the same output file (unless your computer and the mine don't works in the same way).

After this test you should copy the Batch file to the folder that contain the *.xml files and run it; this should produce the desired result, unless a new/different problem exist with your real data that is not present in the example data...

__________________________________________________________

Aacini asked about the output file. That is "PhotoPositions.CSV" in my original message.

Not exactly:

You missed to show us an example of the output file.

Your example data contain all these lines with the requested fields:

Code: Select all

<LatitudeDeg>38.1709177777778</LatitudeDeg>
<LongitudeDeg>-120.549282222222</LongitudeDeg>
<AltitudeMeters>1711.23999023438</AltitudeMeters>
<LatitudeDeg>-90</LatitudeDeg>
<LongitudeDeg>-180</LongitudeDeg>
<AltitudeMeters>-30000</AltitudeMeters>
<LatitudeDeg>38.1713978007073</LatitudeDeg>
<LongitudeDeg>-120.547633701498</LongitudeDeg>
<AltitudeMeters>402.219909667969</AltitudeMeters>
<LatitudeDeg>-90</LatitudeDeg>
<LongitudeDeg>-180</LongitudeDeg>
<AltitudeMeters>-30000</AltitudeMeters>

Which ones are the desired values? The ones below "<ImagePos>" tag of course, but this involves to open the file and search for the tag. If you would just posted:

"The desired output file is:"

Code: Select all

Filename1.xml,38.1713978007073,-120.547633701498,402.219909667969

... this would be easier...

Antonio

[Compo]

The script from Compo mostly works, the others had errors. Compo's script displayed exactly the right data, but wouldn't save a file, so I cut and pasted. It worked perfectly with Pix4D. Is there a way to save to a filename.csv?

You could use powershell and run it from a batch file.

RunPSfile.cmd

Code: Select all

@PowerShell -ExecutionPolicy Bypass -File "LatLonAlt.ps1">%~dp0PhotoPositions.CSV

The above works only if you're running the cmd file from the same directory as the ps1. The csv output file will be found in the same directory as the cmd file!

If you ran the command from the commandline your missing output csv would probably be found in %userprofile%

To make sure place LatLonAlt.ps1 and RunPSfile.cmd on your desktop, make sure the correct path was entered in the first line of the ps1 file, then just double-click RunPSfile.cmd, on your desktop PhotoPositions.CSV should quickly appear.

You could also replace with a full path

Code: Select all

@PowerShell -ExecutionPolicy Bypass -File "C:\Users\Mark\Desktop\LatLonAlt.ps1">%~dp0PhotoPositions.CSV

[foxidrive]

Thank-you everyone, very much. I'm new to PowerScript, but I got your scripts to work, somewhat. Here are notes on the three different versions:

Summary:
#1 from Aacini in Mexico City: Both the first and simplified produced errors, and no PhotoPositions.CSV file

#2 from Compo: This worked the closest. Running the script displayed the right looking content to the screen, but not to a file. Please help me understand how to redirect what scrolls by on the screen to a filename.csv.

#3 from foxdrive: produced errors and a file. The resulting file had filename.xml, 3 commas, but no data

Did you try the exact data that you provided as a sample? if you did then what error message appeared when you tried them.

I tested the code that i provided and it worked flawlessly here, so there is likely to be an issue with the OS version or permissions or the way the script was launched.

[foxidrive]

The string desired is the number values in the following portion of the XML file (entire file below)
-<ImagePos>
-<LatLonAlt>
<LatitudeDeg>38.2104710594137</LatitudeDeg>
<LongitudeDeg>-120.526212270026</LongitudeDeg>
<AltitudeMeters>620.315856933594</AltitudeMeters>

I just realised that your values above don't appear in your sample data. I used the header information you showed above to locate the strings but they are different numbers than the ones you have shown.

The example file you showed is clearly a different file and your files may all have differing data structures and which is why the scripts failed.

[markz]

We are deploying to a fire tomorrow in Yosemite, so I've only had time to work on Compo's code. As well I am very slow as a neophyte.

1) the batch file version runs on my PC and creates the CSV. On the command center's PC the batch file will only run while on my thumb drive. When the .bat and .ps1 are copied to the local hard drive sub-directory, the batch file fails saying "access denied" (i had to turn Echo On and put in pause to see this). I think this is due to security on the PC. I ran the batch as administrator and tried to open up the security, but no luck. When loading the .ps1 into Powershell and clicking Run, the script runs as usual painting the lat,long,alt to the screen as before, and as before does not write a filename.csv. Any ideas on how to run the script on the PC's local drive? Is there a difference between using Powershell ISA vs no ISE? We accidentally copied files to the Desktop of a PC once, and the filename.csv did get created, so maybe there is a sub-directory file permissions problem.

2) I incorrectly described using the filename.XML as the first comma delimited item. Actually filename.JPG is desired. Is there a way to read the XML filename and then write it to the CSV-file as filename.jpg? The easy work around that worked, is to open the just created CSV in Notepad, and do a global replace XML -> JPG.

-MarkZ