Exclamationmark in variable causes parsing-errors

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
Reino
Posts: 20
Joined: 14 May 2015 06:13
Contact:

Exclamationmark in variable causes parsing-errors

#1 Post by Reino » 21 Jun 2015 11:23

VPWON_1226681.jsonp:

Code: Select all

parseMetadata({"STATUS":"OK","VERSION":"1.11.13\n","prid":"VPWON_1226681","titel":"De IJzeren Eeuw","aflevering_titel":"Ten oorlog!","info":"Geschiedenisserie over de negentiende eeuw.","ratio":"16:9","medium":"tv","gidsdatum":"2015-06-19","tijdsduur":"00:44:06","url":"","webcast":1,"images":[{"size":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x480\/622080.jpg"},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/s720\/c720x405\/622080.jpg"}],"omroepen":[{"naam":"NTR"},{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","h264_std"],"tt888":"ja","serie":{"srid":"VPWON_1226669","serie_titel":"De IJzeren Eeuw","serie_url":null},"sitestat":{"baseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","programurl":"uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619","programurlpost":"category=uitzendinggemist&thema=documentaire&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat.com\/klo\/po\/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.de_ijzeren_eeuw","subtitleurlpost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame":"http:\/\/pubads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/deijzereneeuw&url=_url_&cust_params=genre%3Ddocumentaire%26dur%3D2646%26prid%3DVPWON_1226681%26srid%3DVPWON_1226669%26player%3D_player_","streamSense":{"episode":"de_ijzeren_eeuw","program":"de_ijzeren_eeuw","station":"nederland_2","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619","sko":"TRUE","sko_dt":"20150619","sko_pr":"de_ijzeren_eeuw","sko_stid":"2","sko_ty":"tv.ep","sko_prid":"vpwon1226681","sko_t":"2110","sko_cl":"2646"}})
//epc
The JSON object of concern: "aflevering_titel":"Ten oorlog!"

This batchscript works fine:

Code: Select all

@ECHO off

SET xidel="xidel.exe"
SET jq="jq.exe"

FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON:
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
(Xidel strips everything outside the outer brackets. Jq makes it human-readible.)

Output:

Code: Select all

JSON:
{
  "STATUS": "OK",
  "VERSION": "1.11.13\n",
  "prid": "VPWON_1226681",
  "titel": "De IJzeren Eeuw",
  "aflevering_titel": "Ten oorlog!",
  "info": "Geschiedenisserie over de negentiende eeuw.",
  "ratio": "16:9",
  "medium": "tv",
  "gidsdatum": "2015-06-19",
  "tijdsduur": "00:44:06",
  "url": "",
  "webcast": 1,
  "images": [
    {
      "size": "640x480",
      "ratio": "4:3",
      "url": "http://images.poms.omroep.nl/image/sx480/c640x480/622080.jpg"
    },
    {
      "size": "720x405",
      "ratio": "16:9",
      "url": "http://images.poms.omroep.nl/image/s720/c720x405/622080.jpg"
    }
  ],
  "omroepen": [
    {
      "naam": "NTR"
    },
    {
      "naam": "VPRO"
    }
  ],
  "pubopties": [
    "adaptive",
    "h264_bb",
    "h264_sb",
    "h264_std"
  ],
  "tt888": "ja",
  "serie": {
    "srid": "VPWON_1226669",
    "serie_titel": "De IJzeren Eeuw",
    "serie_url": null
  },
  "sitestat": {
    "baseurl": "http://b.scorecardresearch.com/p?c1=2&c2=17827132&ns_site=po-totaal",
    "programurl": "uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619",
    "programurlpost": "category=uitzendinggemist&thema=documentaire&po_source=video",
    "baseurl_subtitle": "http://nl.sitestat.com/klo/po/s",
    "subtitleurl": "uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.de_ijzeren_eeuw",
    "subtitleurlpost": "category=uitzendinggemist&po_source=video&po_sitetype=webonly"
  },
  "reclame": "http://pubads.g.doubleclick.net/gampad/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=/9233/_site_/deijzereneeuw&url=_url_&cust_params=genre%3Ddocumentaire%26dur%3D2646%26prid%3DVPWON_1226681%26srid%3DVPWON_1226669%26player%3D_player_",
  "streamSense": {
    "episode": "de_ijzeren_eeuw",
    "program": "de_ijzeren_eeuw",
    "station": "nederland_2",
    "sitestatname": "uitzendinggemist.publiekeomroep.ondemand.tv.de_ijzeren_eeuw.20150619",
    "sko": "TRUE",
    "sko_dt": "20150619",
    "sko_pr": "de_ijzeren_eeuw",
    "sko_stid": "2",
    "sko_ty": "tv.ep",
    "sko_prid": "vpwon1226681",
    "sko_t": "2110",
    "sko_cl": "2646"
  }
}
However, when I have to use ENABLEDELAYEDEXPANSION for further extraction of JSON objects, this causes the batchscript to parse ampersands because of the exclamationmark in the episode-title.

Batchscript:

Code: Select all

@ECHO off

SET xidel="xidel.exe"
SET jq="jq.exe"

SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON:
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
  ECHO.
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %xidel% -q - -e "$json/(t:=tijdsduur,ss:=start,to:=eind)" --output-format=cmd') DO %%B
  ECHO Tijdsduur: !t!
  IF DEFINED ss (
    SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
    SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
    ECHO Start:     !ss! (!_ss!s^)
    ECHO Einde:     !to! (!_to!s^)
  )
)
ENDLOCAL

Output:

Code: Select all

JSON:
'c2' is not recognized as an internal or external command,
operable program or batch file.
'ns_site' is not recognized as an internal or external command,
operable program or batch file.
'thema' is not recognized as an internal or external command,
operable program or batch file.
'po_source' is not recognized as an internal or external command,
operable program or batch file.
'po_source' is not recognized as an internal or external command,
operable program or batch file.
'po_sitetype' is not recognized as an internal or external command,
...
etc...
What can I do to prevent SETLOCAL ENABLEDELAYEDEXPANSION to see the exclamationmark as syntax for a variable?

jeb
Expert
Posts: 967
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Exclamationmark in variable causes parsing-errors

#2 Post by jeb » 21 Jun 2015 13:54

Hi CoRoNe,

normally delayed expansion doesn't have the effect you described here.
Delayed expansion can remove carets and exclamation marks from texts, but it will not activate ampersands inside quotes.

But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B

And there the delayed expansion of this part fails
"Ten oorlog!","info":"Geschiedenisseri ....

This will be expanded to
"Ten oorlog"Geschiedenisseri ....
As the parser tries to expand the expression !","info": but as it can't find a variable named ","info" it simply remove that part.

The solution to handle exclamation marks within for-loops is the toggling technic.

Code: Select all

SETLOCAL DisableDelayedExpansion
FOR /F "delims=" %%A IN ('^"%xidel% -q VPWON_1226681.jsonp -e "json(extract(.,'\((.+)\)',1))"^"') DO (
  ECHO JSON: %%A
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
  echo(
  FOR /F "delims=" %%B IN ('ECHO %%A ^| %xidel% -q - -e "$json/(t:=tijdsduur,ss:=start,to:=eind)" --output-format=cmd') DO %%B

  SETLOCAL EnableDelayedExpansion
  ECHO Tijdsduur: !t!
  IF DEFINED ss (
    SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
    SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
    ECHO Start:     !ss! (!_ss!s^)
    ECHO Einde:     !to! (!_to!s^)
  )
  ENDLOCAL
)


This ensures that only the relevant part operates with the delayed expansion mode and it doesn't destroy any variable contents.

Reino
Posts: 20
Joined: 14 May 2015 06:13
Contact:

Re: Exclamationmark in variable causes parsing-errors

#3 Post by Reino » 21 Jun 2015 14:23

Hello jeb,

jeb wrote:But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
Ah! I didn't know that. We learn everyday. :wink:

jeb wrote:The solution to handle exclamation marks within for-loops is the toggling technic.
The delayed expansion has always been a thing I find hard to understand, but I get it now. Only apply it where it's really necessary. I was actually really close in finding the same solution just 30minutes ago.
Thanks a lot for your quick support, jeb!

jeb
Expert
Posts: 967
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Exclamationmark in variable causes parsing-errors

#4 Post by jeb » 21 Jun 2015 14:49

CoRoNe wrote:jeb wrote:
But in your case you activate the delayed expansion a second time by using the ECHO %%A in the line
FOR /F "delims=" %%B IN ('ECHO %%A ^| %jq% .') DO ECHO %%B
Ah! I didn't know that. We learn everyday. :wink:

And I'm not quite correct here :( ,
the delayed expansion is only once parsed,
but the content of %%A will be parsed again,
normally without delayed expansion, as delayed expansion is not activated as default when a new cmd.exe instance will be created.

Post Reply