JREPL.BAT v7.15 - regex text processor now with Unicode and XRegExp support

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
aGerman
Expert
Posts: 3563
Joined: 22 Jan 2010 18:01
Location: Germany

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#376 Post by aGerman » 20 Jul 2018 02:39

I see. To be honest I really expected to have ADO by default at least on XP and onwards because it was already introduced in the late 1990's and I never worked on a machine where it wasn't installed. I'll keep my eyes open. If I find something about its availability then I'll get back to you.

Steffen

dbenham
Expert
Posts: 2203
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#377 Post by dbenham » 20 Jul 2018 06:04

There may be confusion over ADO vs ADO.NET.

ADO.NET certainly is not guaranteed to be present, and I may have read something that confused the two, or I may have misinterpreted what I read.

I can't find anything that indicates ADO is not standard, even for XP Home. I see articles about it not working in some cases, but they seem to be about compatibility issues that are solved by standard updates.

My StackOverflow tester was on XP SP3 without .NET, and 7.13 worked just fine. But he did not specify Home or Pro.

I'm almost ready to pull the trigger and drop the CERTUTIL code, but not quite there.


Dave Benham

aGerman
Expert
Posts: 3563
Joined: 22 Jan 2010 18:01
Location: Germany

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#378 Post by aGerman » 20 Jul 2018 13:52

dbenham wrote:
20 Jul 2018 06:04
ADO vs ADO.NET
I think you nailed it.
Finally you need to worry about upward compatibility rather than downward compatibility. Microsoft already tries forcing us to use .NET. Powershell is set as default console on Win 10. They want us to use C#.NET rather than C/Win32 API and thus, it's a logical consequence to have ADO.NET as successor of ADO ...

Steffen

fdart17
Posts: 2
Joined: 18 Sep 2018 11:18

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#379 Post by fdart17 » 18 Sep 2018 11:32

Hi dbenham,

Thank you very much for writing and maintaining this script.

Within the context of a cmd.exe in Windows 10, I'm calling a batch script which then invokes an exe to do a C++ build. I want to modify the output to the cmd prompt to color warnings in yellow and errors in red. This seems like a good application of this script. Is it possible to invoke jrepl in such a way that it will attach itself to the existing cmd prompt, running continuously to intercept incoming output and modify it before it is displayed to add ansi escape characters, while still allowing the user to interact with the existing cmd prompt to invoke more commands?

Thanks,
Justin

Squashman
Expert
Posts: 4097
Joined: 23 Dec 2011 13:59

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#380 Post by Squashman » 18 Sep 2018 14:21

fdart17 wrote:
18 Sep 2018 11:32
Hi dbenham,

Thank you very much for writing and maintaining this script.

Within the context of a cmd.exe in Windows 10, I'm calling a batch script which then invokes an exe to do a C++ build. I want to modify the output to the cmd prompt to color warnings in yellow and errors in red. This seems like a good application of this script. Is it possible to invoke jrepl in such a way that it will attach itself to the existing cmd prompt, running continuously to intercept incoming output and modify it before it is displayed to add ansi escape characters, while still allowing the user to interact with the existing cmd prompt to invoke more commands?

Thanks,
Justin
Going to assume this is you: https://stackoverflow.com/questions/523 ... -add-color

fdart17
Posts: 2
Joined: 18 Sep 2018 11:18

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#381 Post by fdart17 » 18 Sep 2018 15:12

Yes, that is me.

I wanted to post an update here to not waste anyone's time. I've found another solution to the problem which happens to not leverage JREPL.BAT.
https://stackoverflow.com/questions/523 ... -add-color

It's still not as good as I would like, so I will keep searching for a better solution :)

Michael.Uray
Posts: 5
Joined: 12 Oct 2018 19:02

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#382 Post by Michael.Uray » 12 Oct 2018 19:10

I got the message

Code: Select all

JScript runtime error: Nicht genügend Arbeitsspeicher.
(means not enough RAM / memory)

when I did run a regex search/replace:

Code: Select all

JREPL.BAT "^\d+ 0 obj \n(?:(?!endobj $).*\n)*?\/Name \/Paragraph(?:.*\n)*?endobj ?$" "" /F input.pdf /M /O output.pdf
It happens when the cscript.exe process memor runs up to 3GB.

There is actually enough free RAM.
Any ideas?

Squashman
Expert
Posts: 4097
Joined: 23 Dec 2011 13:59

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#383 Post by Squashman » 12 Oct 2018 20:03

You are using the /M option. You are limited to 2GB of ram for 32 bit applications.

Michael.Uray
Posts: 5
Joined: 12 Oct 2018 19:02

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#384 Post by Michael.Uray » 13 Oct 2018 03:07

Squashman wrote:
12 Oct 2018 20:03
You are using the /M option. You are limited to 2GB of ram for 32 bit applications.
I need to parse multi line files like this https://regex101.com/r/fA6wE2/155 to search for objects which contain "Name /Paragraph".
The file has about 1.5 GB.

Any idea how to get around this issue?

dbenham
Expert
Posts: 2203
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#385 Post by dbenham » 14 Oct 2018 17:36

Unfortunately there is no way to circumvent the file size limit when using the /M option. I'm not sure the exact limit, but it is actually less than 1 GB.

You may still be able to use JREPL if you ditch the /M option and combine the /T option with /JMATCH and custom JScript. You don't state what your end goal is. The code below simply preserves object entries that contain a "/Name /Paragraph" line, and throws everything else away.

Code: Select all

@echo off
setlocal

set "init=var obj=false, keep=false, eol='\n'"

set "find1=^\d+ 0 obj\s*$"
set "repl1=obj=$0+eol;$txt=false;"

set "find2=^/Name /Paragraph\s*$"
set "repl2=if (obj){obj+=$0+eol;keep=true;}$txt=false;"

set "find3=^endobj\s*$"
set "repl3=$txt=keep?obj+$0:false;obj=keep=false;"

set "find4=^.*$"
set "repl4=if (obj){obj+=$0+eol;}$txt=false;"

set "find=%find1%|%find2%|%find3%|%find4%"
set "repl=%repl1%|%repl2%|%repl3%|%repl4%"

call jrepl find repl /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf

exit /b
Dave Benham

dbenham
Expert
Posts: 2203
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#386 Post by dbenham » 15 Oct 2018 05:58

Upon looking closer at your original code, it looks like you are trying to remove all the objects that contain "/Name /Paragraph".

Code: Select all

@echo off
setlocal

set "init=var obj='', keep=true, eol='\n'"

set "find1=^\d+ 0 obj\s*$"
set "repl1=obj=$0+eol;$txt=false;"

set "find2=^/Name /Paragraph\s*$"
set "repl2=if (obj)keep=$txt=false; else $txt=$0;"

set "find3=^endobj\s*$"
set "repl3=$txt=keep?obj+$0:false;obj='';keep=true;"

set "find4=^.*$"
set "repl4=if (obj){$txt=false;if (keep)obj+=$0+eol;}else $txt=$0;"

set "find=%find1%|%find2%|%find3%|%find4%"
set "repl=%repl1%|%repl2%|%repl3%|%repl4%"

call jrepl find repl /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf

exit /b
Note - It would probably be just as easy to write a custom JScript script as it is to use JREPL as I have done. And custom JScript would probably be a bit faster.


Dave Benham

Michael.Uray
Posts: 5
Joined: 12 Oct 2018 19:02

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#387 Post by Michael.Uray » 15 Oct 2018 07:05

dbenham wrote:
15 Oct 2018 05:58
Upon looking closer at your original code, it looks like you are trying to remove all the objects that contain "/Name /Paragraph".
This is correct.

I had to replace the variable assignment to the command to run it, but it does not create the expected output.

Code: Select all

call jrepl "%find%" "%repl%" /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf
The output file (71MB) gets much larger than the input file (7MB) for some reason.

dbenham
Expert
Posts: 2203
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: JREPL.BAT v7.13 - regex text processor now with Unicode and XRegExp support

#388 Post by dbenham » 15 Oct 2018 08:21

Michael.Uray wrote:
15 Oct 2018 07:05
I had to replace the variable assignment to the command to run it, but it does not create the expected output.

Code: Select all

call jrepl "%find%" "%repl%" /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf
The output file (71MB) gets much larger than the input file (7MB) for some reason.
No, the find and repl must not be enclosed by percents in the JREPL call.

I tested the code against the input that you provided in the link, and it worked perfectly for me. But then I added some additional lines at the top of the input that were not contained in an "object", and it began behaving badly. I did some testing, and it appears there is a newly discovered bug - there are internal str and obj objects within the code that interfere with the user supplied JScript if /JBEG defines a variable named str or obj.

I was able to get the code to work properly if I changed obj to Obj:

Code: Select all

@echo off
setlocal

set "init=var Obj='', keep=true, eol='\n';"

set "find1=^\d+ 0 obj\s*$"
set "repl1=Obj=$0+eol;$txt=false;"

set "find2=^/Name /Paragraph\s*$"
set "repl2=if (Obj)keep=$txt=false; else $txt=$0;"

set "find3=^endobj\s*$"
set "repl3=$txt=keep?Obj+$0:false;Obj='';keep=true;"

set "find4=^.*$"
set "repl4=if (Obj!==''){$txt=false;if (keep)Obj+=$0+eol;}else $txt=$0;"

set "find=%find1%|%find2%|%find3%|%find4%"
set "repl=%repl1%|%repl2%|%repl3%|%repl4%"

call jrepl find repl /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf

exit /b

Update - I fixed the bug and released version 7.14. The original code using obj should now work.

Dave Benham

dbenham
Expert
Posts: 2203
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: JREPL.BAT v7.14 - regex text processor now with Unicode and XRegExp support

#389 Post by dbenham » 15 Oct 2018 13:21

Here is version 7.14
JREPL7.14.zip
Downloaded 75 times over 5 days from the main release page while v7.14 was the current version.
(27.24 KiB) Downloaded 41 times

This is simply a bug fix to allow user defined variables named str and/or obj, and to hide the internal xbytes variable from user code.

Code: Select all

    2018-10-15 v7.14: Bug fix - User defined variables declared in /JBEG named
                      str and/or obj were getting clobbered.
                      Bug fix - Internal variable xbytes was visibile to user
                      supplied JScript.

Dave Benham

Michael.Uray
Posts: 5
Joined: 12 Oct 2018 19:02

Re: JREPL.BAT v7.14 - regex text processor now with Unicode and XRegExp support

#390 Post by Michael.Uray » 15 Oct 2018 14:33

dbenham wrote:
15 Oct 2018 08:21
Michael.Uray wrote:
15 Oct 2018 07:05
I had to replace the variable assignment to the command to run it, but it does not create the expected output.

Code: Select all

call jrepl "%find%" "%repl%" /t "|" /jmatchq /jbeg init /v /u /f input.pdf /o output.pdf
No, the find and repl must not be enclosed by percents in the JREPL call.
Hmm, I had to add that again in the .cmd file.
"find" and "repl" are envirnment variables so I have to use them with "%" on the beginn and on the end, right?
I have to also put it under quotes because the string (^\d+ 0 obj\s*$|^/Name /Paragraph\s*$|^endobj\s*$|^.*$) contains empty spaces, right?

The output is still a 70 MB file, maybe I should send you the input file via PM that you are able to recreate the problem.

Post Reply