Pages

Wednesday, January 11, 2023

A warning: Batch newlines and EnableDelayedExpansion

Remember when I said this?

but if I want something quick that can run anywhere [someone's on Windows], I use a batch file, and over the years, kinda like VIm, I've slowly become if not proficient then at least competent.

What a naïve time that was.


Take this seemingly innocuous question on SO:

How can you you insert a newline from your batch file output?

And then check this answer:

Here you go, create a .bat file with the following in it :

@echo off
REM Creating a Newline variable (the two blank lines are required!)
set NLM=^


set NL=^^^%NLM%%NLM%^%NLM%%NLM%
REM Example Usage:
echo There should be a newline%NL%inserted here.

echo.
pause

Wha? set NL=^^^%NLM%%NLM%^%NLM%%NLM%??? What in the world? Huh?

Luckily (?) one of the commenters asked the same question.

Here's part of the answer:

The caret is an escape character for the next character, or at the line end it is used as multiline character, but this is nearly the same.

At the line end it simply escapes the next character, in this case the <Linefeed>, but there is a hidden feature, so if the escaped character is a <LF> it is ignored and the next character is read and escaped, but this charater will be always escaped, even if it is also a <LF>.

...

So you need to add an escaped linefeed into the line and that is the NL-Variable. The NL-Variable consists of only three characters.
NL=^<LF><LF> And if this is expanded, it creates only one escaped <LF> as the first <LF> after the caret will be ignored.

I mean, um, that requires Inception level cognition to understand. 😏 (Okay, actually it's pretty straightforward, but talk about a lack of discoverability.)

What's even more interesting is that dev's answer to the original question, which is this:

Like the answer of Ken, but with the use of the delayed expansion.

setlocal EnableDelayedExpansion
(set \n=^
%=Do not remove this line=%
)

echo Line1!\n!Line2
echo Works also with quotes "!\n!line2"

Aside

Okay, before I point out what I want to point out, note that %=_____=% is an inline comment format for batch.

From ss64.com, a common domain when I'm in batchland:

Batch variable names can contain spaces and punctuation, we can take advantage of this fact by typing our comment as a non-existent variable. Because it doesn’t exist, this variable will expand to nothing and so will have no effect when the script is run.

n.b. This only works in batch files, not directly at the command prompt.

To be sure that we don’t accidently choose a name which is in fact a real variable, start and end the comment with "="

@Echo off
Echo This is an example of an %= Inline Comment =% in the middle of a line.

(Variable names starting with "=" are reserved for undocumented dynamic variables. Those dynamic variables never end with "=", so by using an "=" at both the start and end of our comment, there is no possibility of a name clash.)


And we're back

Okay, now what I'd like to point out here is the setlocal EnableDelayedExpansion, something I think I had to use recently in a script I wrote. It's a bizarre new world.

Here's some explanation from ss64.com:

EnableDelayedExpansion

Delayed Expansion will cause variables within a batch file to be expanded at execution time rather than at parse time[;] this option is turned on with the SETLOCAL EnableDelayedExpansion command.

Variable expansion means replacing a variable (e.g. %windir%) with its value C:\WINDOWS

  • By default expansion will happen just once, before each line is executed.
    In this case, a "line" includes bracketed expressions even if they are formatted to run over several lines.

  • When !delayed! expansion is turned on, the variables will be expanded each time the line is executed, or for each loop in a FOR looping command.

And here's a decent example from that page showing how it works in practice.

@echo off
SETLOCAL EnableDelayedExpansion
Set "_var=first"
Set "_var=second" & Echo %_var% !_var!

The output is

first second

Note that the above example has to be in a batch file to work properly.

Anyhow, it's an insane rabbit hole that I didn't schedule for this sprint and fell into anyhow. I'm going to cut my losses now that I've warned others who may have been as naïve about batch scripts as I was, oh so long (less than three weeks 😉) ago.


Oh good heavens. I guess this makes sense. The first example can be taken by itself as a brain teaser, I think.

From ss64.com, again on EnableDelayedExpansion:

@echo off
Setlocal
Set _html=Hello^>World
Echo %_html%

In the above, the Echo command will create a text file called 'world' - not quite what we wanted!

Got that figured out? If not, sit and think about it for a second before reading more.

Got it? Good. Clever, huh?

This is because the variable is expanded at parse time, so the last line is executing Echo Hello > World and the > character is interpreted as a redirection operator.

If we now try the same thing with EnableDelayedExpansion:

Setlocal EnableDelayedExpansion
Set _html=Hello^>World
Echo !_html!

With delayed expansion, the variable (including the > ) is only expanded at execution time so the > character is never interpreted as a redirection operator.

This makes it possible to work with HTML and XML formatted strings in a variable.