How does the Windows Command Interpreter (CMD.EXE) parse scripts?

windows parsing batch-file cmd variable-expansion
Benoit picture Benoit · Nov 4, 2010 · Viewed 68.8k times · Source

I ran into ss64.com which provides good help regarding how to write batch scripts that the Windows Command Interpreter will run.

However, I have been unable to find a good explanation of the grammar of batch scripts, how things expand or do not expand, and how to escape things.

Here are sample questions that I have not been able to solve:

  • How is the quote system managed? I made a TinyPerl script
    ( foreach $i (@ARGV) { print '*' . $i ; } ), compiled it and called it this way :
    • my_script.exe "a ""b"" c" → output is *a "b*c
    • my_script.exe """a b c""" → output it *"a*b*c"
  • How does the internal echo command work? What is expanded inside that command?
  • Why do I have to use for [...] %%I in file scripts, but for [...] %I in interactive sessions?
  • What are the escape characters, and in what context? How to escape a percent sign? For example, how can I echo %PROCESSOR_ARCHITECTURE% literally? I found that echo.exe %""PROCESSOR_ARCHITECTURE% works, is there a better solution?
  • How do pairs of % match? Example:
    • set b=a , echo %a %b% c% → %a a c%
    • set a =b, echo %a %b% c% → bb c%
  • How do I ensure a variable passes to a command as a single argument if ever this variable contains double quotes?
  • How are variables stored when using the set command? For example, if I do set a=a" b and then echo.%a% I obtain a" b. If I however use echo.exe from the UnxUtils, I get a b. How comes %a% expands in a different way?

Thank you for your lights.

Answer

jeb picture jeb · Nov 4, 2010

We performed experiments to investigate the grammar of batch scripts. We also investigated differences between batch and command line mode.

Batch Line Parser:

Here is a brief overview of phases in the batch file line parser:

Phase 0) Read Line:

Phase 1) Percent Expansion:

Phase 2) Process special characters, tokenize, and build a cached command block: This is a complex process that is affected by things such as quotes, special characters, token delimiters, and caret escapes.

Phase 3) Echo the parsed command(s) Only if the command block did not begin with @, and ECHO was ON at the start of the preceding step.

Phase 4) FOR %X variable expansion: Only if a FOR command is active and the commands after DO are being processed.

Phase 5) Delayed Expansion: Only if delayed expansion is enabled

Phase 5.3) Pipe processing: Only if commands are on either side of a pipe

Phase 5.5) Execute Redirection:

Phase 6) CALL processing/Caret doubling: Only if the command token is CALL

Phase 7) Execute: The command is executed

Here are details for each phase:

Note that the phases described below are only a model of how the batch parser works. The actual cmd.exe internals may not reflect these phases. But this model is effective at predicting behavior of batch scripts.

Phase 0) Read Line: Read line of input through first <LF>.

  • When reading a line to be parsed as a command, <Ctrl-Z> (0x1A) is read as <LF> (LineFeed 0x0A)
  • When GOTO or CALL reads lines while scanning for a :label, <Ctrl-Z>, is treated as itself - it is not converted to <LF>

Phase 1) Percent Expansion:

  • A double %% is replaced by a single %
  • Expansion of arguments (%*, %1, %2, etc.)
  • Expansion of %var%, if var does not exist replace it with nothing
  • Line is truncated at first <LF> not within %var% expansion
  • For a complete explanation read the first half of this from dbenham Same thread: Percent Phase

Phase 2) Process special characters, tokenize, and build a cached command block: This is a complex process that is affected by things such as quotes, special characters, token delimiters, and caret escapes. What follows is an approximation of this process.

There are concepts that are important throughout this phase.

  • A token is simply a string of characters that is treated as a unit.
  • Tokens are separated by token delimiters. The standard token delimiters are <space> <tab> ; , = <0x0B> <0x0C> and <0xFF>
    Consecutive token delimiters are treated as one - there are no empty tokens between token delimiters
  • There are no token delimiters within a quoted string. The entire quoted string is always treated as part of a single token. A single token may consist of a combination of quoted strings and unquoted characters.

The following characters may have special meaning in this phase, depending on context: <CR> ^ ( @ & | < > <LF> <space> <tab> ; , = <0x0B> <0x0C> <0xFF>

Look at each character from left to right:

  • If <CR> then remove it, as if it were never there (except for weird redirection behavior)
  • If a caret (^), the next character is escaped, and the escaping caret is removed. Escaped characters lose all special meaning (except for <LF>).
  • If a quote ("), toggle the quote flag. If the quote flag is active, then only " and <LF> are special. All other characters lose their special meaning until the next quote toggles the quote flag off. It is not possible to escape the closing quote. All quoted characters are always within the same token.
  • <LF> always turns off the quote flag. Other behaviors vary depending on context, but quotes never alter the behavior of <LF>.
    • Escaped <LF>
      • <LF> is stripped
      • The next character is escaped. If at the end of line buffer, then the next line is read and processed by phases 1 and 1.5 and appended to the current one before escaping the next character. If the next character is <LF>, then it is treated as a literal, meaning this process is not recursive.
    • Unescaped <LF> not within parentheses
      • <LF> is stripped and parsing of the current line is terminated.
      • Any remaining characters in the line buffer are simply ignored.
    • Unescaped <LF> within a FOR IN parenthesized block
      • <LF> is converted into a <space>
      • If at the end of the line buffer, then the next line is read and appended to the current one.
    • Unescaped <LF> within a parenthesized command block
      • <LF> is converted into <LF><space>, and the <space> is treated as part of the next line of the command block.
      • If at the end of line buffer, then the next line is read and appended to the space.
  • If one of the special characters & | < or >, split the line at this point in order to handle pipes, command concatenation, and redirection.
    • In the case of a pipe (|), each side is a separate command (or command block) that gets special handling in phase 5.3
    • In the case of &, &&, or || command concatenation, each side of the concatenation is treated as a separate command.
    • In the case of <, <<, >, or >> redirection, the redirection clause is parsed, temporarily removed, and then appended to the end of the current command. A redirection clause consists of an optional file handle digit, the redirection operator, and the redirection destination token.
      • If the token that precedes the redirection operator is a single unescaped digit, then the digit specifies the file handle to be redirected. If the handle token is not found, then output redirection defaults to 1 (stdout), and input redirection defaults to 0 (stdin).
  • If the very first token for this command (prior to moving redirection to the end) begins with @, then the @ has special meaning. (@ is not special in any other context)
    • The special @ is removed.
    • If ECHO is ON, then this command, along with any following concatenated commands on this line, are excluded from the phase 3 echo. If the @ is before an opening (, then the entire parenthesized block is excluded from the phase 3 echo.
  • Process parenthesis (provides for compound statements across multiple lines):
    • If the parser is not looking for a command token, then ( is not special.
    • If the parser is looking for a command token and finds (, then start a new compound statement and increment the parenthesis counter
    • If the parenthesis counter is > 0 then ) terminates the compound statement and decrements the parenthesis counter.
    • If the line end is reached and the parenthesis counter is > 0 then the next line will be appended to the compound statement (starts again with phase 0)
    • If the parenthesis counter is 0 and the parser is looking for a command, then ) functions similar to a REM statement as long as it is immediately followed by a token delimiter, special character, newline, or end-of-file
      • All special characters lose their meaning except ^ (line concatenation is possible)
      • Once the end of the logical line is reached, the entire "command" is discarded.
  • Each command is parsed into a series of tokens. The first token is always treated as a command token (after special @ have been stripped and redirection moved to the end).
    • Leading token delimiters prior to the command token are stripped
    • When parsing the command token, ( functions as a command token delimiter, in addition to the standard token delimiters
    • The handling of subsequent tokens depends on the command.
  • Most commands simply concatenate all arguments after the command token into a single argument token. All argument token delimiters are preserved. Argument options are typically not parsed until phase 7.
  • Three commands get special handling - IF, FOR, and REM
    • IF is split into two or three distinct parts that are processed independently. A syntax error in the IF construction will result in a fatal syntax error.
      • The comparison operation is the actual command that flows all the way through to phase 7
        • All IF options are fully parsed in phase 2.
        • Consecutive token delimiters collapse into a single space.
        • Depending on the comparison operator, there will be one or two value tokens that are identified.
      • The True command block is the set of commands after the condition, and is parsed like any other command block. If ELSE is to be used, then the True block must be parenthesized.
      • The optional False command block is the set of commands after ELSE. Again, this command block is parsed normally.
      • The True and False command blocks do not automatically flow into the subsequent phases. Their subsequent processing is controled by phase 7.
    • FOR is split in two after the DO. A syntax error in the FOR construction will result in a fatal syntax error.
      • The portion through DO is the actual FOR iteration command that flows all the way through phase 7
        • All FOR options are fully parsed in phase 2.
        • The IN parenthesized clause treats <LF> as <space>. After the IN clause is parsed, all tokens are concatenated together to form a single token.
        • Consecutive unescaped/unquoted token delimiters collapse into a single space throughout the FOR command through DO.
      • The portion after DO is a command block that is parsed normally. Subsequent processing of the DO command block is controled by the iteration in phase 7.
    • REM detected in phase 2 is treated dramatically different than all other commands.
      • Only one argument token is parsed - the parser ignores characters after the first argument token.
      • The REM command may appear in phase 3 output, but the command is never executed, and the original argument text is echoed - escaping carets are not removed, except...
        • If there is only one argument token that ends with an unescaped ^ that ends the line, then the argument token is thrown away, and the subsequent line is parsed and appended to the REM. This repeats until there is more than one token, or the last character is not ^.
  • If the command token begins with :, and this is the first round of phase 2 (not a restart due to CALL in phase 6) then
    • The token is normally treated as an Unexecuted Label.
      • The remainder of the line is parsed, however ), <, >, & and | no longer have special meaning. The entire remainder of the line is considered to be part of the label "command".
      • The ^ continues to be special, meaning that line continuation can be used to append the subsequent line to the label.
      • An Unexecuted Label within a parenthesized block will result in a fatal syntax error unless it is immediately followed by a command or Executed Label on the next line.
        • ( no longer has special meaning for the first command that follows the Unexecuted Label.
      • The command is aborted after label parsing is complete. Subsequent phases do not take place for the label
    • There are three exceptions that can cause a label found in phase 2 to be treated as an Executed Label that continues parsing through phase 7.
      • There is redirection that precedes the label token, and there is a | pipe or &, &&, or || command concatenation on the line.
      • There is redirection that precedes the label token, and the command is within a parenthesized block.
      • The label token is the very first command on a line within a parenthesized block, and the line above ended with an Unexecuted Label.
    • The following occurs when an Executed Label is discovered in phase 2
      • The label, its arguments, and its redirection are all excluded from any echo output in phase 3
      • Any subsequent concatenated commands on the line are fully parsed and executed.
    • For more information about Executed Labels vs. Unexecuted Labels, see https://www.dostips.com/forum/viewtopic.php?f=3&t=3803&p=55405#p55405

Phase 3) Echo the parsed command(s) Only if the command block did not begin with @, and ECHO was ON at the start of the preceding step.

Phase 4) FOR %X variable expansion: Only if a FOR command is active and the commands after DO are being processed.

  • At this point, phase 1 of batch processing will have already converted a FOR variable like %%X into %X. The command line has different percent expansion rules for phase 1. This is the reason that command lines use %X but batch files use %%X for FOR variables.
  • FOR variable names are case sensitive, but ~modifiers are not case sensitive.
  • ~modifiers take precedence over variable names. If a character following ~ is both a modifier and a valid FOR variable name, and there exists a subsequent character that is an active FOR variable name, then the character is interpreted as a modifier.
  • FOR variable names are global, but only within the context of a DO clause. If a routine is CALLed from within a FOR DO clause, then the FOR variables are not expanded within the CALLed routine. But if the routine has its own FOR command, then all currently defined FOR variables are accessible to the inner DO commands.
  • FOR variable names can be reused within nested FORs. The inner FOR value takes precedence, but once the INNER FOR closes, then the outer FOR value is restored.
  • If ECHO was ON at the start of this phase, then phase 3) is repeated to show the parsed DO commands after the FOR variables have been expanded.

---- From this point onward, each command identified in phase 2 is processed separately.
---- Phases 5 through 7 are completed for one command before moving on to the next.

Phase 5) Delayed Expansion: Only if delayed expansion is on, the command is not in a parenthesized block on either side of a pipe, and the command is not a "naked" batch script (script name without parentheses, CALL, command concatenation, or pipe).

  • Each token for a command is parsed for delayed expansion independently.
    • Most commands parse two or more tokens - the command token, the arguments token, and each redirection destination token.
    • The FOR command parses the IN clause token only.
    • The IF command parses the comparison values only - either one or two, depending on the comparison operator.
  • For each parsed token, first check if it contains any !. If not, then the token is not parsed - important for ^ characters. If the token does contain !, then scan each character from left to right:
    • If it is a caret (^) the next character has no special meaning, the caret itself is removed
    • If it is an exclamation mark, search for the next exclamation mark (carets are not observed anymore), expand to the value of the variable.
      • Consecutive opening ! are collapsed into a single !
      • Any remaining unpaired ! is removed
    • Expanding vars at this stage is "safe", because special characters are not detected anymore (even <CR> or <LF>)
    • For a more complete explanation, read the 2nd half of this from dbenham same thread - Exclamation Point Phase

Phase 5.3) Pipe processing: Only if commands are on either side of a pipe
Each side of the pipe is processed independently and asynchronously.

  • If command is internal to cmd.exe, or it is a batch file, or if it is a parenthesized command block, then it is executed in a new cmd.exe thread via %comspec% /S /D /c" commandBlock", so the command block gets a phase restart, but this time in command line mode.
    • If a parenthesized command block, then all <LF> with a command before and after are converted to <space>&. Other <LF> are stripped.
  • This is the end of processing for the pipe commands.
  • See Why does delayed expansion fail when inside a piped block of code? for more about pipe parsing and processing

Phase 5.5) Execute Redirection: Any redirection that was discovered in phase 2 is now executed.

Phase 6) CALL processing/Caret doubling: Only if the command token is CALL, or if the text before the first occurring standard token delimiter is CALL. If CALL is parsed from a larger command token, then the unused portion is prepended to the arguments token before proceeding.

  • Scan the arguments token for an unquoted /?. If found anywhere within the tokens, then abort phase 6 and proceed to Phase 7, where the HELP for CALL will be printed.
  • Remove the first CALL, so multiple CALL's can be stacked
  • Double all carets
  • Restart phases 1, 1.5, and 2, but do not continue to phase 3
    • Any doubled carets are reduced back to one caret as long as they are not quoted. But unfortunately, quoted carets remain doubled.
    • Phase 1 changes a bit - Expansion errors in step 1.2 or 1.3 abort the CALL, but the error is not fatal - batch processing continues.
    • Phase 2 tasks are altered a bit
      • Any newly appearing unquoted, unescaped redirection that was not detected in the first round of phase 2 is detected, but it is removed (including the file name) without actually performing the redirection
      • Any newly appearing unquoted, unescaped caret at the end of the line is removed without performing line continuation
      • The CALL is aborted without error if any of the following are detected
        • Newly appearing unquoted, unescaped & or |
        • The resultant command token begins with unquoted, unescaped (
        • The very first token after the removed CALL began with @
      • If the resultant command is a seemingly valid IF or FOR, then execution will subsequently fail with an error stating that IF or FOR is not recognized as an internal or external command.
      • Of course the CALL is not aborted in this 2nd round of phase 2 if the resultant command token is a label beginning with :.
  • If the resultant command token is CALL, then restart Phase 6 (repeats until no more CALL)
  • If the resultant command token is a batch script or a :label, then execution of the CALL is fully handled by the remainder of Phase 6.
    • Push the current batch script file position on the call stack so that execution can resume from the correct position when the CALL is completed.
    • Setup the %0, %1, %2, ...%N and %* argument tokens for the CALL, using all resultant tokens
    • If the command token is a label that begins with :, then
      • Restart Phase 5. This can impact what :label is CALLed. But since the %0 etc. tokens have already been setup, it will not alter the arguments that are passed to the CALLed routine.
      • Execute GOTO label to position the file pointer at the beginning of the subroutine (ignore any other tokens that may follow the :label) See Phase 7 for rules on how GOTO works.
        • If the :label token is missing, or the :label is not found, then the call stack is immediately popped to restore the saved file position, and the CALL is aborted.
        • If the :label happens to contain /?, then GOTO help is printed instead of searching for the :label. The file pointer does not move, such that code after the CALL is executed twice, once in the CALL context, and then again after the CALL return. See Why CALL prints the GOTO help message in this script?And why command after that are executed twice? for more info.
    • Else transfer control to the specified batch script.
    • Execution of the CALLed :label or script continues until either EXIT /B or end-of-file is reached, at which point the CALL stack is popped and execution resumes from the saved file position.
      Phase 7 is not executed for CALLed scripts or :labels.
  • Else the result of phase 6 falls through into phase 7 for execution.

Phase 7) Execute: The command is executed

  • 7.1 - Execute internal command - If the command token is quoted, then skip this step. Otherwise, attempt to parse out an internal command and execute.
    • The following tests are made to determine if an unquoted command token represents an internal command:
      • If the command token exactly matches an internal command, then execute it.
      • Else break the command token before the first occurrence of + / [ ] <space> <tab> , ; or =
        If the preceding text is an internal command, then remember that command
        • If in command line mode, or if the command is from a parenthesized block, IF true or false command block, FOR DO command block, or involved with command concatenation, then execute the internal command
        • Else (must be a stand-alone command in batch mode) scan the current folder and the PATH for a .COM, .EXE, .BAT, or .CMD file whose base name matches the original command token
          • If the first matching file is a .BAT or .CMD, then goto 7.3.exec and execute that script
          • Else (match not found or first match is .EXE or .COM) execute the remembered internal command
      • Else break the command token before the first occurrence of . \ or :
        If the preceding text is not an internal command, then goto 7.2
        Else the preceding text may be an internal command. Remember this command.
      • Break the command token before the first occurrence of + / [ ] <space> <tab> , ; or =
        If the preceding text is a path to an existing file, then goto 7.2
        Else execute the remembered internal command.
    • If an internal command is parsed from a larger command token, then the unused portion of the command token is included in the argument list
    • Just because a command token is parsed as an internal command does not mean that it will execute successfully. Each internal command has its own rules as to how the arguments and options are parsed, and what syntax is allowed.
    • All internal commands will print help instead of performing their function if /? is detected. Most recognize /? if it appears anywhere in the arguments. But a few commands like ECHO and SET only print help if the first argument token begins with /?.
    • SET has some interesting semantics:
      • If a SET command has a quote before the variable name and extensions are enabled
        set "name=content" ignored --> value=content
        then the text between the first equal sign and the last quote is used as the content (first equal and last quote excluded). Text after the last quote is ignored. If there is no quote after the equal sign, then the rest of the line is used as content.
      • If a SET command does not have a quote before the name
        set name="content" not ignored --> value="content" not ignored
        then the entire remainder of the line after the equal is used as content, including any and all quotes that may be present.
    • An IF comparison is evaluated, and depending on whether the condition is true or false, the appropriate already parsed dependent command block is processed, starting with phase 5.
    • The IN clause of a FOR command is iterated appropriately.
      • If this is a FOR /F that iterates the output of a command block, then:
        • The IN clause is executed in a new cmd.exe process via CMD /C.
        • The command block must go through the entire parsing process a second time, but this time in a command line context
        • ECHO will start out ON, and delayed expansion will usually start out disabled (dependent on the registry setting)
        • All environment changes made by the IN clause command block will be lost once the child cmd.exe process terminates
      • For each iteration:
        • The FOR variable values are defined
        • The already parsed DO command block is then processed, starting with phase 4.
    • GOTO uses the following logic to locate the :label
      • Parse the label from the first argument token
      • Scan for the next occurrence of the label
        • Start from the current file position
        • If end of file is reached, then loop back to the beginning of file and continue to the original starting point.
      • The scan stops at the first occurrence of the label that it finds, and the file pointer is set to the line immediately following the label. Execution of the script resumes from that point. Note that a successful true GOTO will immediately abort any parsed block of code, including FOR loops.
      • If the label is not found, or the label token is missing, then the GOTO fails, an error message is printed, and the call stack is popped. This effectively functions as an EXIT /B, except any already parsed commands in the current command block that follow the GOTO are still executed, but in the context of the CALLer (the context that exists after EXIT /B)
      • See https://www.dostips.com/forum/viewtopic.php?t=3803 for a more precise description of label parsing rules, and https://www.dostips.com/forum/viewtopic.php?t=8988 for label scanning rules.
    • RENAME and COPY both accept wildcards for the source and target paths. But Microsoft does a terrible job documenting how the wildcards work, especially for the target path. A useful set of wildcard rules may be found at How does the Windows RENAME command interpret wildcards?
  • 7.2 - Execute volume change - Else if the command token does not begin with a quote, is exactly two characters long, and the 2nd character is a colon, then change the volume
    • All argument tokens are ignored
    • If the volume specified by the first character cannot be found, then abort with an error
    • A command token of :: will always result in an error unless SUBST is used to define a volume for ::
      If SUBST is used to define a volume for ::, then the volume will be changed, it will not be treated as a label.
  • 7.3 - Execute external command - Else try to treat the command as an external command.
    • If in command line mode and the command is not quoted and does not begin with a volume specification, white-space, ,, ;, = or + then break the command token at the first occurrence of <space> , ; or = and prepend the remainder to the argument token(s).
    • If the 2nd character of the command token is a colon, then verify the volume specified by the 1st character can be found.
      If the volume cannot be found, then abort with an error.
    • If in batch mode and the command token begins with :, then goto 7.4
      Note that if the label token begins with ::, then this will not be reached because the preceding step will have aborted with an error unless SUBST is used to define a volume for ::.
    • Identify the external command to execute.
      • This is a complex process that may involve the current volume, current directory, PATH variable, PATHEXT variable, and or file associations.
      • If a valid external command cannot be identified, then abort with an error.
    • If in command line mode and the command token begins with :, then goto 7.4
      Note that this is rarely reached because the preceding step will have aborted with an error unless the command token begins with ::, and SUBST is used to define a volume for ::, and the entire command token is a valid path to an external command.
    • 7.3.exec - Execute the external command.
  • 7.4 - Ignore a label - Ignore the command and all its arguments if the command token begins with :.
    Rules in 7.2 and 7.3 may prevent a label from reaching this point.

Command Line Parser:

Works like the BatchLine-Parser, except:

Phase 1) Percent Expansion:

  • No %*, %1 etc. argument expansion
  • If var is undefined, then %var% is left unchanged.
  • No special handling of %%. If var=content, then %%var%% expands to %content%.

Phase 3) Echo the parsed command(s)

  • This is not performed after phase 2. It is only performed after phase 4 for the FOR DO command block.

Phase 5) Delayed Expansion: only if DelayedExpansion is enabled

  • If var is undefined, then !var! is left unchanged.

Phase 7) Execute Command

  • Attempts to CALL or GOTO a :label result in an error.
  • As already documented in phase 7, an executed label may result in an error under different scenarios.
    • Batch executed labels can only cause an error if they begin with ::
    • Command line executed labels almost always result in an error

Parsing of integer values

There are many different contexts where cmd.exe parses integer values from strings, and the rules are inconsistent:

  • SET /A
  • IF
  • %var:~n,m% (variable substring expansion)
  • FOR /F "TOKENS=n"
  • FOR /F "SKIP=n"
  • FOR /L %%A in (n1 n2 n3)
  • EXIT [/B] n

Details for these rules may be found at Rules for how CMD.EXE parses numbers

For anyone wishing to improve the cmd.exe parsing rules, there is a discussion topic on the DosTips forum where issues can be reported and suggestions made.

Hope it helps
Jan Erik (jeb) - Original author and discoverer of phases
Dave Benham (dbenham) - Much additional content and editing

Related questions

print the folder tree with sizes, windows

I'm looking for a way to print (or write to file) the folder tree of my windows PC, including the size of each folder - but not of single files. The output should look like: - My Documents/pictures/selfies/ …

Read More
How can you find out which process is listening on a port on Windows?

How can you find out which process is listening on a port on Windows?

Read More
How can I install pip on Windows?

pip is a replacement for easy_install. But should I install pip using easy_install on Windows? Is there a better way?

Read More