Why does delayed expansion fail when inside a piped block of code?

Question

Here is a simple batch file that demonstrates how delayed expansion fails if it is within a block that is being piped. (The failure is toward the end of the script) Can anyone explain why this is?

I have a work-around, but it requires creation of a temporary file. I initially ran into this problem while working on Find files and sort by size in a Windows batch file

@echo off
setlocal enableDelayedExpansion

set test1=x
set test2=y
set test3=z

echo(

echo NORMAL EXPANSION TEST
echo Unsorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
)
echo(
echo Sorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
) | sort

echo(
echo ---------
echo(

echo DELAYED EXPANSION TEST
echo Unsorted works
(
  echo !test3!
  echo !test1!
  echo !test2!
)
echo(
echo Sorted fails
(
  echo !test3!
  echo !test1!
  echo !test2!
) | sort
echo(
echo Sort workaround
(
  echo !test3!
  echo !test1!
  echo !test2!
)>temp.txt
sort temp.txt
del temp.txt

Here are the results

NORMAL EXPANSION TEST
Unsorted works
z
x
y

Sorted works
x
y
z

---------

DELAYED EXPANSION TEST
Unsorted works
z
x
y

Sorted fails
!test1!
!test2!
!test3!

Sort workaround
x
y
z

Answer 1

As Aacini shows, it seems that many things fail within a pipe.

echo hello | set /p var=
echo here | call :function

But in reality it's only a problem to understand how the pipe works.

Each side of a pipe starts its own cmd.exe in its own ascynchronous thread.
That is the cause why so many things seem to be broken.

But with this knowledge you can avoid this and create new effects

echo one | ( set /p varX= & set varX )
set var1=var2
set var2=content of two
echo one | ( echo %%%var1%%% )
echo three | echo MYCMDLINE %%cmdcmdline%%
echo four  | (cmd /v:on /c  echo 4: !var2!)

Update 2019-08-15:
As discovered at Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?, cmd.exe is only used if the command is internal to cmd.exe, if the command is a batch file, or if the command is enclosed in a parenthesized block. External commands not enclosed within parentheses are launched in a new process without the aid of cmd.exe.

EDIT: In depth analysis

As dbenham shows, both sides of the pipes are equivalent for the expansion phases.
The main rules seems to be:

The normal batch parser phases are done
.. percent expansion
.. special character phase/block begin detection
.. delayed expansion (but only if delayed expansion is enabled AND it isn't a command block)

Start the cmd.exe with C:\Windows\system32\cmd.exe /S /D /c"<BATCH COMMAND>"
These expansions follows the rules of the cmd-line parser not the the batch-line parser.

.. percent expansion
.. delayed expansion (but only if delayed expansion is enabled)

The <BATCH COMMAND> will be modified if it's inside a parenthesis block.

(
echo one %%cmdcmdline%%
echo two
) | more

Called as C:\Windows\system32\cmd.exe /S /D /c" ( echo one %cmdcmdline% & echo two )", all newlines are changed to & operator.

Why the delayed expansion phase is affected by parenthesis?
I suppose, it can't expand in the batch-parser-phase, as a block can consist of many commands and the delayed expansion take effect when a line is executed.

(
set var=one
echo !var!
set var=two
) | more

Obviously the !var! can't be evaluated in the batch context, as the lines are executed only in the cmd-line context.

But why it can be evaluated in this case in the batch context?

echo !var! | more

In my opionion this is a "bug" or inconsitent behaviour, but it's not the first one

EDIT: Adding the LF trick

As dbenham shows, there seems to be some limitation through the cmd-behaviour that changes all line feeds into &.

(
  echo 7: part1
  rem This kills the entire block because the closing ) is remarked!
  echo part2
) | more

This results into
C:\Windows\system32\cmd.exe /S /D /c" ( echo 7: part1 & rem This ...& echo part2 ) "
The rem will remark the complete line tail, so even the closing bracket is missing then.

But you can solve this with embedding your own line feeds!

set LF=^


REM The two empty lines above are required
(
  echo 8: part1
  rem This works as it splits the commands %%LF%% echo part2  
) | more

This results to C:\Windows\system32\cmd.exe /S /D /c" ( echo 8: part1 %cmdcmdline% & rem This works as it splits the commands %LF% echo part2 )"

And as the %lf% is expanded while parsing the parenthises by the parser, the resulting code looks like

( echo 8: part1 & rem This works as it splits the commands 
  echo part2  )

This %LF% behaviour works always inside of parenthesis, also in a batch file.
But not on "normal" lines, there a single <linefeed> will stop the parsing for this line.

EDIT: Asynchronously is not the full truth

I said that the both threads are asynchronous, normally this is true.
But in reality the left thread can lock itself when the piped data isn't consumed by the right thread.
There seems to be a limit of ~1000 characters in the "pipe" buffer, then the thread is blocked until the data is consumed.

@echo off
(
    (
    for /L %%a in ( 1,1,60 ) DO (
            echo A long text can lock this thread
            echo Thread1 ##### %%a > con
        )
    )
    echo Thread1 ##### end > con
) | (
    for /L %%n in ( 1,1,6) DO @(
        ping -n 2 localhost > nul
        echo Thread2 ..... %%n
        set /p x=
    )
)