ShellExecuteEx returning 42 in hInstApp when expecting a 31 (No file association)

Question

When using ShellExecuteEx in Delphi 7 to open a file using a verb, I always seem to be getting 42 back as a result in hInstApp, even though I'm expecting to get a failure and a 31 result, as there is no file association. I am moving from ShellExecute, to ShellExecuteEx so that I can use WaitForInputIdle with the process handle.

ShellExecute returns a 31 as expected when I try to open an XLS file when I dont have Excel installed, but ShellExecuteEx appears to succeed and return 42, even though it has actually failed and popped up the default Windows file association dialog.

Am i doing something wrong? Using Delphi 7 on WinXP and Win7.

Sample code below. Using Delphi 7 on a Win XP 32 bit OS, but also getting the same results on Win 7 64 bit. Just doing a showmessage on the hInstApp value returns 42 when I would expect to get 31 as I don't have Excel installed.

var
  ExecInfo: TShellExecuteInfo;
begin
  ZeroMemory(ExecInfo, sizeof(ExecInfo));
  with ExecInfo do
  begin
    cbSize := sizeOf(ExecInfo);
    fMask  := SEE_MASK_NOCLOSEPROCESS;
    lpVerb := PChar('open');
    lpFile := PChar('c:\windows\temp\test.xls');
    nShow  := 1;
  end;
  ShellExecuteEx(@ExecInfo);
  if ExecInfo.hInstApp<32
  then WaitForInputIdle(ExecInfo.hProcess, 10000);
end;

Answer 1

ShelLExecuteEx return values are not the same as ShelLExecute. Read the documentation here: http://msdn.microsoft.com/en-us/library/bb762154%28v=VS.85%29.aspx. Also check you have set the proper flags in SHELLEXECUTEINFO for the proper behaviour when an error occurs.