How to wait for a Power Query refresh to finish? I've tried "DoEvents" with "BackgroundQuery = False", and more, nothing works

Question

Setup:

• Windows 7 (at work)
• Windows 10 (at home)
• Excel 2016 (build 4627 at work)
• Excel 2016 (build 8730 at home)
• Power Query is setup to import, append, and transform a folder of Excel Files. This step works.

Description of Problem

After using any combination of any technique to wait for a Power Query to finish refreshing as described in the "Things I have tried:" section shown below. A message box can be displayed and any other code can be executed before the Power Query tables have finished updating according to the refreshing indicator (spinner?) icon shown in the Queries and Connections pane.

The exception to the above statement is the "OnTime" method of the Application class, shown in the "Code" section below, which doesn't appear to interrupt the polling for a power query refresh. However, the problem with this method is that it uses a hard coded amount of time to pause VBA code, and this will not always work since the size, amount, and duration of data being queried will change over time.

Things I have tried:

• I have read all the StackOverflow (and other website resources) that state the use of the "DoEvents", "BackgrgoundQuery = False", and CalculateUntilAsyncQueriesDone methods and properties.
• I have tried creating a class to Create Before/After Query Update Events as suggested at this link (not shown in the code example below).
• I have tried using Do Until/While Loops with the .Refreshing = True/False property of the QueryTable method to wait for a refresh to finish.
• I have tried setting the BackgroundQuery property in the Excel menu (menubar --> Data --> Connections --> Properties) to False as suggested by "subro" here: Wait until ActiveWorkbook.RefreshAll finishes - VBA, with an image of the Menu here:

.

Code:

Private Sub sht_sub_Refresh_AllConnections_dev()
'Name: sht_sub_Refresh_AllConnections_dev
'Purpose: An attempt at using VBA to wait for Queries to finish updating before displaying a message.
'Description: Waits for a hard coded period of time before dislpaying the message box.
'State: WIP.
'Dev: Needs a way to look at the connection stream to somehow detect when its finished.

  'DECLARATIONS:
  '------------'
  Dim procName As String              'Stores this procedure's name.
  Dim qTblLst As QueryTables          'A query table collection object.
  Dim qTblObj As QueryTable           'A query table object.
  Dim conLst As Connections           'A connection collection object.
  Dim conObj As WorkbookConnection    'A connection object.
  Dim idx As Long                     'A loop counter.

  'INITIALIZATIONS:
  '---------------'
  procName = "sht_sub_Refresh_AllConnections_dev"    'Store this procedure's name.
  Linit.ini_Setup_Project                            'Setup the project if needed.
  Set conLst = ThisWorkbook.Connections              'Set the connections list object.
  Set conObj = conLst.Item(conLst.Count)             'Set an initial connection object.
  idx = 0                                            'As an exit if the do loop continues without end.

  'MAIN CODE BODY:
  '--------------'
  'Turn off backgroundquery for each connection type.
  For Each conObj In conLst                           'For each connection object,
    With conObj
      Select Case .Type                               'Check the connection type,
        Case 1                                        'If its an OLEDB connection then,
          .OLEDBConnection.BackgroundQuery = False    'Set it's backgroundquery property to false.
        Case 2                                        'If its an ODBC connection the,
          .ODBCConnection.BackgroundQuery = False     'Set it's backgroundquery property to false.
      End Select
    End With
  Next conObj

  ThisWorkbook.RefreshAll                             'Refresh all connections.

'DEV: Using loops, DoEvents and a query name starting with the letters "zzzz" as suggsted here:
'https://social.technet.microsoft.com/Forums/en-US/bc3f7748-8a52-498d-951c-4566b8adf45a/in-excel-2016-power-queries-dont-refresh-in-the-background-anymore?forum=powerquery
'and here:
'https://www.myonlinetraininghub.com/excel-forum/vba-macros/pause-macro-until-power-queries-finished-refreshing
  'Attempt to wait until the last connection has finished refreshing.
  Do Until Linit.gvTbl_ZZZZZ.QueryTable.Refreshing = True   'Wait until the last table starts refreshing,
    idx = idx + 1                                           'Icrement a loop count,
    If idx > 3000 Then Exit Do                              'If the loop goes longer then 3000 iterations exit,
  Loop                                                      'otherwise continue waiting.
  VBA.DoEvents                                              'Do events before continueing (doens't work).
  Do Until Linit.gvTbl_ZZZZZ.QueryTable.Refreshing = False  'Wait until the last table finishes refreshing,
    idx = idx + 1                                           'Icrement a loop count,
    If idx > 3000 Then Exit Do                              'If the loop goes longer then 3000 iterations exit,
  Loop                                                      'otherwise continue waiting.
  VBA.DoEvents                                              'Do events before continueing (doens't work).
'DEV: The following is an attempt to get connections to
'     finish refreshing before code continues as suggested here:
'https://stackoverflow.com/questions/22083668/wait-until-activeworkbook-refreshall-finishes-vba
  Application.CalculateUntilAsyncQueriesDone         'This is placed here as well as after the refresh.
  VBA.DoEvents                                              'Do events before continueing (doens't work).
  Application.EnableEvents = False                          'Maybe turning off events helps? (nope...),
  Application.ScreenUpdating = False 'This is reset in the procedure called as an argument to the next line:
  Application.OnTime DateAdd("s", 3, Now), _
                     "Lwksh.sht_sub_Msg_RefreshDone"        'The called procedure just displays a message box.
  Application.EnableEvents = True                           'Restore events,
  Application.ScreenUpdating = True                         'Restore screen updating.

  'MEMORY CLEANUP:
  '--------------'
EXIT_CLEAN:
  procName = Empty                                     
  Set qTblLst = Nothing
  Set qTblObj = Nothing
  Set conLst = Nothing
  Set conObj = Nothing
  idx = 0
End Sub

Code Notes:

• Anything in the code preceded by "Linit." is an object or variable that is set globally outside the procedure by the "Linit.ini_Setup_Project" procedure call in the "INITIALIZATIONS:" section of the code.
• For example "Linit.gvTbl_ZZZZZ" is an object variable that points to an empty one row Table which has a name that stars with the characters "zzzz" and is generated by Power Query and loaded to an Excel sheet. The code shows the link to the website where the suggestion for using an empty table like this was made.

Questions

1. Is this a lost cause due to Power Query not having a built in callback to let Excel know that it has finished updating any refresh processes?
2. If this is not a lost cause, is there any other way, not described here, that could be used to trigger an error to occur in some way if a connection has not yet finished refreshing, or to trigger the error when a connection is finished? (The thinking here being that this error could be trapped as a possible way of detecting weather or not the refresh has completed provided the error doesn't stop the query from finishing).
3. Is there any way to probe the connection stream directly using VBA to look for the connection closed or finished state?
4. Is there any way to directly access the refresh process via some call to a program outside of Excel written in some other language such as C# or Python?
5. Can you think of anything else that might be tried or tested to make this work? I'll keep searching for an answer on my own, but after a full year of searching I'm feeling a bit out of luck.

Thank you for your time and attention to this issue. If I have forgotten to include any information or might need to reword something to make this question more clear, please let me know in the comments.

Answer 1

I understand your pain @neurojelly. I have been there. But as it turns out the solution is quite simple and is not using VBA. In the Query properties window, you need to uncheck the "Enable background refresh" and then use DoEvents. I know for sure this works as I have been using this method for over a year now.

Please find the link to a sample file that has the code in it.
https://drive.google.com/uc?export=download&id=1ZLxSMEXPLda3QhaQoTyGGv3_sC-tpN-X

As for your second question, it is possible to use Iferror/OnEror method's to detect if a query returns an error, but it doesn't necessarily check for errors in the query. It identifies if the query itself is returning an error pop-up which is skipped by default while running the VBA code. This method works most of the time, but not always.