Order of OK, APPLY, CANCEL buttons
An extenstion of this question: https://stackoverflow.com/questions/50335/ok-cancel-or-cancel-ok
Where should the APPLY button go (clicking the APPLY button has the same effect as clicking OK, except the dialog remains open)?
Windows typically uses OK-CANCEL-APPLY, but my inclination is to use OK-APPLY-CANCEL.
Also, if the APPLY button is clicked, should the CANCEL button text be changed to CLOSE until another change is made in the dialog? I'm assuming the APPLY button will be disabled if there are no changes to apply.
Answer
To answer your first question, the Windows User Experience Interaction Guidelines for Windows 7 and Windows Vista specify the following order for your command buttons (p506):
OK
Cancel
Apply
Help
Now, are you smarter than Microsoft? Well, you’d hardly be the first, but you should prove it before releasing your design. Run a usability test on a bunch of users specifically creating scenarios to check if:
Your alternative button order produces superior user performance.
It does not cause a performance deficit when the user switches to another application that uses the standard order.
Go against the Windows UX Guide only if both of the above are demonstrated to be true.
Regarding your second question, I would recommend that you do not change Cancel to Close after the user selects Apply. A Close button usually implies that any subsequent changes cannot be reverted. Users may not have noticed the button’s initial caption thus may believe the dialog never supports canceling, making the user reluctant to explore the dialog further. Leaving the caption as Cancel assures users they can discard whatever changes they make next. If some users worry that Cancel will undo whatever they Applied, then I expect they’ll just select OK. In theory. Testing would once again tell you what users really think and do, and which design makes the best trade-off.
I’m with you that these OK/Apply hybrid single-use/multi-use dialogs are klugey and confusing. One alternative that gets around the whole problem is to use “immediate commit,” where whatever changes the user makes are instantly shown in the application (this may be a “property inspector” as the Windows UX Guide calls it). Immediate commit eliminates the need for OK, Apply and Cancel. Instead you have Close, and I would also suggest you also have an Undo button that works like an Undo menu item, sequentially reverting each change the user made with each selection. In addition to avoiding the OK/Apply/Cancel/Close confusion, this design is faster (fewer clicks to try a change), makes it clear what user input has what effect, and supports incremental undo (Cancel is all or nothing).