MochaUI is very intuitive and the modal iframes almost perfectly replicate Windows. Unfortunately, I have scripts written in Jquery that I use, and I hear there are conflicts when putting both Mootools and Jquery on one html file (is this true?).
How can I get the MochaUI features in Jquery?
At the very least, is there a similar modal dialog system?
I've seen JqueryUI Dialog but it makes the background go dark and nonfunctional, which is not what I am looking for.
With some tweaking (enable JQuery noConflict-Mode, replace the $ selector by something like $jq) you should be able to get to work both.
However you might easily end up with a Frankenstein app that mixes MooTools and Jquery plugins. Also the javascript load is quite enormous, because you have to include Mootools, JQuery AND their top-level UI toolkits.
I'd recommend that you investigate if JQuery UI could replace MochaUI (or vice versa). One combination of JS lib + UI toolkit is far easier to maintain.
The downside of MochaUI is that it has some issues with IE8, furthermore it is not very up-to-date (last release Sept 2008).
JQuery UI is much more modular (you can just pick the parts you need), has a visual CSS Theme Editor (Themeroller) and offers a 2-month release schedule. Its disadvantages: it doesn't yet cover essential UI components (such as a menu, a "viewport" manager that renders you UI components, palettes). However, these features are expected soon.