Is there a way to tell the Chrome debugger (or maybe Firebug?) to not break within certain files? To assume they're not broken, essentially? This seems like something they might build in.
Blackboxing JS files is now possible in Firefox https://developer.mozilla.org/en-US/docs/Tools/Debugger
And in Chrome Canary using Experimental Dev tools. http://www.divshot.com/blog/tips-and-tricks/ignoring-library-code-while-debugging-in-chrome/
Update. In Chrome v.75 there is a separabe tab for blackboxing.
Above works in stable Chrome build as well now.