I keep jumping into projects where Modernizr is located in the head, but it isn't being used (at least for feature detection). Now, I love Modernizr, I use it frequently for feature detection and fallbacks on projects that require it; however, the last three projects I've come into it's been sitting in the head without any of the feature detection classes being called. These projects are using vanilla javascript and/or do not require jQuery polyfills at all. 1 That being said... if you're not using the feature detection and really don't need to load a jQuery library 2, is Modernizr really doing anything aside from making an addition HTTP request and resource to load?
I'm not strong enough with jQuery/javascript to understand if it's affecting anything else under the hood.
Edit
1 & 2 — Modernizr is javascript and doesn't require the jQuery library (which makes me wonder why the jQuery library is being loaded also, at least in these cases).
Modernizr.min with only #-shiv-cssclasses-load
is 7.57 KB whereas html5shiv.min is only 3 KB.
Generally speaking, Modernizr does three things.
If you don't use any of these features, then there's no point in including Modernizr at all. If you only use it for the html5shiv then you could probably just include that instead to save a few bytes, although I doubt there's a relevant size difference at all.
That said, you should never include feature detects in a modernizr build that you don't use. That's nothing but a waste.