Every time a video/audio streaming starts the Internet Download Manager (IDM) feels that there is a multimedia content on the page and allows you to download that multimedia content.
I don't think it's possible to prevent IDM from grabbing video/audio but is there a way to fool IDM to get a wrong video instead or make it very hard for IDM to guess which is the real video/audio playing?
EDIT: Actually My Question was from the server side point of view not the client running an IDM. So i guess the misunderstanding here is my fault. Many good answers here but not the one i was looking for
You need to open "Options->General" IDM dialog and press "Edit" near "Customize IDM download panels in browsers".
Then enable "Don't start downloading from web-players automatically" option (arrow 1 on the image) and press "OK" (arrow 2 on the image).
source: http://www.internetdownloadmanager.com/register/new_faq/video14.html