I want to know if JMS API uses any protocol to transfer messages or if uses its own. If the former, which protocol?
I have read many articles over the net but I couldn't find an answer for this.
The standard JMS API is merely a set of interfaces; JMS providers (such as WebSphere MQ) provide their own implementations for these interfaces.
The only thing that you can say for sure about all JMS implementations is that they all adhere to the JMS API; other than that, a JMS implementation may use any protocol whatsoever in order to fulfill the JMS API contracts.
Now, when you're asking specifically about "protocols", you should also define which "layer" in the communication you are referring to (have a look at the OSI Model, for example). When your JMS client has to talk to a JMS server that is located on another machine on the network (a typical case), the protocol used between the client and the server will be based, in one way or another, on TCP/IP. Over the wire, you'll be able to see TCP/IP packets being exchanged back and forth.
At the higher level, there are no guarantees; you are likely to find proprietary protocols varying between different implementors. Remember that, with JMS, performance is often crucial; JMS vendors put a lot of efforts into ensuring that their protocols ("above" TCP/IP) perform well. HTTP, for example, won't do.