I want to use a HSM (hardware security module) to create a signature of a XML file. I did a bit of research and am now a bit confused, though.
Can you please clarify those questions:
What is meant with a key handle in JCE?
A key handle (in JCE, PKCS#11, or most other cryptographic APIs) is simply a reference that enables you to use a key without seeing its actual value. That is good: you can have the key permanently stored in a secure place (e.g. an HSM) with the assurance that nobody will be able to copy it and run away with it - as it may happen if the key is the application space. Unlike a physical safe though, you can still perform cryptographic operation without running any security risk of key leakage.
Does the PKCS#11 standard define a way so that the signature is generated in the HSM?
PKCS#11 is a C API for cryptographic tokens. A token is a PKCS#11 abstraction for any device or program that offers services described by such API. The API defines which operations you can perform using the objects inside the PKCS#11 token: some objects are non sensitive, and can be extracted (e.g. public keys); some others are sensitive and can only be used, via handles.
If you have a handle to an object that supports signing, you can use the C function C_Sign to ask the token to authenticate some data provided by your application. The key does not leave the HSM.
The featurelist of my HSM states JCE and PKCS#11 separately. What does that mean?
Your HSM supports JCE in the sense that it comes with a native library that qualifies as a Cryptographic Service Provider.
It supports PKCS#11 in the sense that it comes with a native library that offers a C PKCS#11 API.
I thought PKCS#11 is a standard, and JCE defines classes to use that standard. Does JCE specify its own protocols?
Indeed PKCS#11 is a standard; but it is not directly usable by languages other than C. You need a mapping layer that translates it into something compatible to your language. A PKCS#11 library (and the physical tokens that it abstracts) can be mapped to a JCE provider.
However, a JCE provider may have nothing to do with PKCS#11.