How an orchestration engine works
I have several questions on this topic. For example, I have found a lot of papers like "Towards Dynamic Orchestration of Semantic Web Services" "Decentralized Orchestration of Composite Web Services" and so on... but in practice, I have only found orchestration linked to Bitztalk or ESB (I mean big software programmers).
Is it possible to develop and orchestration language yourself? What is the best way to develop an orchestration engine?
Answer
Perhaps the best source of information on Purpose and Application of Orchestration are the papers themselves which you quoted.
Background
We surf the web on Firefox, type a document on Microsoft Office. These are centralized applications. These types of software sit and work from one place. They work on your computer, they work on my computer.
You go to a supermarket, pickup an item and check-out from one of the many cash counters. Each clerk at each desk has his own barcode scanner and his own swipe card reader. Each of the cash registers on these counters are linked to one server in a back room somewhere. In this setup, the billing software is distributed across the clerk's counters, but the whole application is still centralized. The server manages the stock and records the sales. It is the center of the centralization.
Then you write an email and send it. Say you send an email from your Gmail account to my Hotmail account. There is Gmail's server, and there is Hotmail's server. There are two centers, instead of just one. Now things are no longer centralized - we have a distributed system. Here, failure of one center does not cripple the whole system. If Hotmail goes down, Gmail still survives.
Rather than purchasing from local supermarket, purchase something from an online store. Say eBay or Amazon. In this example, there is eBay's server, and there are the suppliers to eBay. The suppliers manage their own inventory on their own servers, not on eBay's server. There is also the courier company which brings the package to the buyer. The buyers have their own servers as well. The online payment that the buyer made was through MasterCard, yet another separate server. Now we are talking about a really big distributed system.
Purpose
Now that you are making an online purchase, a hell lot of things are bout to happen, which would need more than a bunch of servers. There has to be a master puppeteer who synchronizes activities among these servers. An account has to be deducted. An email has to sent. A warehouse has to be notified. A courier needs to be arranged. who controls this intricate dance? This is your orchestrator.
Application
Most of the time there are many different and independent servers, each owned by different entities. Yet when all these entities need to work together to create a business flow, a "user interaction session", we need orchestration.
Orchestration of the activities among a set of servers is achieved through a master-puppeteer server. In reality, the orchestrator is itself a set of servers. So one set of servers directs another set of servers. These second set of servers is where actual work is being done. eMails are being sent. Images are being compressed. Addresses are being sorted, and so on. The first set of servers (the orchestrator) makes sure things happen in the order they needs to happen.
Implementation
One answer: queues. The one activity that started this whole story was an attempt to make an online purchase. From there, the clicks you made, the commands you sent, were all queued up in these orchestration servers. The command like purchase-this-item or make-a-payment and then payment-received are all queued and then processed one after another.
The orchestration system accepts these commands on one thread, and on a different thread, the orchestration server asynchronously dispatches them to the respective worker servers. So the purchase-this-item command is dispatched to eBay's server while make-a-payment command is dispatched to MasterCard's server.
The worker servers might produce further commands. The MasterCard server, after validating your card number, might decide to send you an SMS. so it add a send-sms command to the queue. That command is dispatched to, say, a Vodaphone server.
This queuing and dispatching logic is called the "orchestration engine". From there, things can complicate. Vodaphone server might be offline. MasterCard might reject the card. The warehouse server might return an out-of-stock response. Then that response will need to be routed to eBay's server which could re-queue the purchase command to some different warehouse. There are server crashes. Disk failures. Power outages, and so on.
Finally
Orchestration is to make sure that so many diverse components, distributed geographically, in different points of time, some are parallel, some are faulty, some are slow, some are malicious, some are illegal,.. all work together towards getting you the damn headphones that you purchased over internet.