So this question isn't about integrating an existing payment gateway into my site. This is more of a architectural question.
I want to build a system similar to Paypal. Now I understand that Paypal offers a lot of features under the roof and I can't implement all of them at once. I want to implement the core functionality of Paypal and other such services.
So my question is (rather discussion is) around how would one go about building such a system. Some points to discuss:
Thoughts?
What you're talking about is becoming a payment service provider. I have been there and done that. It was a lot easier about 10 years ago than it is now, but if you have a phenomenal amount of time, money and patience available, it is still possible.
You will need to contact an acquiring bank. You didnt say what region of the world you are in, but by this I dont mean a local bank branch. Each major bank will generally have a separate card acquiring arm. So here in the UK we have (eg) Natwest bank, which uses Streamline (or Worldpay) as its acquiring arm. In total even though we have scores of major banks, they all end up using one of five or so card acquirers.
Happily, all UK card acquirers use a standard protocol for communication of authorisation requests, and end of day settlement. You will find minor quirks where some acquiring banks support some features and have slightly different syntax, but the differences are fairly minor. The UK standards are published by the Association for Payment Clearing Services (APACS) (which is now known as the UKPA). The standards are still commonly referred to as APACS 30 (authorization) and APACS 29 (settlement), but are now formally known as APACS 70 (books 1 through 7).
Although the APACS standard is widely supported across the UK (Amex and Discover accept messages in this format too) it is not used in other countries - each country has it's own - for example: Carte Bancaire in France, CartaSi in Italy, Sistema 4B in Spain, Dankort in Denmark etc. An effort is under way to unify the protocols across Europe - see EPAS.org
Communicating with the acquiring bank can be done a number of ways. Again though, it will depend on your region. In the UK (and most of Europe) we have one communications gateway that provides connectivity to all the major acquirers, they are called TNS and there are dozens of ways of communicating through them to the acquiring bank, from dialup 9600 baud modems, ISDN, HTTPS, VPN or dedicated line. Ultimately the authorisation request will be converted to X25 protocol, which is the protocol used by these acquiring banks when communicating with each other.
In summary then: it all depends on your region.
Once you are registered and accredited you'll then be able to accept customers and set up merchant accounts on behalf of the bank/s you're accredited against (bearing in mind that each acquirer will generally support multiple banks). Rinse and repeat with other acquirers as you see necessary.
Beyond that you have lots of other issues, mainly dealing with PCI-DSS. Thats a whole other topic and there are already some q&a's on this site regarding that. Like I say, its a phenomenal undertaking - most likely a multi-year project even for a reasonably sized team, but its certainly possible.