What is the difference between device tree and ATAGs? Also, are ATAGs a must and does the kernel expect them at a fixed address, or does it expect them in r0-r3?
A device tree describes everything about the hardware which the kernel uses to select which drivers to load, where all the MMIO interfaces are, etc... at runtime. ATAGs just describes stuff like where to find an initrd and kernel parameters, memory, etc... - everything else about the machine is hard coded into the kernel.
The preferred method now is to use device trees instead of ATAGs. One of the advantages include the fact that adding a new platform doesn't always require adding new code to the kernel.
To answer your second question, if you had read the documentation for booting Linux on the ARM platform, you'd find that the kernel expects the following to be placed in the registers before control is handed over to the kernel:
r0 = 0,
r1 = machine type number discovered in (3) above.
r2 = physical address of tagged list in system RAM, or
physical address of device tree block (dtb) in system RAM
I believe there is (or at least was) an option to load ATAGs from a fixed location instead of the address found in r2. However, the use of ATAGs is now being deprecated and obsolete and new platforms shouldn't be using it.