I have looked it up but haven't been able to find many differences between purging and deleting a set of files. Is there more to PURGE than simply BACKUP AND DELETE?
EDIT: Specifically, this is about trace files and log files in Oracle
Purge runs on a regular basis. It delete things that marked as delete. You can set up criteria. For example, you can set criteria that to remove all the rows have a flag as delete, and remove those rows from a table.
Delete removes the rows on going. For example, if you have a work queue, and the tasks will be finished after the queue, you can delete those rows since they are not needed anymore.