Can anybody shed some light on how Magento determine whether to create a new URL rewrite for a product? Every time I run the Catalog URL Rewrite reindex process the number of rows in core_url_rewrite increases by roughly 10,000 rows. Since no product data has been modified in the mean time, why is a new URL generated?
This problem still exists in Magento 1.7, but I have no clue about 1.8 or 1.9. Magento does not delete old entries it seems. This can slow down your database seriously. The easiest way to solve this is to truncate the table and to reindex it.
truncate table core_url_rewrite;
php -f indexer.php -- -reindex catalog_url
Then check in the admin: Catalog > URL Rewrite Management, and see if the new entries have been created. If this shows no items, URL rewrites won't work!