How do I allow Google to index login-required parts of my site?
It seems like Google can index certain sites or forums (I can't name any offhand as its been months since I last saw one) and when accessing you are prompted with a request to register or login.
How would I make my site open for Google to index and have a regular login for others?
Answer
Google supports the concept of flexible sampling where you can show a limited amount of content to users from Google search results while allowing Googlebot to crawl all of it.
Google allows you to either show users a limited number of articles (metering) or show a portion of each article (lead-in).
From a technical standpoint you need to:
- Allow Googlebot to access all your content by user agent or by validating Googlebot's IP addresses.
- Mark up your pages with structured data that indicates that they have paywall content.
- Implement showing limited amounts of content to users who come to your site from Google.
This is a replacement for Google's older "First Click Free" policy under which you had to show a full page of content every time somebody clicked to your site from a Google search result.