Live is only the start
A page opening in the browser is a useful check. It does not tell you what a crawler can reach.
Check the address you actually publish
While publishing Website Chatbot Workshop, I found that the pages and downloads worked, but the shared hosting address served a robots.txt file that blocked crawling.
The file in my deployment folder allowed access. The file served by the host did not. Checking the local copy would have missed the difference.
Keep the observation narrow
That tells me about the published address at the time of the check. It does not tell me whether another host behaves the same way, or when a search engine will index a page.
- Checked
- Published pages, downloads and robots.txt
- Found
- Working pages; crawling blocked at the shared address
- Next check
- Repeat on the production address after any host change
Make access part of the launch check
I would keep the exact production URL in the launch record, open its robots.txt file and read the response the host actually serves. I would also check the page’s indexing directives, internal links and downloads on that address.
These checks answer different questions. A working link confirms a route for the visitor. A crawl rule concerns access for a crawler. Neither is proof that the page has been indexed or that it will rank.
Google’s robots.txt guide explains the distinction: blocking crawling does not reliably keep a URL out of search results.
Save what you found
Record the URL, date, response and any change you made. If you move the site, repeat the check on the new address. A clean local folder is useful preparation; the published response is the observation that belongs in the launch record.
For a wider check of technical access, I keep the page-access audit on the tools shelf. For this particular observation, the served file was enough to identify the restriction.