How to get a list of all URLs indexed by Google on a specific website

When I carrying out a technical audit of the website, often there is a need to make the list of all pages of the website, which are in Google index.

The formula of Google ranking works on two factors: internal and external. Internal factors are a set of all pages, which Google indexed on your website. Therefore, it is important to know what pages Google indexed on your website.

Besides, if you want to remove outdated content of the website, you will also need to make the list of website pages, which are in the Google index.

I will offer you two methods of making the list of the indexed pages. The first method is suitable for the websites, which consists of less than 120 pages. If your website has more than 120 pages, then you should choose the second method. The matter is that Google shows in a snippet only 120 results, when the quantity of the results is more than 120, it begins to hide the index.

The analysis of website indexation (less than 120 pages)

1. Enter the request in google

site: https://yourdomain.com

2. You will receive the list of the indexed pages in results page of Google.

Google SERP

Now we copy the results and insert them into Google Docs. At the end, we should get something like this.

From SERP to Doc

To separate addresses of the indexed URL pages, we have to do the following:
1. Choose the first column.
2. Sort results by names, from A to Z. This is how we get a range of URL addresses.

Google Doc for SEO

You copy this list and work with it as you wish.

The most obvious option of what to do with this information is to start the Screaming Frog program and choose the mode List and download your list into the program for the further analysis.

Screaming Frog

2. The analysis of big website indexation (more than 120 pages)

There is more in this than meets the eye.

To begin with, you will need to index the website by yourself by means of any popular program: Screaming Frog, Netpeak Spider or Xeny.

After you have completely scanned the website, you need to check the indexation of these pages with the help of the Seo Tools for Excel plug-in.

  1. Start Spider, choose the mode of indexation List.
  2. Specify in the program the address of the text file in which you saved the list of pages of your website.
  3. Choose the field GoogleCachedate and Url.
  4. Push Generate.

Scan website

Finally, you receive the list of URL addresses, and the date of indexation of these pages will be specified in the next column. If there is no page in index, then the date of indexation will not be specified.

Check which URLs have been indexed

The verification of pages in the Google index can be done by means of the Netpeak Checker program.

How to get a list of all URLs indexed by Google on a specific website
5 (100%) 1 vote

One thought on “How to get a list of all URLs indexed by Google on a specific website”

  1. Cool SEO Blog! Thanks for your detailed description of the process. I was thinking about carrying out a technical audit and now the question is about indexed pages aroused. I have looked through a lot of instruction, but yours is the best! Thanks to it, I could do this part of work myself without any troubles.

Leave a Reply

Your email address will not be published. Required fields are marked *