“Site indexing robots.txt: rules for setting up the file”

[subornaakter20]

Complete
A complete analogue worsens the internal optimization factors of the entire site and makes it difficult to advance to the TOP, which is extremely undesirable. Therefore, they must be eliminated immediately after detection.



UTM tags . They are used to transmit data for advertising analysis and the source of transitions. Search engines usually do not index them, but sometimes there are exceptions.

GET parameters in URL . Sometimes when transferring data, GET parameters get into the address:


Sections generated by a referral link . They are usually distinguished by a special parameter added to the URL. Such a link should have a redirect to a regular URL, but this is often neglected.

If you set up the 404 error section incorrectly , it nurses email list will cause an infinite number of duplicates. Any characters in the site address will automatically turn into a link and will return a 404 error without a redirect.


Read also!

Read more
You can stop duplicates from appearing with a full match by setting up a redirect, removing the error programmatically, or closing the resource from indexing.

Partial
Duplicates with partial matches have less impact on the site. It is difficult to say how many duplicates of site pages are acceptable. But if there are too many, it reduces the resource's position in search results and also hinders its promotion for specific key queries. It remains to find out how partial duplicates appear on the site.

Development of a multi-page website

Source: shutterstock.com

Characteristics in the product card

The product description on the page may have additional tabs, such as reviews or specifications. By switching to one of them, you can track the URL change, although most of the information does not change. This creates a duplicate.

Pagination

If the content management system is set up incorrectly, then when you go to the next page in one category, the URL changes, but the Title and Description remain the same. As a result, we get several different links with the same meta tags:



The search algorithm will index these URLs as different documents. This can be avoided by simply checking the implementation of product output and autogeneration from the technical side.

In addition, each pagination document will need to specify the canonical page as the main page on the site.

Substituting content

When resource owners want to increase the number of visits for queries specifying a city, they add a region selection button to the site. When you go to another city, the phone number on the contact page changes. But there are cases when such an argument as, for example, "wt_city_by_default=.." appears in the address bar. As a result, the document gets several duplicates with different links. Such generation is unacceptable, try to use 301 redirect.

Print version

Implies an exact copy of the contents of a section, but in a different, converted format, for example: