How to accelerate the indexing of the site in Yandex and Google: a full guide after publication

You launched your site through Claude Code, put together a landing page in Cursor, published an article — and wait. Day, two, three. The search is empty. The search engine doesn't know you exist yet.

That’s okay, but only if you don’t do anything. If done correctly, the first pages appear in the index in 1-3 days, not in 2-4 weeks.

In this guide - a complete order of action: from the sitemap file to IndexNow and Yandex. Webmaster. No water, just what really works in 2026.

## Why the page is not indexed at all

Before speeding up, you need to understand how it works.

Search robots (Googlebot and Yandex.Bot) constantly bypass the Internet through links. When they find a new page, they analyze its content and decide whether to add to the index or not. Without indexing, the page does not exist for search – it is physically there, but it cannot be found in the SERPs.

The problem is that robots don’t know your site’s schedule. A new site without external links may not be found for weeks. A site with thousands of pages does not crawl every page every day – it distributes a “crowling budget” and visits pages according to priority.

Your task is to give the robot as many signals as possible that the new page exists, is important and ready for indexing.

## Step 1. Make sure the page is open for indexing

This is the first thing to do before any other action. It sounds obvious, but this is where thousands of sites are lost, assembled through AI tools with no SEO experience.

### Check out robots.txt

The robots.txt file at the root of the site tells the robots what they can and cannot touch. Open in your browser:

https://ваш-сайт.ru/robots.txtXX

A normal file for an open site looks like this:

User-agent: *
Allow:

Sitemap: https://ваш-сайт.ru/sitemap.xml

If you see Disallow: /, your site is completely closed from indexing. This is a common problem with sites assembled through builders or generators: they include a default indexing ban “at development time” and then forget to take it down.

### Check the noindex tag

Open the source code of the page (Ctrl+U in the browser) and find the line:

<meta name="robots" content="noindex">

If there is one, the page is clearly excluded from indexing. Remove this tag or replace it with:

<meta name="robots" content="index, follow">

### Check the server response status

The page should give the code 200 OK. If it gives 301, 302 (redirects), 404 (not found) or 500 (server error) – the index will not get.

You can check through Yandex. Webmaster → Tools → Check the server response, or through online tools like httpstatus.io.

## Step 2. Set up sitemap.xml correctly

Sitemap is a file map of the site. He tells the searcher, “Here are all the pages you need to know.” Without a sitemap, the robot has to independently find pages through links, which takes much longer.

### Basic sitemap structure

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://ваш-сайт.ru/</loc>
    <lastmod>2026-06-07</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://ваш-сайт.ru/blog/статья-1/</loc>
    <lastmod>2026-06-07</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

### What works in 2026 and what doesn’t

<lastmod> - use honestly. This is the only optional tag that really affects indexing in 2026. Google uses it to prioritize bypassing: if the date is fresh and real, the page gets priority. The key word “real” – putting the same date on all pages or updating it on a schedule without real changes – is counterproductive. The search engine notices this and stops trusting your lastmod.

<changefreq> and <priority> - Google ignores both. This is Google's official position in 2026. Yandex formally takes into account changefreq, but you should not rely on this. You can specify for order, but do not waste time on their optimization.

In sitemap, only work pages. Do not include redirected URLs, noindex tagged pages, 404 pages, duplicate content, and non-canonical versions. The inclusion of “dead” URLs in the sitemap confuses robots and spends a crawling budget.

Limits: One sitemap – maximum 50,000 URLs and no more than 50MB unpacked. For larger sites, use sitemap_index.xml, which links to multiple child maps.

### How to generate a sitemap

• WordPress: Yoast SEO or Rank Math plugins automatically generate
• Next.js/Nuxt: built-in generation or next-sitemap library
• Static sites (Hugo, Astro, 11ty): Built-in sitemap generator
• Any site: xml-sitemaps.com (for smaller sites)
• Through AI: ask Claude Code or Cursor to write a sitemap generation script for your stack

After creating the file, add a link to robots.txt:

Sitemap: https://ваш-сайт.ru/sitemap.xml

## Step 3. Register with Yandex. webmaster

Yandex. Webmaster is the main tool for Russian traffic. In 2026, this is not an option, but a basic necessity: this is where you see how Yandex sees your site and can directly influence indexing.

### Addition of the site

1. Open webmaster.yandex.ru
2. Click "+" → Add the URL of the site
3. Confirm ownership: upload an HTML file to the root of the site, or add a meta tag to <head>, or add a TXT record to the DNS

### Add the sitemap

After confirmation: Setup → Sitemap files → Add a file. Insert the full URL of your sitemap.xml.

### Turn on "Counter bypass"

This is a little-known but powerful setup. When the user comes to the page with Yandex. Metrico and Webmaster are aware of this visit - it automatically receives a signal to bypass this page out of turn. Live traffic literally speeds up indexing.

Settings → Metric → switch “Covering the counters”. To work: The metric must be installed on the site, the counter is tied to the same account as the Webmaster.

### Request a manual retake

Tools → Page Review → Enter the URL of a particular page. Limit: A few pages per day. Use it for the most important pages or after major changes.

### What to watch on the Webmaster

Indexing → Search Pages – How many pages are actually in the index.

Indexing → Excluded pages – why specific pages are not indexed (no text, duplicate, forbidden in robots.txt, etc.).

Tools → Server Response Check – Quickly check what the server gives to a specific URL.

After 2-5 days after adding a sitemap, check the “Search Pages” section. If the pages are much smaller than in the sitemap – see “Excluded pages” and understand the reasons.

## Step 4: Google Search Console – What to Do in 2026

Here is an important nuance for Russian projects. Creating new Google Search Console accounts for sites with Russian domains and audiences has become much more difficult. However, if your site is aimed at an international audience or you already have an account, use it.

### Addition and verification

1. Open search.google.com/search-console
2. Add a site via Resource with Domain (covers all protocols and subdomains) or Resource with URL prefix
3. Confirm via HTML file, meta tag, Google Analytics or DNS record

### Send the sitemap

Indexing → Sitemap files → Add. Google will start processing the file within a few hours.

### Request indexing through “URL Checker”

This is a direct way to ask Google to index a specific page:

1. Insert the URL in the check bar at the top
2. Click on “Request Indexing”

It is important to understand that this is a request, not a guarantee. Google will add a page to the queue, but the timing and result is at its discretion. Usually the result is visible in 1-7 days.

### Coverage report

The most important section for diagnosis. Shows:

• Valid - Indexed
• Valid with warnings - indexed, but there are problems
• Excluded - not in the index, with cause
• **Errors ** Critical problems that interfere with indexing

Frequent reasons for exclusion are: “Detected – no crawl planned” (Google found, but did not consider priority), “Duplicated page without canonical tag”, “Page blocked robots.txt”.

## Step 5. IndexNow is the fastest way to notify searchers

IndexNow is an open protocol developed by Microsoft. It lets your site instantly notify search engines of the changes — not wait for the robot to come on schedule, but say, “Hey, here’s a new page, come here.”.

Yandex has supported IndexNow since 2021. Google doesn’t officially support it, but Bing does — and Bing and Google have an agreement to share data on new URLs.

### How IndexNow Works

1. Generate a unique key (any string, usually a UUID)
2. Place the file with this key in the root of the site: https://ваш-сайт.ru/ключ.txt
3. Send a POST request to https://yandex.com/indexnow with the page URL and key
4. Yandex receives a notification and puts the page in the priority queue of the bypass

Yandex does not guarantee instant indexing, but it speeds up the queue.

### Manual setting (one URL)

First, create a key file. For example, the abc123xyz key:

Create abc123xyz.txt file at the root of the site
File content: abc123xyz
The file must be available at: https://ваш-сайт.ru/abc123xyz.txt

Then send a notification (can be via curl):

curl -X POST "https://yandex.com/indexnow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "ваш-сайт.ru",
    "key": "abc123xyz",
    "urlList": [
      "https://ваш-сайт.ru/новая-страница/"
    ]
  }'

Server responses:

• 200 OK - accepted page in line
• 400 Bad Request - error in the query (check JSON)
• 403 Forbidden – the key does not match the file root
• 422 Unprocessable Entity – URL does not belong to the host or is already being processed
• 429 Too Many Requests – Exceeded the request limit

### Automation through CMS

If you’re on WordPress, Yoast SEO and Rank Math plugins support IndexNow out of the box. When a new record is published, they automatically send a notification to Yandex.

For Next.js, Nuxt, and other frameworks, ask Claude Code or Cursor to write a hook that triggers depletion and sends IndexNow for modified pages. This is a simple task: webhook + POST query on the IndexNow API.

An example of a script on Node.js:

async function notifyIndexNow(urls) {
  const key = process.env.INDEXNOW_KEY;
  const host = 'ваш-сайт.ru';
  
  const response = await fetch('https://yandex.com/indexnow', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ host, key, urlList: urls })
  });
  
  console.log('IndexNow status:', response.status);
}

// Вызывайте после каждой публикации
notifyIndexNow([
  'https://ваш-сайт.ru/новая-статья/',
  'https://ваш-сайт.ru/обновлённая-страница/'
]);

## Step 6. Internal Linking: An Underappreciated Accelerator

Search robots crawl the site through links. If no link from already indexed pages leads to a new page, it can wait for the first visit of the robot for a very long time, even if it is in the sitemap.

A page without internal links is called an “orphan page.” This is one of the most common mistakes on sites assembled through AI: each page is created separately, there is no connection between them.

What do you do

• After publishing a new article, add a link to it from 2-3 already indexed pages on similar topics
• Check that the navigation menu or “Last Articles” block includes a new page
• Use the 3-click rule: any important page should be accessible a maximum of 3 clicks from the main page

This will not only speed up indexing, but also pass PageRank to a new page from already authoritative pages of the site.

## Step 7. External signals: Telegram, VK, Zen

Live traffic to a page is one of the strongest signals for search engines. If real users begin to click on the link, the search engine receives a signal of interest and bypasses the page out of turn.

Where to place links immediately after publication:

Telegram: If you have a subscriber channel, post it. The audience goes → the robot notices traffic → speeds up the bypass. Yandex actively indexes Telegram channels, so a link from Telegram additionally creates an external signal.

**Publishing in a community with a live audience has a similar effect. VK is well indexed by Yandex.

Zen and Medium: If the material is suitable for these platforms, post an announcement or a brief retelling with a link to the original.

Important: this is not about cheating traffic, but about honestly attracting a real audience. Robots can distinguish between live users and bots.

## Step 8. What to check if the page is not indexed

More than a week has passed - a page in Yandex. Webmaster in the "Excluded" section. What to watch:

** "Robots prohibit indexing"** - check robots.txt and the noindex meta tag on the page.

**No text is a page with almost no text content. Add at least 200-300 words of text.

Duplicate Page / Page is a Mirror – if you have multiple similar URLs (www and no, slash and no, http and https) – configure the canonical <link rel="canonical"> tags and redirects to one version.

"Not updated for too long" - update the page, change the lastmod to the sitemap, request a revisit.

** "Breaking the Rules"** - the page violates the searcher's policies. Read the specific reason on Webmaster.

## Next Next post: What to do immediately after publication

This is the minimum set of actions. It takes 15-20 minutes, gives maximum effect.

● Check: robots.txt is open for indexing (no Disallow: /)
● Check: no tag <meta name="robots" content="noindex" >>
● Check: page gives 200 OK
● Add a page to sitemap.xml with a real lastmod date
● Send IndexNow notification to Yandex
● In Yandex.Webmaster: Tools → Page Cover → Add URL
● In Google Search Console: Verify URL → Request Indexing
● Add internal links to a new page with 2-3 pages
● Publish a link in Telegram / VK
In 2-3 days: check the status in the Webmaster and GSC

## Real deadlines: what to expect

These are landmarks, not guarantees. The indexing speed depends on the authority of the domain, the quality of the content, the technical condition of the site and the load on the search engine servers.

## Automation through AI: How to do it with Claude Code

If you’re vaibcoding your site, it’s logical to automate indexing through the same tools.

Ask for Claude Code or Cursor:

Sitemap automatic generation script – reads the content folder, generates XML with real lastmod dates from the git file history.

** Hook after deploit** - each depletion automatically detects the changed pages and sends IndexNow.

Accessibility Checker Script - After publication, checks that all new pages give 200 OK and are not closed by noindex.

An example of a problem for Claude Code:

Write the script on Node. js which:
1. Reads all .md files from /posts folder
2. Generates sitemap.xml with real dates from fs.stat
3. After generation sends IndexNow request to Yandex
Pages with a change date in the last 24 hours
4. Add an env variable to the IndexNow key

These are 20-30 lines of code that completely automate technical SEO when published.

## Outcome

Indexing doesn’t work on its own, especially on a new website. The search engine should be directed: give a site map, notify about changes, show a lively interest of the audience, eliminate technical blocking.

Order of priorities:

1. **First, make sure there are no technical locks (robots.txt, noindex, server status)
2. Then – set up sitemap.xml and register with the Webmaster
3. At each publication - IndexNow + revisit request + internal links
4. System – Automate through scripts or plugins

If you’re a vibcodite, automation through Claude Code or Cursor shuts it down once and for all.