Allow the KonversAI bot on your website

Summary

KonversAI reads your website with a fetcher that identifies itself as KonversAI-Bot. Bot protection in front of a website, for example a web application firewall or a content delivery network, can refuse that fetcher. The KonversAI console then shows Your server is blocking us from fetching data on the Start import screen, or Failed on a page in a website import. This page shows you how to recognise the block and what to ask your web team to change. KonversAI reads public pages only.

Who can do this

A tenant administrator, or any team member whose role has the View knowledge base and Modify knowledge base permissions, sees the message in the KonversAI console. The change on your website is made by the person who administers your website’s bot protection or content delivery network. That is often your web agency or your IT department.

Before you start

Have this text ready to send to your web team. The KonversAI fetcher sends this user agent with every request:

KonversAI-Bot/1.0 (+https://konvers.ai; contact@konvers.ai)

The name in brackets in the console message tells you which bot protection refused the request, for example cloudflare, cloudfront, akamai or imperva. When the console says generic, KonversAI could not tell which product refused the request. Your web team knows which one you use.

Steps

1. Recognise the block. On Knowledge > Import > + Import in the KonversAI console, select the Website card, type your address in Website URL and select Check. A blocked website shows an amber line: Your server is blocking us from fetching data (cloudflare, HTTP 403). The name and the number differ from site to site. A website import that already exists shows the block on the Sync status tab instead. The page has the status Failed and the text Your server is blocking us from fetching this page. Ask your web team to allow the user agent “KonversAI-Bot”.
2. Send your web team this request: Allow requests whose User-Agent contains KonversAI-Bot on our whole public website, including the sitemap file. Add the full user agent string from Before you start. Ask for a rule that matches on the user agent, not on an IP address.
3. Ask your web team to make the rule in the bot protection product your website uses:
   • Cloudflare: add a firewall or WAF rule that skips bot checks when the user agent contains KonversAI-Bot. Bot Fight Mode needs the same exception.
   • Amazon CloudFront with AWS WAF: add an allow rule for the user agent before the bot control rules.
   • Akamai: add KonversAI-Bot as an allowed bot in Bot Manager.
   • Imperva: add the user agent to the allowed bots, or add an exception for it in the bot access control settings.
   • Another product: the same idea applies. Allow the user agent, and place the rule before the rule that refuses unknown bots.
4. Check again. On Start import, select Check. The line changes to Front page loads. Then select Create & configure import. For a website import that already exists, open the import under Knowledge > Import, open the Sync status tab, and select Re-fetch on a failed page, or select Sync at the top to fetch every selected page again.
5. If the check still shows the block after your web team made the change, wait a few minutes and select Check again. Some products need time to apply a new rule. If the block stays, contact KonversAI support and send the exact text of the message.

FAQ

What is KonversAI-Bot?

KonversAI-Bot is the name the KonversAI fetcher uses when it reads a website for a website import. Every request carries the user agent KonversAI-Bot/1.0 (+https://konvers.ai; contact@konvers.ai). The fetcher reads the public pages you select on the Inventory tab of a website import in the KonversAI console, and the sitemap file of your website. It does not log in, and it does not read pages behind a password.

Why does my website block KonversAI?

Many websites sit behind bot protection, for example Cloudflare, Amazon CloudFront, Akamai or Imperva. The default rules of these products refuse requests from a program they do not know. KonversAI-Bot is a program they do not know. The KonversAI console shows this as Your server is blocking us from fetching data. The fix is a rule on your side that allows the user agent KonversAI-Bot. Your web team makes that rule.

Is it safe to allow the KonversAI bot?

Yes. The KonversAI fetcher reads pages that you already publish to everyone. It identifies itself in every request, so your web team can limit the rule to the user agent KonversAI-Bot and nothing else. Your web team can remove the rule at any time. Removing the rule stops new fetches. The pages already in your knowledge base stay there.

Can my web team allow an IP address instead of the user agent?

Ask for a rule on the user agent KonversAI-Bot. KonversAI does not publish a fixed list of IP addresses for the fetcher. If your web team can only allow an IP address, contact KonversAI support.

The front page is blocked, but the sitemap check passed. Can I continue?

Yes. On the Start import screen in the KonversAI console, Create & configure import is enabled when the check shows Sitemap found even when the front page is blocked. KonversAI reads the addresses from your sitemap. A page that the bot protection still refuses gets the status Failed on the Sync status tab, with the text Your server is blocking us from fetching this page. Ask your web team to allow the user agent KonversAI-Bot, then select Sync to fetch those pages again.

How do I find every page that failed because of the block?

Open the website import under Knowledge > Import in the KonversAI console and open the Sync status tab. In the status filter, select Failed. Each blocked page shows the text Your server is blocking us from fetching this page. Ask your web team to allow the user agent “KonversAI-Bot”. Other failed pages show a different text, for example The page returned no content. or We could not fetch this page.

My web team allowed the bot. How do I make KonversAI fetch the pages again?

Open the website import under Knowledge > Import in the KonversAI console. Select Sync at the top of the screen to fetch every selected page again. To fetch one page, open the Sync status tab and select Re-fetch on that page. A page shows Ready when its content is in the knowledge base.