Keep a website import up to date

Summary

You manage an existing website import on the website import screen in the KonversAI console. Select Knowledge in the left sidebar, select Import, find the website card on the Imports screen, and select View. The screen has two tabs: Inventory, where you choose which pages stay in the knowledge base, and Sync status, where you see how each page is doing. This page is about an import that already exists. To make your first import, read Connect your website as a knowledge source.

Who can do this

A tenant administrator, or any team member whose role has the View knowledge base and Modify knowledge base permissions. An administrator sets these on Settings > Users & permissions in the KonversAI console. The View knowledge base permission alone lets you read the import but not change a rule. If you do not see Knowledge > Import in the left sidebar, you do not have this access. Ask an administrator to change your role.

Before you start

You manage an existing website import in the KonversAI console. Select Knowledge in the left sidebar, then select Import. The Imports screen opens with one card for each import. Find the website card and select View.

The import must have finished its first discovery run. A card that still shows discovery in progress has no inventory to manage yet.

Website import is not behind a feature flag. Every tenant has Knowledge > Import.

Know what you want to change before you start. Adding a folder to the knowledge base and removing a folder from it use the same control, and a removal deletes the documents for those pages on the next sync.

Steps

1. Select Knowledge in the left sidebar of the KonversAI console, then select Import. Find the website card on the Imports screen and select View. The website import screen opens on the Inventory tab.
2. Read the top line under the import name. It shows Added with the date, the number of pages crawled, the number of pages ingested, and Sitemap detected when KonversAI found a sitemap.
3. On the Inventory tab, read the heading URLs selected for sync, with the count in the form 12 of 40 selected for sync. Use Expand all and Collapse all to open and close the folder tree.
4. Tick a folder to add every page under it, or untick it to remove them. KonversAI writes the sync rules for you. A tick adds an include rule, or deletes the exclude rule that kept the folder out. An untick deletes the include rule, or adds an exclude rule when a wider include rule still covers the folder. KonversAI shows Updating sync selection… while it saves. A folder tick covers pages that appear under that folder later, so tick the folder rather than each page.
5. Tick or untick a single page for a page that does not belong with its folder. A single page tick covers that one address only.
6. Scroll to Existing rules below the tree. The section explains: Sync rules decide which URLs are kept in sync with your knowledge base. When new pages appear on your website, matching rules determine if they are downloaded automatically. An empty list reads No sync rules yet. Each rule row shows an include or exclude badge, the address, and either include sub-pages or only this page.
7. Select Add rule to write a rule by hand. The window Add a rule opens. Type the address in URL, for example https://example.com/news/.
8. Choose Include these URLs in the knowledge base to keep the pages, or Prevent these URLs from being downloaded to keep them out. An exclude rule always beats an include rule, so use it to cut one folder out of a larger included folder.
9. Leave Apply to all URLs matching this prefix ticked to cover every page under that address. Untick it to cover the one address only.
10. Read the live count below the field. It shows Checking matches… while KonversAI counts, then a line such as 8 URLs match this rule. with up to 5 example addresses, or No matches. Select Add rule to save, or Cancel to stop.
11. Select Edit on a rule row to change it. The window Edit rule opens with the same fields, and the save button reads Save changes.
12. Select Delete rule to remove a rule. KonversAI asks Delete this rule? Matching URLs will be deselected on the next sync pass. Confirm it.
13. Select the Sync status tab. The heading is Sync status, with a line in the form 31 of 40 URLs currently in sync and 9 pending sync. A green dot marks a page that a rule covers.
14. Filter the list. Type in Filter URLs by title or path…, or choose All statuses, New, Processing, Ingested or Failed in the list beside it.
15. Read the badge on each row: New, Crawling, Processing, Ready, Failed or Excluded. Ready means the page content is in the knowledge base. Select View document to open the document.
16. Act on one page from its row. Select Ingest to fetch a page with the status New or Failed. Select Re-fetch to fetch a page with the status Ready again. Select Remove to delete the address and its document from the import; KonversAI finds the address again on the next discovery run when the page is still on your website, so use an exclude rule to keep it out for good.
17. Select Sync at the top of the screen to run the whole import now. KonversAI reads your website again, fetches the new and changed pages that a rule covers, and deletes the documents for pages that no rule covers any more. The Sync button appears after the first discovery run completes.

FAQ

How often does KonversAI refresh a website import by itself?

KonversAI refreshes each import on its own timer. A sitemap import refreshes every 60 minutes. A crawl import refreshes every 7 to 10 days. KonversAI adds a fixed spread of up to 3 days to each crawl import, so the heavy re-crawls do not all start on the same day. KonversAI checks for due imports every 10 minutes. You cannot change the interval in the console today. To refresh now, open the import in the KonversAI console under Knowledge > Import, select View, and select Sync at the top of the screen.

I added new pages to my website. When does the assistant know about them?

On the next refresh, when a sync rule covers them. KonversAI reads your sitemap again, adds the new addresses to the import, and fetches the pages that a rule matches. This is why a folder rule is better than a page rule: a rule on https://example.com/news/ with Apply to all URLs matching this prefix picks up every new article by itself. Open the import in the KonversAI console under Knowledge > Import, select View, open the Inventory tab, and check that a rule covers the folder. Select Sync to fetch now.

How do I stop KonversAI from importing one folder of my website?

Open the import in the KonversAI console under Knowledge > Import and select View. On the Inventory tab, untick the folder in the tree. Or select Add rule, type the folder address in URL, choose Prevent these URLs from being downloaded, leave Apply to all URLs matching this prefix ticked, and select Add rule. An exclude rule always beats an include rule, so this works even when a wider include rule covers the same folder. The documents for those pages disappear from the knowledge base on the next sync.

What is the difference between the Inventory tab and the Sync status tab?

Both tabs are on the website import screen in the KonversAI console, which you open with Knowledge > Import and View. The Inventory tab is where you decide: it shows the folder tree with tick boxes and the sync rules below it, under the heading URLs selected for sync. The Sync status tab is where you check the result: it shows only the pages a rule covers, with a status badge for each one, under the heading Sync status. Change what is in the knowledge base on Inventory. Find out why a page is not in the knowledge base on Sync status.

The Sync status tab says no URLs are in sync. What did I do wrong?

The message No URLs are currently in sync. Use the Inventory tab to means no sync rule covers any address in this import, so KonversAI has nothing to fetch. Open the website import in the KonversAI console under Knowledge > Import, select View, and open the Inventory tab. Tick the folders you want, or select Add rule and write a rule with Include these URLs in the knowledge base. Then select Sync at the top of the screen. When the list is empty but the count is not zero, the message reads No in-sync URLs match the filter. — clear the text in Filter URLs by title or path… instead.

A page is in the Excluded URLs list. How do I bring it back?

Use Restore. In the KonversAI console, select Knowledge in the left sidebar, then select Import. Scroll to the Excluded URLs section at the bottom of the Imports screen. The section explains: Excluded URLs allow you to select all URLs in an import while keeping some explicitly excluded. Each row shows the address, Excluded on with the date and the person, the reason, and an Import: link back to the import. Find the address, or type in Filter by URL, import, source, or reason, and select Restore. KonversAI answers Restoration started for the address, and fetches the page again. The list reads No URLs currently excluded. when nothing is excluded. A page lands in this list when somebody selects Exclude URL on the document, not when a sync rule leaves it out.

My import has thousands of pages and the tree has no tick boxes. Why?

An import with 1000 or more addresses uses a different Inventory tab in the KonversAI console. The heading reads URL inventory instead of URLs selected for sync, the tree shows folders only, and a line below it says the import has that many URLs and tells you to use the sync rules to manage which paths stay in sync. Ticking each page one at a time does not work at that size. Select Add rule on a folder row, or use the Add rule button under Existing rules, and manage the import with rules. The Sync status tab then also shows the cards Total discovered, Selected for sync, In sync, Pending sync and Failed.

A page shows Failed on the Sync status tab. What now?

Failed means KonversAI could not fetch or read that page. Open the website import in the KonversAI console under Knowledge > Import, select View, and open the Sync status tab. The reason appears in red under the address. Three causes are common: the page needs a login, so KonversAI cannot read it; the page returns an error; or the page has no readable text. Select Ingest on the row to try again. When the page needs a login, exclude it with Add rule and Prevent these URLs from being downloaded, and bring the content in with Knowledge > Import > + Import and the File upload card instead.

The Sync button gives the error “Sync is already in progress.” What do I do?

Wait. Sync is already in progress. on the website import screen in the KonversAI console means a run is already going. A banner near the top of the screen shows what KonversAI is doing: Crawling website… with the number of addresses found, Processing… while it fetches and stores pages, or Building URL tree… while it rebuilds the folder view. The screen updates by itself. A run that stops responding is reset after 2 hours with no activity, and the import then shows the status failed, so you can select Sync again.

What happens to my documents when I untick a folder?

KonversAI deletes them. On the Inventory tab of a website import in the KonversAI console, under Knowledge > Import and View, unticking a folder deletes the include rule for that folder. When a wider include rule still covers the folder, KonversAI adds an exclude rule instead. Both ways leave the pages outside your sync rules. On the next sync, KonversAI deletes the knowledge document for every page that no rule covers, and the page returns to the status New. The assistant stops answering from that content at once. To keep the content but stop the updates, do not untick the folder; copy the text into a manual document under Knowledge > Documents first.