What is a dataset in your Shop Bot Pro control panel?
A dataset is a collection of database records where each individual records contains a specific site URL, the text content of the URL resource, an AI generated search results excerpt (summary of the resource) and some other optional features such as a search results image.
The dataset is used by the AI tools to generate the intelligent, relevant responses to user questions and searches. The content in the dataset record’s index field which is generated based on the text content of the URL resource is used by the AI engine to compare the user’s search query with your site content and provide the most relevant resources based on their intent. Although we sometimes refer to the text content of the dataset record as the “index”, the actual dataset URL index is another field in your dataset that is stored as “vector” (embeddings) data. The embeddings field is used with the OpenAI language model to return the AI relevance ranked content results as well as AI generated resource summaries.
You can insert more content into individual dataset record’s text to provide more accurate responses for specific search situations such as Product SKUs and IDs. The default content that is indexed by the Shop Bot Pro “scraper” is only the content that is available on the URL. However, you are free to add more content to any dataset URL record to best match your site search requirements.
Whenever you make a change to a dataset record, the data is automatically reindexed by our AI engine. We also will periodically scan your site URLS for changes to your content. If you want to exclude a specific record from being reindexed by our site scraper, please mark that record as such. If you manually add content to a dataset record for Product IDs, SKUs or specific keywords or phrases that you want to directly target, you should exclude that record from auto-indexing which will prevent your custom content additions from being replaced by the site URL’s default page content.
You can have a unlimited number of data sets in your account and each dataset can have different URLs or you can have the same URLS segmented into specific datasets for knowledge base documents, online store products or other content that you want separated from the main site search.
