Most AI-powered search or knowledge retrieval tools use data-indexing. It&rsquo;s chosen because it&rsquo;s supposed to provide quick and efficient retrieval of relevant data.
Typically, it works by copying data to a vector database, which uses algorithms to index and store numerical representations of the data &mdash; so it can be retrieved more efficiently.&nbsp;
Although data-indexing is the most common approach it has a number of drawbacks, particularly for enterprise deployment, with reduced security, poor precision, and complex data management.&nbsp;
<h3>1. Reduced data-security</h3>
One of the most significant problems of data-indexing for enterprises is that it risks compromising the security of company data.&nbsp;
The process of indexing effectively creates a copy of your data, usually on a third-party server. This may make searches quicker in some cases, but it increases the potential attack surface for hackers and massively increases the potential damage of a data breach.&nbsp;
In addition, rather than having information stored on multiple distributed applications, indexes bring all that data together in one place. Ordinarily, if a specific app has a data-security problem, the damage is limited to the information stored within that app. But if the data-index provider is breached, the hackers could access to a much larger range of data, including everything you have indexed.&nbsp;
One might think that vector embeddings don't pose a major security risk, as the numerical representations don't mean anything on their own. However, <a href="https://arxiv.org/pdf/2310.06816.pdf" rel="follow noopener" target="_blank">a recent study from researchers at Cornell University</a> found that 92% of the meaning can be recovered exactly. The conclusion was that "embeddings should be treated with the same precautions as raw data".&nbsp;
<h3>2. Poor precision&nbsp;</h3>
The second problem with data-indexing is precision and accuracy, which degrades very significantly for larger datasets. This is because data-indexes use a vector database to index information.&nbsp;
Vector databases work by transforming the source information into a vectorized numerical form, where it is then stored in high-dimensional space. Nearest-neighbor algorithms are used to resolve searches, identifying the vector in the dataset with the smallest distance to the query vector and marking it as the nearest neighbor.
However, as more and more data is added, the volume of the space increases exponentially, and the data becomes increasingly sparse. This sparsity means that the distance between many data points converges in relative terms, making it difficult to distinguish between a genuine nearest neighbor and a non-relevant data point.
The practical consequence of this loss of precision is that the prevalence of false positives will increase. This means the AI generated answer may be grounded in real information from the business, but the lack of precision means it&rsquo;s using the wrong information to generate the answer.&nbsp;
This is different to the problem of hallucination where information in a large language model&rsquo;s (LLM) training data is used to resolve queries, generating completely fabricated responses.
<h3>3. Complex data management</h3>
The third problem with data-indexing is the complex data management that comes with it. Enterprise companies generate large volumes of data from hundreds of software applications. This adds significant complexity (and cost) to the data management requirements to enable data-indexing, not least because the index needs to be kept constantly up to date if it is going to be relied upon.&nbsp;
While the solution provider will take some of the load, there is still a large burden of responsibility on the customer to make this possible. This might include:
<ul>
<li>
Data mapping - Creating accurate mappings from diverse data sources to guide the indexing process requires a deep understanding of the data and its structure.
</li>
<li>
Managing duplications - Handling duplicate information across different systems is critical to maintain the integrity and usefulness of the indexed data.
</li>
<li>
Monitoring data pipelines - Monitoring the data pipelines for errors, latency, or other issues is crucial for maintaining the reliability of the system.
</li>
<li>
Data cleaning - Data may need to be cleaned or transformed to ensure it&rsquo;s consistent and high-quality, which is fundamental for the effectiveness of the solution.
</li>
</ul>
Making this happen across an enterprise sized dataset that&rsquo;s constantly expanding and shifting is no easy task. For this reason, it&rsquo;s entirely possible that you&rsquo;ll need a dedicated team whose job is to manage internal data systems, so that the indexing works as it should. This will add significant overheads to the real cost of the solution or take up valuable time that could be better spent elsewhere.&nbsp;&nbsp;
<h3>The alternative&nbsp;</h3>
The good news is that there are much better alternatives. Qatalog&rsquo;s <a href="https://qatalog.com/resources/action-query/">ActionQuery AI engine</a> uses a combination of federated search (using real-time APIs) and retrieval augmented generation (RAG). This approach has a number of advantages:
<ul>
<li>
Secure - It&rsquo;s highly secure as there&rsquo;s no data transfer or duplication needed, as the relevant information is pulled directly from the source in real time and discarded after use.
</li>
<li>
Accurate - Even with large datasets, it delivers consistently accurate query resolutions, making it much more scalable.&nbsp;
</li>
<li>
Easy to manage - The data management is much less onerous, as there&rsquo;s no data-index to maintain and, therefore, no need for constant updates from hundreds of tools.&nbsp;
</li>
</ul>
<h4>Request<a href="https://qatalog.com/demo/?utm_medium=&amp;utm_source=&amp;utm_campaign=blog&amp;utm_content=data-indexing" rel="follow noopener" target="_blank"> a demo</a> to learn more or <a href="https://qatalog.com/resources/action-query/" rel="follow">download the ActionQuery whitepaper</a>.</h4>

Data-indexing might be the most common approach, but it has a number of drawbacks for enterprise customers.

Why data-indexing is a problem for AI-powered search

Keeping track of tasks and projects is crucial for getting work done, but when there&rsquo;s too much to do it&rsquo;s easy to lose sight of where your priorities lie.
That&rsquo;s where our new&nbsp;<a data-saferedirecturl="https://www.google.com/url?q=https://cu.qatalog.com/e/c/eyJlIjoxMDcyMDYsImVtYWlsX2lkIjoiZXhhbXBsZSIsImhyZWYiOiJodHRwczovL2hlbHAucWF0YWxvZy5jb20vaGMvZW4tdXMvYXJ0aWNsZXMvMTczNTcwNzUyNzA3OTc_dXRtX2NhbXBhaWduPUp1bmUrJTI3MjMrLStQcm9kdWN0K05ld3NsZXR0ZXJcdTAwMjZ1dG1fY29udGVudD1KdW5lKzIzKy0rTkxcdTAwMjZ1dG1fbWVkaXVtPWVtYWlsXHUwMDI2dXRtX3NvdXJjZT1jdXN0b21lci5pbyIsInQiOjE2ODg2NTAwMDl9/54738fdff93245fac9f0032a59908c42fd428aaefef933608de3e0c13b0ace57&amp;source=gmail&amp;ust=1688739375167000&amp;usg=AOvVaw1XZBgsGIQkCiPUtYbMxHY4" target="_blank" href="https://cu.qatalog.com/e/c/eyJlIjoxMDcyMDYsImVtYWlsX2lkIjoiZXhhbXBsZSIsImhyZWYiOiJodHRwczovL2hlbHAucWF0YWxvZy5jb20vaGMvZW4tdXMvYXJ0aWNsZXMvMTczNTcwNzUyNzA3OTc_dXRtX2NhbXBhaWduPUp1bmUrJTI3MjMrLStQcm9kdWN0K05ld3NsZXR0ZXJcdTAwMjZ1dG1fY29udGVudD1KdW5lKzIzKy0rTkxcdTAwMjZ1dG1fbWVkaXVtPWVtYWlsXHUwMDI2dXRtX3NvdXJjZT1jdXN0b21lci5pbyIsInQiOjE2ODg2NTAwMDl9/54738fdff93245fac9f0032a59908c42fd428aaefef933608de3e0c13b0ace57" rel="noopener">Timeline View</a>&nbsp;comes in, to show you tasks, projects, campaigns, and more in a clear timeline so you can see what's due soon, what status things are, and more.
Try it out by going to any Module or Tasks browse views and switching the view from list to timeline, in the right-hand corner.
<h3 data-pm-slice="1 1 []">What else is new?</h3>
<ul>
<li>
Export from browse views: Whenever you have a specific filtered view on any of your Modules, you can download that data as a CSV file.
</li>
<li>Dashboard uplift: We've added more options for sorting and filtering your browse views.</li>
<li>
Version history for pages: Your pages now have version history, to ease your collaboration. You&rsquo;ll find it under the three dots menu on any page.
</li>
<li>Label filtering on tasks: You can now filter and group tasks by label</li>
</ul>
<h3>Keep up to date on everything Qatalog</h3>
If you want more updates on all things Qatalog, you can subscribe to our monthly newsletter by entering your email on the menu to your right.

Here’s what we’ve launched or updated in Qatalog in the past month.

What’s new on Qatalog: June 2023

What’s new on Qatalog: Updates from June 2023

If we're talking about work, every individual, team, and organization strives to achieve specific goals. But how can you make progress if you're not tracking it? Enter: <a href="https://help.qatalog.com/hc/en-us/articles/16175977405325" rel="follow">Measurements</a>.
Measurements allow you to track and measure specific metrics or progress toward particular goals. Essentially it&rsquo;s a flexible feature that enables you and your teammates to measure anything you want in Qatalog.
<h3>What can I use Measurements for?</h3>
<ul>
<li>
Goals setting and tracking: This is the primary use case for measurements. Whether you use OKR, SMART, or a different type of goal framework, measurements can accommodate it.
</li>
<li>
Key Performance Indicator (KPI) tracking and reporting: KPIs are great for tracking important, everyday metrics that you want to keep your eye on. Unlike goals, they may not have an explicit deadline or target value.
</li>
</ul>
<h2>What else is new?</h2>
<ul>
<li>
Weekly Slack Digest: Automatically generate a digest post to a Slack channel at the end of the week, to keep track of your Item&rsquo;s updates.
</li>
<li>
Workflow assignments: You can now execute them from mobile.
</li>
<li>
Kudos: New UX for Kudos on profiles.
</li>
</ul>
<h2>Keep up to date on everything Qatalog</h2>
If you want more updates on all things Qatalog, you can subscribe to our monthly newsletter by entering your email on the menu to your right.

When it comes to work, every individual, team, and organization strives to achieve specific goals. But how can you make progress if you're n

What’s new on Qatalog: May 2023

Do you have collaborators outside of your company? Giving them access to everything they need, without sharing any sensitive or confidential information is key to successful cooperation. Enter external Page sharing: a way to collaborate with stakeholders outside your company without them becoming a user of your Qatalog instance or having access to an entire Project (which you can also share externally if you want).
Read on to discover more exciting updates, or <a href="https://qatalog.com/demo/" rel="follow">book a call</a> with our friendly team to learn more.
<h3>What&rsquo;s new?</h3>
<img src="https://cdn.buttercms.com/YjteN7SRgusJXXqPXMUj" alt="undefined" width="699" height="352">
<ol>
<li style="font-weight: 400;" aria-level="1">SOC 2 Type II compliant: Our SOC2 compliance confirms that our controls and practices have been independently verified to protect the privacy and security of all our customer data. Soon we&rsquo;ll be sharing how we used Qatalog internally to make this process pain-free.</li>
<li style="font-weight: 400;" aria-level="1">External Page Sharing: You can now externally share Pages with people outside of your Qatalog, they will receive access only to the shared page and not to any other information within a Project (which you can also share externally).</li>
<li style="font-weight: 400;" aria-level="1">New integration: OKTA Universal Directory Sync.</li>
<li style="font-weight: 400;" aria-level="1">BambooHR auto-sync: This automatically pulls your information from BambooHR and keeps data automatically updated in Qatalog.</li>
</ol>

SOC2 compliance, external sharing of Pages and Projects, Okta integration and much more.

What’s new on Qatalog: September 2022

What’s new?