Optimizing Source Material for LLM Assisted Customer Support

As an indie developer in the process of scaling a consumer-first SaaS product, I'm trying to ensure that customer service doesn't degrade as the user count increases. It's not ideal to throw more human support agents into the mix, especially when you consider the low revenue-per user in a consumer-first product.

In my experience, many customer support tickets can be answered by connecting multiple help docs and guides together, which a human agent already does on behalf of the customer.

A better approach is to have a human agent focus on improving the source knowledge base, then using a chatbot as a first-responder to questions that require connecting multiple dots together.

---

If you'd like to keep up with this series of posts, follow me on X for updates.

---

AI support chatbots are not what I'd call "plug and play," but I'll show you how to layer them on top of your existing support process to add significant value almost immediately.

Laying the Foundation for LLM Assisted Support

I've been helping people use technology in healthier ways with my B2C SaaS product, Tech Lockdown. However, the healthy technology/content filtering niche has a high support bar. Canned responses only go so far when user's support needs span multiple devices and problem spaces.

I decided to plug a AI chatbot into my knowledge base to see if it could potentially act as a 24/7 first responder for customer support.

After adding my sitemap as the training data for a chatbot, I was impressed with the responses.

Here's an example of the chatbot I use, SiteGPT, answering a question I've asked it.

This answer was pretty good and it provided the source material for the customer to go further with the topic, but it didn't quite answer the question 100% comprehensively. It left out a point about using iPhone shortcuts to re-enable a VPN.

Here is the follow up:

Ideally, the iPhone shortcuts option would have been included as a point in the response to the initial customers question.

This is where a chatbot can add significant value for internal use. You can ask questions against your knowledge base to find out where there are gaps. If the chatbot can't determine an answer, maybe your customers can't either.

Refining your Source Material

The first use-case for the chatbot is to use it as an internal support tool which can be used to refine your source material and improve the efficiency of your human support agents. Since chatbots can hallucinate information, you should field test it with real customer questions before releasing it into the wild.

The chatbot-assisted workflow is pretty simple:

1. Add your knowledge base sitemap to the chatbot's training data. This is simple to do with SiteGPT and it will crawl the URLs in your sitemap and examing the page content.
2. After the Chatbot trains on your sitemap, open the chat as if you were a customer and copy/paste a real customer question into the chat window.
3. Compare the expected answer with the chatbot's answer and then create a follow up ticket to investigate if the chatbot couldn't answer the question correctly.

You can use the thumbs-up button for correct answers to help refine future responses.

If the chatbot's answer needs refinement, you should examine the sources used to answer the question and try to improve the source material instead of simply overriding the chatbot's answer.

LLM-friendly Source Material

Making your knowledge-base LLM-friendly is not that difficult to do. In fact, if you are following SEO best practices and creating content in a way that is easy to understand and links together, you may not have much to do.

Is your content structured in such a way that you could assign filters to it?

If not, this is an indicator that your general content tree lacks structure and a chatbot might not be able to link related concepts together.

Traditional content search/filtering is a key part of self-service support user experience since customers often don't know what question to ask.

Filtering provides a starting point and requires the author to think about where the content fits within the overall content tree.

If a customer knows what question to ask, can they can they find the answer by searching your knowledge base?

If not, a chatbot might not be able to answer the question correctly either.

You might discover that your search engine is not indexing content well enough to answer questions correctly.

If you are only indexing titles and excerpts, you'll likely miss answers to questions that are within the context of a guide. The search I designed for my product's knowledge base allows a user to jump to specific anchor points in a guide.

In the case where source material exists, but search indexing is poorly optimized, the chatbot still might be able to answer the question, but your site's search results are probably not as helpful as they could be.

Best Practices for Content

When you are creating guides and help docs in an effort to make them LLM-friendly, your main focus needs to be on making sure that your content is easy to understand and links together. A chatbot will evaluate content in a similar way that a human would, so your content should still be created for people first.

There are a few best-practices that are often overlooked which I want to re-iterate here.

Interlinking Related Content

Using inline internal links to associate content is good for user experience, search engine optimization, and helping a LLM find related content.

I'm specifically talking about inline citations to other content that fits naturally within the context of the current content.

Additionally, these internal hyperlinks should describe the content being linked to in a few words.

For example:

// BAD
To learn how to do this, [click here](/how-to-block-ads)
// GOOD
Read our guide on [how to block ads](/how-to-block-ads)

Another best practice is to use descriptive slugs for your content; ideally indicating topic specificity.

For example, here are two guides that talk about a similar topic but with different levels of specificity:

/block-websites
/block-websites-iphone

Your internal hyperlinks and the keywords used in your page URLs can help a chatbot or search engine crawler understand the relationships between topics.

The parent-child-sibling relationship should be respected.

With this in mind, you should be picky with your internal hyperlinks and the keywords used in your page URLs. Don't link to content that isn't related, and don't use keywords that aren't related.

Conclusion

An AI chatbot will continue to be a key part of my support stack and I'll continue to iterate on the chatbot's involvement with supporting my product. If you'd like to keep up with how I'm using AI to scale, follow me on X for updates.

If you'd like to try out the chatbot I'm using, you can sign up with my SiteGPT referral link here.