Executive Summary

In this video, I reviewed Guru Answers, OpenAI's large language model that organizes and makes sense of your own data. The product's potential to resolve the issues of redundant and unorganized data across multiple platforms is remarkable. Upon testing, the product was unwilling to provide an answer to a complex question but successfully synthesized information from various sources to answer a more ambiguous query. Despite some initial limitations, I found Guru Answers to be transformative and am excited about its future potential. My feedback for the development team is that even if the AI can’t provide a comprehensive answer, it should synthesize whatever information it has, appending a disclaimer if necessary.

Introduction and Product Praise (0:00 - 1:17)

Addressing the Beta Group

Hi there, beta group. I'm not entirely sure if this is the kind of beta feedback you're looking for, with video demonstrations of individual questions and interactions, but let's give it a shot.

Commendation for Guru Answers

First and foremost, thank you. This product—Guru Answers—is incredibly cool and enormously useful. I'm not sure how other organizations manage it, but in my small operation, we struggle with redundant data.

Huge Problem Solved: Redundant and fragmented info

This redundancy often arises when someone doesn't effectively search for the information they need. They assume it's not there and then create another card, leading to multiple blurbs on the same topic over time. It exacerbates the issue we all face of storing data across too many platforms, making it impossible to search or organize comprehensively.

Great Expectations

The prospect of using a large language model like Guru Answers to make sense of this mess is mind-blowing. To aim this AI at your own data and have it instantly provide accessible and organized information—this is fantastic!

Product Test and Initial Impressions (1:17 - 3:23)

Sharing Experience with OpenAI's GPT-4

Now, let's explore the product further. I've been using GPT-4 extensively for a couple of months, so I was a bit taken aback when I saw an OpenAI-based product declare its inability to answer a question. But I wasn't shocked, as I see similar "restricted" behavior with other OpenAI API based tools compared to the Chat GPT.

Comparison with Other Tool: Cocounsel by Casetext

I also have a subscription to co-counsel by Case Text, a GPT-4 based legal tool. Comparing Guru Answers to it, I find Guru Answers seems a bit more locked down, less creative, but it doesn’t hallucinate like ChatGPT, which is good.

Context and Testing of Guru Answers

Here's what I tried: One of my team members, who's new, wanted an overall comprehensive view of our properties—information that’s scattered across various small knowledge cards. We've never really consolidated all of this information before, and it's not in a single accessible place.

Outcome of Initial Testing

So, I thought I'd test Guru Answers. I asked it to provide a comprehensive overview of our portfolio and its individual properties. However, instead of an organized response, the system suggested some potential sources and recommended rephrasing the question or using the search feature.

Comparison with the Browser Extension

This response was similar to what I’ve seen with the browser extension. I was expecting Guru Answers to synthesize the information from various sources and deliver a comprehensive response, but that didn't happen (in this example).

Analysis of Successful Question and Feedback for Developers (3:23 - 6:01)

Instance of Successful Question Answering

Despite the previous hiccup, there was a case where Guru Answers worked beautifully. When I asked, "What are the details of Scofield?"—even though the question was quite ambiguous—the AI was able to provide a comprehensive and well-written answer. It managed to extract relevant pieces of information from different sources and present them coherently.

Puzzlement Over Inconsistent Performance

However, I'm left wondering why Guru Answers couldn't provide a synthesized answer to the previous, more comprehensive question. Especially when considering GPT-4's purported verbal IQ of 155.

Feedback for Dev Team: When in doubt, give an answer but also a disclaimer.

My feedback for the development team is that even if the AI can’t provide a comprehensive answer, it should synthesize whatever information it has. If the AI is unsure about meeting the 'comprehensive' requirement, it could append a disclaimer indicating the information provided might not be exhaustive, but at least, it's something. It is quite jarring and creates some cognitive dissonance to see an LLM just say "there is no direct answer." This in itself is deceptive because LLM's do not "point to" "direct" answers - they create answers. The Guru Answer product subtly implies its answers simply point "directly" to information on cards already written by humans.

Conclusion and Gratitude (6:01 - end)

This Rocks

To conclude, I find Guru Answers to be truly magical. Today is one of the most exciting days of my life—yes, that's how much of a tech geek I am!

Closing Remarks

I am truly grateful for this amazing tool and would like to thank the development team for their work. I'm signing off now. See you later!

Thank you so much for this thorough feedback, @christopher reeves ! I’m checking in with our data science team to see if we can get any further insight into the performance of your first query. 

You’re welcome and thanks for making the product and asking us what we think. We live in interesting times, that much we can all agree on.