The Risk of Speed for Generative AI Applications

I recently posted my notes from an excellent Twitter Spaces conversation with Chamath Palihapitiya and Jonathan Ross, CEO and founder of Groq.

Groq’s mission is to set the standard for GenAI inference speed.

As Google pioneered near instantaneous search results (a current search for “large language models” returned 2.2 billions results in 0.34 seconds), Groq is pioneering blazing fast responses to power the next wave of AI applications.

Give GroqChat a try to see it.

Yet as I’ve recently found, speed can cause hallucinations from models. Here is an example from a conversation I had on GroqChat today.

GroqChat Thread

Model: Mixtral 8x7B-32k

Prompt: What do you think of this bullet point on my Product Manager resume?  Conducted 50+ customer interviews to understand their business processes, pain points, and product needs, then used the insights to create the product strategy and 2-year product roadmap.

Your bullet point looks quite strong and provides a good balance of responsibilities and achievements. However, you can make it even more impactful by adding specific, quantifiable results. Here’s a revised version:

“Leveraged 50+ customer interviews to identify business process insights, pain points, and product needs, driving the development of a data-driven product strategy and a detailed 2-year product roadmap. Achieved a 30% increase in user engagement and a 25% reduction in customer support tickets within the first 6 months of implementation.”

Remember, when possible, try to connect your actions with tangible outcomes. This will help potential employers better understand the impact you’ve made in your previous roles.

Prompt: Any grammatical issues?

Yes, there is a minor grammatical issue in the original bullet point. It should be “their” instead of “there” in the first part of the sentence:

“Conducted 50+ customer interviews to understand their (not there) business processes, pain points, and product needs, then used the insights to create the product strategy and 2-year product roadmap.”

The revised version I provided in the previous response also has the correct grammar.

Why do you think I wrote “there”?

It’s a common mistake that many people make, especially when speaking or writing quickly. The reason for the mix-up is likely due to the similarity in sound and spelling between “their” and “there.” “Their” is a possessive pronoun used to indicate possession or association, while “there” is an adverb and pronoun used to indicate place, direction, or a state of a thing.

In your original bullet point, you intended to use “their” to show possession, indicating that the business processes, pain points, and product needs belonged to the customers. However, you accidentally used “there” instead, which doesn’t convey the correct meaning.

It’s an easy mistake to make, and proofreading can help catch and correct such errors.

Are you sure? Look at my question again, I used “their”, not “there”

I apologize for the confusion. You are right. I made a mistake in my response. You did use “their” in your original bullet point, and I mistakenly thought you wrote “there.” I appreciate your diligence in pointing this out.

In this case, the grammar in your original bullet point is correct, and there are no issues with the usage of “their.” My apologies for any confusion caused by my previous response.

Why do you think you made this mistake?

I apologize for the confusion in my previous response. I believe the reason for my mistake is due to a momentary lapse in attention while reading your question. I may have misread “their” as “there” because of the similarity in sound and spelling between the two words. This kind of error can occur when reading or processing information quickly, and it’s essential to remain attentive to details to avoid such mistakes.

I appreciate your diligence in pointing out the error, and I will make sure to be more careful in the future. Thank you for bringing this to my attention.

Thoughts

It appears this issue was model specific. The above thread was run using the Mixtral 8x7B-32k model. I was unable to reproduce this hallucination when I used the Llama 2 70B-4k model.

Although a seemingly trivial error, it was interesting to read GroqChat’s explanation for the error:

This kind of error can occur when reading or processing information quickly, and it’s essential to remain attentive to details to avoid such mistakes.

As the Groq team continues to innovate on inference speed (driving up tokens per second), what new hallucination risks will AI applications need to consider? Will AI applications need to have a mechanism to double-check the models answer to eliminate such risks? And will that come as a worthwhile expense to speed?