Glean Automatic Slack Answers

EmptySea

With regard to Glean Bot answering Slack questions and posting a message that states.

The stated issues are:

1. People who see the question don't know what response was given from Glean so Subject Matter Experts cannot verify or challenge the response which can result in someone accepting incorrect or incomplete information which has been confidently presented to them by Glean.
2. This response from Glean without active follow up from the asker in the thread implies to readers that Glean has successfully answered the question, so Glean has effectively automatically shut down the conversation by saying "don't worry, I already answered the question." If it turns out that Glean hasn't answered the question and the asker hasn't noted such yet or hasn't gotten back to the thread, everyone who saw that thread in the meantime could have bounced, thus reducing engagement when the asker rejects Glean's response.

This raises some thoughts:
1. If someone else clicks the 'Generate' button and gets an answer is it the SAME answer that was presented to the original person or does it depend upon the access to documentation (in this case an App that all engineers should have access to all materials that it uses but could be very different if it was using the default Glean Assistant based bot).

2. Some users are not interacting with the answer, so we dont know if they read the answer and simply did not engage with the accept/reject which potentially masks the problem OR a solution (if the answer was correct) from any other person reading the thread.

3. If Glean bot only answers when its high confidence, should be allow that answer to be presented rather than masked?

4. Could there be a reddit style up/down voting on such an answer (emoji or a link in the generated answer to allow feedback) so others could weigh in on the quality, accuracy etc of the provided answer. If Glean has a high confidence, for instance, but the data being used is out of date or inaccurate, then full exposure of the answer could also help correct source information.

dcieslak

This is precisely why we do not deploy the Glean Answer Bot and have written our own instead. With our homegrown bot, we can present Glean's answer to the "channel experts" who can then approve the message, copy parts that were correct and add further nuance, or simply reply with the correct answer.

Our custom bot also has a "test mode" to allow "channel experts" to gain confidence that the bot will respond correctly. In our opinion, the worst thing is for the bot to give plausible-but-wrong answers (or even answers that were once correct but are now outdated). The bot will respond in a temporary channel with the original user question and it's response. The channel experts can then get a sense for how Glean's responses will work. We often will set targets for "correctness" (e.g. the bot must be right 50% of the time)

There was also, at some point, a way to get Glean's assessment of the "accuracy" of the response. This would be a useful metric to expose and allow be set on a per-channel basis (e.g. HR may want 90% accuracy, while another department might be fine with 30% accuracy)