Structured Data Analysis Best Practices

Joe_Gardener · 2025-06-30T16:34:30+00:00

There was an error rendering this rich post.

I am building a Bill of Materials Comparison tool. I have the agent doing about 85% job well. The issue I am experiencing is honing down the non-deterministic nature of the LLM even configuring Factual and providing details step instructions to try to ensure 100% accuracy. Anyone have any luck..

Find more posts tagged with

Glean Agent

Accepted answers

Blair

Hey Joe,

Given the nature of LLMs, perfect determinism will be difficult, if not impossible, unfortunately. That said, you may get better results using a couple of techniques.

For steps with a defined expected output, provide the LLM with an example of what you're looking for after the written instructions in the step. For tables, you can create those tables in Google Docs and then use the "Copy as Markdown" feature to get a version that the Agent can read correctly.
Claude especially tends to respond well to instructions that remind it to be careful, or avoid certain pitfalls.
Once you have the generated output, you can then use another Think step to analyze the output compared to the information available to the agent and revise it accordingly.

However, even with these instructions, there's no way to guarantee 100% accuracy or compliance — that's just a restriction of the underlying technology. Good luck!

All comments

Gleanie Kat

@Joe_Gardener Have you tried switching the model for the reasoning step or adding a think step with more specific instructions? We've seen better results with a different model (like 4.1 mini, o3-mini, or Gemini 1.5 Pro).

Joe_Gardener

Hey Kat…. Thanks for the quick response. I have a pretty detailed 9 step process that provides Read Step Instructions, then individual think steps that have detailed instructions for each doc that needs to be analyzed in the workflow. Ex. Golden BOM vs New BOM and maintain its 1 to 1 mapping, extract sku's between BOM versions and analyze discrepancies like, new items, matching items, mismatched items, removed items. Provide those sku extract to see the count of skus between BOMs and then provide a full detail BOM with this information that can be exported to Excel or Word. I have Claude Sonnet 4 as the LLM. I can test but this needs to be 100% deterministic since we cannot afford missing or wrong items.

Blair

Hey Joe,

Given the nature of LLMs, perfect determinism will be difficult, if not impossible, unfortunately. That said, you may get better results using a couple of techniques.

For steps with a defined expected output, provide the LLM with an example of what you're looking for after the written instructions in the step. For tables, you can create those tables in Google Docs and then use the "Copy as Markdown" feature to get a version that the Agent can read correctly.
Claude especially tends to respond well to instructions that remind it to be careful, or avoid certain pitfalls.
Once you have the generated output, you can then use another Think step to analyze the output compared to the information available to the agent and revise it accordingly.

However, even with these instructions, there's no way to guarantee 100% accuracy or compliance — that's just a restriction of the underlying technology. Good luck!

Joe_Gardener

Blair. Awesome. Thank you very much. I really appreciate everything from Kat and yourself.

Bector

Hey @Joe_Gardener. In addition to what @Blair said -

"count the number of characters in a string" - deserves a deterministic output and it can be hard to get GPT models to give a deterministic output just on that 😀. LLMs are predictive in nature so the games we play are just to reduce the lack of determinism as opposed to aiming for being deterministic.

Some food for thought :

You could have different models perform some of the same steps (at-least critical ones) and consider comparing/blending the output. I also try to have it "double check it's work" using a different model and if so, you could have it re-run somelogic.

Using model X, produce output
Feed output to Model Y and have it compare it to what it would have produced. Have it check if model X output missed something and store the <something> in a <missed-item-list>
If <missed-item-list> is not empty, re-run model X. OR add another think step to merge output from model X and model Y removing duplicates.

As a bonus step, you could have Model Y do "critical rating" on the items that model X missed and only include "critical items" in the <missed-item-list>

Joe_Gardener

Bector. Awesome. Thanks for more detail. So we want to have QA steps after initial Think Steps that is performing the data analysis to check the previous step to audit it. So Step 2 Think Step Claude review Golden BOM and Step 3 Think Step using 4.1 mini to audit the step 2 telling it from the the whole workflow context or the only look at the step prior information and have that repeated for each BOM to then feed the remaining workflow. SO once the 2 BOMs are done performing that analysis then audit serially, then compare those results for an updated compared BOM. Correct?

Bector

Sounds like a plan @Joe_Gardener . It'll likely take some experimentation but sounds like fun to me!

Bector

The only other thing I'd recommend is - once you find a combination that works well with a couple queries/questions, be sure to evaluate it on a few more examples. Sometimes, it takes a little bit of eval to fine tune the steps.

Joe_Gardener

Yes. Will do. Thanks

Getting Started

Events

Help Center

glean.com