Store Evaluation Agent.

Before Intelligems could deploy on a store, our solutioning engineer had to evaluate the storefront by hand: walk the purchase funnel, identify the third-party apps and architecture in play, and decide whether complex integration work was needed. About 25 minutes of manual work per store, and none of it could be automated or chained into anything downstream. A manual step sitting directly in front of a revenue workflow.

I worked with the solutioning engineer to define the outcomes the agent had to reach: solutioning required, no solutioning required, or flagged items to review. The design choice was admitting that not every store fits those buckets. Anything the agent can't resolve routes to the solutioning engineer for a detailed follow-up, so automation handles the clear cases and expert time goes where it's actually needed.

• Encoded the solutioning engineer's evaluation as a cloud-native managed AI agent driving a remote headless browser, so it runs anywhere with no local setup.
• Walks a store's full purchase funnel, detects the third-party apps and architecture in play, and classifies integration complexity as required, not required, known-solutions, or inconclusive.
• Returns machine-readable JSON with severity-tagged callouts and recommended actions, so it feeds downstream systems instead of living in one person's head.
• Built the surrounding control plane: deterministic deploys, structured test output, automatic secret redaction, and 39 unit tests covering the harness.
• Solved the parts that aren't in any documentation: container networking constraints and binary-output retrieval.

43 deployed versions, each with a recorded hypothesis, the change, the test result, and a verdict. Three learnings generalized: tell the model what to do, not how; promote critical signals to the top of the prompt so they are never silently dropped; and structurally aligning the agent with its source reference reduced fragility more than any single prompt tweak.

~7 min

per evaluation, versus about 25 min of manual work

~$0.62

per run

43

deployed versions with recorded hypotheses

39

unit tests covering the harness