Rendered at 22:49:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nucleus777 1 days ago [-]
Interesting. Most of the failures I've seen aren't the agent doing something obviously daft, it's a tool call that returns fine but the result shouldn't be trusted. Stale cached data, or a web page or file with text in it that's basically trying to give the agent instructions. Everything after that looks reasonable, it's just built on a bad input.
Can flow-1 flag that kind of thing? And how does it tell the difference between the tool giving back something wrong and the agent misreading a result that was actually fine? From the trace alone those look pretty similar to me.
skull8888888 1 days ago [-]
this is exactly what flow-1 in Signal harness catches. highly highly recommend checking this out https://laminar.sh/blog/flow-1-in-action. We got examples of traces and errors we caught in them there
scottfalconer 1 days ago [-]
We ran into an interesting side effect of agent performance that we needed to explain in our results to our executive team. In our case we were rolling out an AI support agent that could help resolve customer problems. At the start we had calculated the total time and human cost per support ticket going back a few years. After a few months of running the AI agent, we found that both our time and cost per ticket had increased - but then digging in, the issue was that our overall cost and time had decreased and it was just that the more complicated issues the agent wasn't able to answer were the ones now going to the humans, skewing our per ticket cost up.
So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.
skull8888888 24 hours ago [-]
hm, did the cost of tickets processed by an agent go down tho?
scottfalconer 24 hours ago [-]
Yes, significantly - even when adding in vendor costs etc. In general though it is best at answering routine questions or ones a customer could get by reading the docs. The net result then is the ones left for the humans are the harder ones.
We’ve also heard from other companies that as the first line paper cut questions get answered, customer adoption increases - which further increases the “hard” support questions.
Lesson learned for me on that one that was rolling out a support agent led to both a better experience for our customers and an increased need for talented human engineers.
khalidsaidi 3 hours ago [-]
we publish every agent transcript and most of our partial results (35 of 192) werent the model being dumb, they were the site returning a 403 or an empty js page and the agent working around it. a tool that flags those as errors would be mostly noise for us. does flow-1 separate environment failures from actual reasoning failures?
mattbaconz 7 hours ago [-]
The failure I keep hitting with small local models isn’t a messy trace, it’s a confident “done” on a change that fails the tests. On a slice of 51 Aider Polyglot exercises with qwen3.5:4b, the bare model reported success 43 times and 37 of those failed.
Does Laminar catch that case, where the trace looks fine but the outcome is wrong, or does it need a test result to compare against?
dmos62 1 days ago [-]
How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
dinm 1 days ago [-]
Good questions.
Agent observability is based on similar primitives as regular observability (OTEL), but is wildly different. Trace payloads are huge, and errors are much more subtle, they are not increased latency or elevated 500 response rate. We recommend tracing everything in detail, and reading a few traces manually at the beginning while you iterate. At scale, the answer is almost always use another LLM look at your traces, which is what this post is all about.
Evals are critical. I really recommend adopting the Anthropic's capability / regression eval model [1]. We also deeply believe that agent evals should be end-to-end, and as high-level as possible. This again often involves a quality LLM-as-a-judge, tuned on a set of manually labelled data.
With this idea in mind (to answer your question), we think that, for eval datasets, the most crucial bit is to be able to store input of what your agent was asked about in production. Collect a certain distribution of inputs for happy and unhappy cases, and then run evals on those inputs. That's why Laminar went great lengths to help our users automatically get actual inputs their users send to their agents [2].
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making.
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
skull8888888 1 days ago [-]
>for creating traces from many test calls to speed up testing of voice bots
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
paulariquelmee 1 days ago [-]
The point about paying to store traces nobody reads hit home. We had the same problem with customer interviews. We did 8 of them and the notes just sat in a doc until we tagged each problem by how many customers it affected. Curious how you decide which flagged errors matter most. Is it how often they show up across traces, or do you weight them by user impact?
skull8888888 1 days ago [-]
>Curious how you decide which flagged errors matter most
flow-1 actually predicts how critical each finding is
sebastienburel 17 hours ago [-]
To your question about how we track ours: one failure I had left no trace at all. A morning run on a resident agent just didn’t happen. The host timer runs on the uptime clock, the Mac slept through the slot, and nothing retries. As far as I can tell from the post, Signals reads traces that exist. Do you do anything for the run that should have happened and didn’t, like diffing against an expected schedule?
SantiDev 1 days ago [-]
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
kolbeyang 1 days ago [-]
Thanks!
Our design is all done completely in-house.
Workflows are constantly changing but our design process happens across Figma, old-fashioned paper, and code.
pmichalas 13 hours ago [-]
Nice. One thing I'd want from a checker like this is a calibrated probability rather than a severity label. Decision models in the Jev style give a probability for every option in one forward pass. You can then pick a threshold and know roughly what share of traces you'll send for review. Does flow-1 expose anything like that, or is severity the only knob?
danielmarkbruce 1 days ago [-]
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
dinm 1 days ago [-]
Thanks for the response! Curious, can you expand on the architecture, how does that work? If you can share of course.
danielmarkbruce 1 days ago [-]
There are all manner of documented architectures in papers and books, I don't know that there is anything new under the sun at this point, the devil is always in the details. That said, some of the things we use off the top of my head: generate multiple responses using different models and have a third model judge the best answer; generate an answer and have another model critique the answer and then a third take the answer and critique and generate a new answer; multi-model debate; iterate in a loop either x times or until some condition is satisfied; pipelines of specific single calls to check for specific conditions.
I'd say in general, with respect to errors/low quality responses etc you have to be extremely clear about what you are doing because once you get into the weeds, what is and is not an error is very grey in many cases.
dinm 1 days ago [-]
Interesting. Seems to me like latency is not critical in your application and you are doing something more like background processing rather than serving humans on the other side in real time.
Also, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
danielmarkbruce 1 days ago [-]
Correct, latency is not critical.
Yes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.
ReplyIntelligen 8 hours ago [-]
Does a trace shoe that the thing it inspected is the thing that shipped?
BoorishBears 1 days ago [-]
You should try the bullet points then earning someone reading the rest, that's where I bounced.
skull8888888 1 days ago [-]
I can write some bullet points if you want. I'm serious, this is not a sarcasm. With which AO platform do you want a comparison?
BoorishBears 1 days ago [-]
Maybe this is me being the wrong persona since I clicked for the model and not the platform, but literally the moment my eye caught
> Agent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles
my brain shut down and bounced minus the comment because it's a launch and maybe you'd want feedback.
I'd be using this for post-training in a custom harness: so what you focused on during post-training and what it excels at are the most interesting things
skull8888888 1 days ago [-]
this is a very valid point. I will expand on RL post train here.
I used GSPO because it worked really well for very long prompts (we're dealing with traces analyses so prompts are very very long, and by prompts I mean what goes into LLM call).
We RL-ed flow-1 in our harness, which made it very efficient at using tools it had. Our harness as I said in the post is basically coding agent, it has grep, read and couple of other aux tools. I spent a loooot of time and money on figuring out the right RL recipe and especially the reward. It was really hard to find the right balance between exploration and exploitation.
Some RL runs are still running, because the prod checkpoint as good as it is, thinks a lot. It produces a lot of thinking tokens, which affects the throughput, especially on the large, complex traces to be analyzed. So the next big challenge for me is to make it efficient thinker - i.e. reduce thinking tokens, without losing quality.
Can flow-1 flag that kind of thing? And how does it tell the difference between the tool giving back something wrong and the agent misreading a result that was actually fine? From the trace alone those look pretty similar to me.
So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.
We’ve also heard from other companies that as the first line paper cut questions get answered, customer adoption increases - which further increases the “hard” support questions.
Lesson learned for me on that one that was rolling out a support agent led to both a better experience for our customers and an increased need for talented human engineers.
Does Laminar catch that case, where the trace looks fine but the outcome is wrong, or does it need a test result to compare against?
Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
Agent observability is based on similar primitives as regular observability (OTEL), but is wildly different. Trace payloads are huge, and errors are much more subtle, they are not increased latency or elevated 500 response rate. We recommend tracing everything in detail, and reading a few traces manually at the beginning while you iterate. At scale, the answer is almost always use another LLM look at your traces, which is what this post is all about.
Evals are critical. I really recommend adopting the Anthropic's capability / regression eval model [1]. We also deeply believe that agent evals should be end-to-end, and as high-level as possible. This again often involves a quality LLM-as-a-judge, tuned on a set of manually labelled data.
With this idea in mind (to answer your question), we think that, for eval datasets, the most crucial bit is to be able to store input of what your agent was asked about in production. Collect a certain distribution of inputs for happy and unhappy cases, and then run evals on those inputs. That's why Laminar went great lengths to help our users automatically get actual inputs their users send to their agents [2].
[1] https://www.anthropic.com/engineering/demystifying-evals-for...
[2] https://laminar.sh/blog/extracting-tasks-from-agent-runs
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
flow-1 actually predicts how critical each finding is
Workflows are constantly changing but our design process happens across Figma, old-fashioned paper, and code.
I'd say in general, with respect to errors/low quality responses etc you have to be extremely clear about what you are doing because once you get into the weeds, what is and is not an error is very grey in many cases.
Also, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
Yes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.
> Agent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles
my brain shut down and bounced minus the comment because it's a launch and maybe you'd want feedback.
I'd be using this for post-training in a custom harness: so what you focused on during post-training and what it excels at are the most interesting things
I used GSPO because it worked really well for very long prompts (we're dealing with traces analyses so prompts are very very long, and by prompts I mean what goes into LLM call).
We RL-ed flow-1 in our harness, which made it very efficient at using tools it had. Our harness as I said in the post is basically coding agent, it has grep, read and couple of other aux tools. I spent a loooot of time and money on figuring out the right RL recipe and especially the reward. It was really hard to find the right balance between exploration and exploitation.
Some RL runs are still running, because the prod checkpoint as good as it is, thinks a lot. It produces a lot of thinking tokens, which affects the throughput, especially on the large, complex traces to be analyzed. So the next big challenge for me is to make it efficient thinker - i.e. reduce thinking tokens, without losing quality.