Faro Build Log: Agentic Retrieval Harness

When a trader asks what's going with bitcoin, they probably don't want to know what the current block number is.

But how do you make sure that an AI agent answers their question in a way that is most relevant to user intent?

Faro's Agentic Retrieval Harness is the machinery that ensures Faro will always give the best and most relevant possible answer.

How most AI tools answer a question

Imagine you ask an AI what Bitcoin is doing today. What you get back is what the AI thinks is going on with Bitcoin today, and LLMs have become good enough that this is usually pretty good. But it can miss pieces, and carry things you never asked for, like the block number.

You probably don't want to know everything that is going on. You only want to know what is important today, for you and your situation. 

On the other hand, if you asked a Bitcoin analyst the same question, they would generate a more useful response by having a frame of reference already in mind—a checklist that tells them what to look at before they look at anything.

This checklist is what we call the Agentic Retrieval Harness.

Skills are a losing battle

The first challenge of building the harness is deciding what data reaches you when you ask a question.

For example, with equities, every asset carries roughly the same types of data. But what matters is different for each one. So how do you configure an AI to know how to navigate every single company?

One method is to write an individual skill for each company. Maybe you have an Nvidia skill, you have a Broadcom skill, and an Apple skill, and then these skills determine what is important to the company. And that's sort of what we were doing at the beginning, but we found this to be a very fragile way to resolve the problem.

Yes, you can create a skill to make sure the LLM itself has some company context embedded into it, but that's only going to be up-to-date as of the last time that the agent was trained.

Take Ethereum as an example — our favorite asset that never tends to go up. (Hopefully it will!) Ethereum's bull thesis in the past was about ultrasound money: burns, going deflationary, yada yada. LLMs are now trained on all of that, but none of it really matters anymore.

So if you rely on the LLM to work out what's relevant, you are fighting a losing battle because every dead thesis is still in the training data.

Compare this to a recipe: the LLM could be a very skilled chef, but It's still impossible to make something good with the wrong ingredients. If ultrasound money goes in, ultrasound money comes out.

So we stopped trying to make the chef smarter, and started controlling the ingredients instead.

The retrieval harness

What the harness does is take the question and break it down into components—what assets does the user care about, what is the intention of the question, and what should be pulled out of the database to best answer this question?

The model (chef) still writes the answer. It just no longer chooses the inputs (ingredients).

Instead, this is handled by the harness, which acts like a recipe that sits above the LLM. It semantically breaks down the question, identifies the right ingredients, and then passes the metrics back to the agent to have it cook it up into an answer.

The key is that instead of relying on the LLM to know what matters, we can pick the ingredients that go into the recipes—selecting the metrics that we know to be important. For example, with a query about Bitcoin trading it might be what stage of the market cycle we are in.

Ultimately we see this as becoming like PageRank for financial data: we rank the metrics based on the question, and increase the quality of the answer by increasing the quality of the ingredients that go into it.

The four questions we ask of every metric

You can picture the harness as a series of four filters:

1. ****Is the data fresh?

Every metric carries its own rate of decay. Funding is an in-the-moment thing, so an eight-hour-old reading is close to worthless. ETF flows settle daily, so an eight-hour-old reading still counts. 

1. Is it relevant to what you asked?

Every metric gets scored against the question itself to make sure the answer corresponds. Ask about the positioning of BTC perpetuals on Hyperliquid and you want liquidation levels, funding, and what the big positions have been doing. ETF flows aren't wrong, but they're not the question, so they would sink down the ranking.

1. Is it unusual right now?

Unusual is usually relevant, because things get unusual for a reason. We score every metric on abnormality and rerank on it. For example, funding at 10.2% doesn't mean much because it isn't a meaningful standard deviation from the median. Funding at 1000%, however, goes straight to the top.

1. Does this user care about it?

The last ranking factor belongs to the user. A user who keeps telling us they don't care about funding sees it drop, even when it is unusual and relevant. It also runs the other way, so someone who wants Bitcoin's block number can say so, and the harness will then carry that preference into the next answer.

This compounds over time, so as you continue to interact with Faro, it will learn what you like, what type of investor you are, and then tailor the way that you receive responses so that it actually makes sense for you.

The same principle works inside a single chat: Ask what's going on with BTC, then ask about ETH, and the metrics from the first question feed into the second. Not always, though: Ask about BTC tax in Australia and the harness splits the session, because that question doesn't share much with the previous one.

Why the extra work is worth it

The whole harness can be thought of as a chain of transformation layers. The questiin goes in one end, and changing what comes out the other side means adding or removing a layer in the middle.

That means when an answer is wrong, we can find which layer put it there, and pull it out to tweak it. Instead of swatting at the air like we would if an LLM handled the whole process. 

Building this way will make refining the answers much easier, but it does mean a full rebuild of the database, both to account for the harness and for things like backtesting.

This is our current focus and is what's taking up most of our development time at the moment.

Follow along to be updated when the harness is released, and keep track of Faro's progress.