I built an AI-powered fantasy football product, discovered it was repeatedly paying an LLM to reconsider facts that had not changed, and eventually arrived at the slightly awkward conclusion that one of the best ways to improve an AI system was to use considerably less AI.

Originally published on Medium.
This is an expanded version of an article originally published on my Medium account. Read the original on Medium →
I have been building a fantasy football application as a personal project, which is the sort of sentence that sounds perfectly harmless until you have been a developer for long enough to know that “personal project” usually means “an unpaid second job involving a database, several scheduled processes, an increasingly elaborate deployment pipeline and at least one dashboard that absolutely nobody asked for.”
Like an entirely normal middle-aged man, rather than simply watching the Chicago Bears disappoint me on a Sunday and getting on with my life, I decided what I really needed was a software platform capable of analysing that disappointment in considerably more detail, preferably with enough data, charts and automated recommendations to make the disappointment feel scientifically rigorous.
The application pulls together player data, fantasy rosters, transactions, injuries, news, values, schedules and other information from several sources, stores and processes that data locally, and then uses an LLM to reason about what has changed and suggest things I might actually want to do about it.
Trade this player.
Pick up that player.
Keep an eye on somebody else.
Perhaps stop drafting Bears players merely because I like the Bears.
The last requirement remains stubbornly resistant to automation.
The first version worked surprisingly well.
The problem was that it worked in roughly the same way as employing an extremely expensive consultant and phoning them every five minutes to ask whether anything had happened, despite the fact that you were standing in the same room and could plainly see that nobody had moved.
Most of the time the answer was:
No.
And I was paying for the privilege of finding out.
That turned out to be a useful lesson, because it forced me to stop thinking about the LLM as the application and start treating it as what it really was: one comparatively expensive, comparatively slow and deliberately non-deterministic component inside a much larger software system.
The first architecture was obvious
The original architecture followed the pattern that I suspect quite a lot of AI prototypes follow:
Get data → build prompt → call model → display answer.
There is nothing particularly wrong with that.
In fact, I would still argue that it is often exactly the right way to build the first version of something, because the purpose of a prototype is to determine whether the idea produces anything useful before you spend three weeks building an abstraction layer, an event bus and seventeen interfaces around something users ultimately do not want.
The problem comes when the prototype works.
Because working software has an unfortunate tendency to become production software.
That is how systems acquire a “temporary” database that survives for five years, a “quick script” that becomes business-critical and a function called processEverything() which eventually requires its own incident response procedure.
My fantasy application runs automatically, which means it periodically collects the latest information about the same players, rosters and leagues.
But American football has the inconvenient characteristic of not completely reinventing itself every few hours.
At 6am, a player’s situation might be exactly the same as it was the previous evening.
The roster probably has not changed.
There may be no new transaction.
There may be no injury.
There may be no meaningful news.
The waiver wire may be identical.
There may not, in fact, be a single new piece of information capable of changing the previous recommendation.
Yet the application could happily collect everything again, construct another large prompt and ask an LLM to reconsider the entire universe.
And because an LLM is not a deterministic function, it would usually oblige.
Perhaps yesterday I should “strongly consider” trading a player.
Today I might be advised to “seriously explore” trading him.
Tomorrow I might be told that moving him would be “a prudent roster-management strategy.”
Three different sentences.
One recommendation.
Several API calls.
At some point I realised I had essentially built a thesaurus with a billing account.
That bothered me, and not simply because of the token cost.
Architecturally, it was stupid.
An LLM call is not a normal function call
This distinction became increasingly important as I worked on the system.
Developers are accustomed to calling functions constantly, and we do not normally stop before invoking:
calculatePlayerValue()
and have a small architectural meeting about whether we can afford it.
Assuming the function is reasonably implemented, it is cheap, fast and deterministic enough that calling it twice is unlikely to trouble either the CFO or the planet.
An LLM call has a rather different set of characteristics.
There is an obvious financial cost, because somebody is charging for input tokens, output tokens, cached tokens, model tier or some combination thereof.
There is latency, because the request leaves your process and goes to an external inference service.
There are rate limits, which become rather more interesting when an application scales beyond the three people you originally expected to use it.
There is an additional availability dependency, because your product is now relying on another provider remaining responsive.
There is model drift, because the service behind your API endpoint can change.
And then there is the particularly interesting characteristic:
non-determinism.
Call a conventional function twice with identical inputs and, unless something has gone badly wrong, you expect identical outputs.
Ask an LLM the same question twice and you can receive two different, perfectly defensible answers.
That variability is part of what makes these models useful; I am not criticising it.
But if the underlying state has not changed, repeatedly invoking a probabilistic reasoning system can introduce variation into a product where no variation is actually warranted.
Your recommendation changes.
The user assumes something happened.
Nothing happened.
The model just fancied a different adjective.
That led me to what became the most important architectural question in the project.
Instead of asking:
“What should the AI think about this?”
I started asking:
“Has anything happened that requires the AI to think at all?”
That sounds like a small change.
It was not.
The agent became a change-detection system
Once I started thinking about the problem that way, the architecture shifted.
The LLM moved towards the end of the pipeline rather than sitting in the middle of everything.
The flow became:
Collect → Normalise → Compare → Decide → Analyse
rather than:
Collect → Analyse everything
That Decide stage is probably the most important component in the entire application.
The collection layer retrieves information from the various upstream sources.
The normalisation layer converts those sources into a consistent internal representation.
The state layer records the information the system currently believes to be true.
The comparison layer establishes what has changed since the previous successful analysis.
The decision layer determines whether any of those changes are capable of affecting an existing recommendation.
Only then does the application construct a prompt and invoke the LLM.
The clever bit, therefore, is increasingly not the model call.
It is everything that prevents the model call.
And this is not particularly exotic AI technology.
It is persistence.
Canonicalisation.
Hashing.
Comparison.
Filtering.
Event detection.
Cache invalidation.
Dependency tracking.
The sort of engineering that does not usually feature heavily in AI product demonstrations because “look at this beautifully designed state fingerprint” does not produce quite the same conference reaction as a chatbot generating a limerick about the CEO.
It is nevertheless where a significant amount of the value lives.
Because:
The cheapest LLM call is the one you never make.
A concrete example: league analysis
Take the league analysis itself.
The original implementation could analyse a league every time the scheduled process ran.
Imagine that at 6am the system sees this simplified state:
league_id: 12345
roster_version: 84
player_news_version: 317
injury_version: 102
transaction_version: 46
It generates its recommendations and stores them.
At midday the collector runs again.
There are newer API retrieval timestamps, because naturally there are, but after normalisation the meaningful state is still:
league_id: 12345
roster_version: 84
player_news_version: 317
injury_version: 102
transaction_version: 46
What exactly am I hoping the LLM will discover?
The answer is nothing.
So now it does nothing.
The existing analysis remains valid.
If, however, injury_version becomes 103 because a starting running back has been ruled out, that is potentially significant.
The system can invalidate the affected analysis, identify which leagues contain that player, construct the appropriate context and ask the model to reconsider the relevant recommendations.
This changes the semantics of the model invocation.
The question is no longer:
“Please analyse my fantasy team.”
It is closer to:
“This was the previous state and recommendation. This meaningful event has occurred. Given the new state, does the recommendation need to change?”
That is a much better question.
It is also a much smaller one.
State fingerprints became surprisingly useful
There are several ways to implement the change detection itself.
One straightforward technique is to build a canonical representation of the relevant state.
Take the fields that actually matter.
Sort collections consistently.
Remove volatile metadata.
Serialise the resulting structure deterministically.
Hash it.
You now have a state fingerprint.
Conceptually:
state_fingerprint =
hash(
roster +
injuries +
relevant_news +
transactions +
player_values
)
If the fingerprint has not changed since the analysis was generated, there is no obvious reason to repeat the analysis.
Of course, production systems become more nuanced than that.
Different changes invalidate different pieces of analysis.
A roster transaction might affect the whole team.
An injury might affect one player and their likely replacements.
A new article might affect nothing at all.
A change in trade value might matter only when it crosses some threshold.
That means a single giant hash eventually becomes less useful than explicit dependency tracking, but the underlying principle remains the same:
Store enough information about the state that produced an answer to determine whether that answer is still valid.
That changed the way I thought about caching.
The cache became part of the intelligence
Initially I thought about caching in the conventional sense.
Make an expensive request.
Store the response.
Give it a TTL.
Reuse it until the TTL expires.
That works, but it is a fairly crude approach to this particular problem because the passage of time is not necessarily what makes an analysis stale.
Changes in the underlying facts make it stale.
Suppose I analyse my roster at 10am.
If absolutely nothing relevant happens for the next 24 hours, why should the recommendation automatically expire at 11am merely because I chose a one-hour cache duration?
Conversely, if my starting quarterback is unexpectedly ruled out at 10:03am, I do not particularly want to wait until 11am because Redis says the previous answer still has 57 minutes left to live.
The better cache key is therefore not simply:
time
but:
state.
The important artefact isn’t:
“Here is what the model said last time.”
It is:
“Here is what the world looked like when the model said it.”
That makes caching part of the reasoning architecture rather than simply a performance optimisation.
New data is not the same thing as new information
This became another surprisingly important distinction.
APIs produce new data all the time.
Timestamps change.
Records get refreshed.
News feeds return the same stories again.
Providers correct formatting.
A player’s display name changes.
An article receives an updated timestamp because somebody fixed a typo.
Technically, something changed.
Semantically, nothing happened.
If every raw data mutation invalidates an analysis, the application will repeatedly wake the model up to tell it that the universe remains essentially identical.
That is not intelligence.
That is a very expensive heartbeat monitor.
The system therefore needs to distinguish between a raw delta and a semantic delta.
A player going from:
healthy → questionable
is potentially meaningful.
A player going from:
Luke McCaffrey → Luke McCaffrey
because somebody added whitespace is not.
A new article saying that a player attended a charity event is new data.
A new article saying that he left training wearing a walking boot is new information.
The interesting engineering problem is therefore not simply change detection.
It is meaningful change detection.
Don’t hire an LLM to decide whether to hire an LLM
There is an obvious temptation here.
We have a model capable of interpreting language.
Why not give every new item to the LLM and ask whether it is important?
Sometimes that is absolutely appropriate.
But if you do it indiscriminately, you have created a rather beautiful circular architecture in which you call the model to determine whether you should call the model.
You have, in effect, hired a consultant to decide whether you need to hire a consultant.
So I push as much of that classification as possible into deterministic code.
Known injury-state transitions can be rules.
Roster transactions are structured events.
Value changes can use thresholds.
Duplicate news can be identified through IDs, canonical URLs, hashes or similarity mechanisms.
Known irrelevant metadata can be excluded before comparison.
Only ambiguous cases need probabilistic interpretation.
That is an important general principle I have taken away from the project:
Use deterministic computation to reduce the problem before asking probabilistic computation to reason about what remains.
Then I made the prompts smaller
Once I had reduced the number of calls, the next obvious question was whether the calls I was still making contained far too much information.
The answer was yes.
Early LLM applications have a tendency to behave like nervous junior developers preparing their first production handover:
“Here is absolutely everything I know, just in case.”
Full player objects.
Complete roster dumps.
Repeated news articles.
Historical information.
Provider metadata.
Previous analysis.
Fields nobody remembers adding.
Fields I was too frightened to remove because perhaps the model had secretly become emotionally attached to them.
All of this consumes tokens, but token cost is only part of the problem.
More context does not automatically mean better context.
Irrelevant information competes with relevant information.
Duplicated information can distort importance.
Old information can conflict with new information.
Large contexts make failures harder to understand.
So prompt construction became a deliberate pipeline of its own.
Select the relevant entities.
Resolve identifiers.
Filter stale information.
Deduplicate news.
Include only the state necessary for the current decision.
Include the previous recommendation when revision is required.
Represent uncertainty explicitly.
Ask a specific question.
Request a constrained output.
In other words, the model stopped receiving the database and started receiving a purpose-built data product.
Prompt engineering became data engineering
This was probably one of the more interesting conclusions.
People talk a lot about prompt engineering, and there is obviously skill involved in specifying tasks clearly, but in a real application the quality of the prompt is heavily determined before anybody writes the instruction.
The difficult questions are things like:
Which player does this record refer to?
How old is this information?
Which source produced it?
Has it already been processed?
Does it supersede another fact?
Is this the current roster state?
Was the previous recommendation generated before or after this event?
Those are data engineering questions.
A prompt is ultimately the presentation layer over those decisions.
Stable identifiers matter.
Timestamps matter.
Provenance matters.
Schemas matter.
Freshness matters.
Ordering matters.
The prompt is not your database.
It is not your event log.
It is not your cache.
It is not your state store.
And despite some heroic attempts I have seen, it is definitely not a substitute for having a schema.
The prompt is an interface between structured application state and a probabilistic reasoning system.
Once I started treating it that way, the entire design became easier to reason about.
Something unexpected happened
I originally made these changes to reduce token consumption.
That worked.
But something more interesting happened.
The application got better.
Responses became more focused because the model received less irrelevant information.
Repeated runs became more consistent because unchanged state did not trigger fresh inference.
Latency improved because there were fewer model calls.
Debugging improved because every model invocation had an identifiable reason.
The deterministic components became responsible for things deterministic software is very good at:
comparison, filtering, deduplication, state management, validation and rules.
The model became responsible for something LLMs are actually rather good at:
reasoning about ambiguous information when something meaningful has happened.
That separation seems obvious now.
It was not obvious when I started.
Debugging AI stopped being quite so mystical
This architecture also improved observability enormously.
When every scheduled run sends a giant prompt to a model and receives another block of prose, answering the question “why did the recommendation change?” can become surprisingly difficult.
Was the input different?
Was there a new news article?
Was the article duplicated?
Did the prompt template change?
Did a provider change some metadata?
Did the model simply interpret identical evidence differently?
Did I accidentally include a player from another league because two people have similar names?
Did Mercury enter retrograde?
With explicit state tracking, the problem becomes much more conventional.
I can inspect the previous snapshot.
I can inspect the new snapshot.
I can see the delta.
I can see which event invalidated the previous analysis.
I can inspect the exact model input.
I can compare the old and new output.
I can record model version, prompt version, token usage, timestamps and relevant state identifiers alongside the result.
“The AI did something weird” stops being a diagnosis and becomes what it should always have been:
the beginning of the investigation.
The architecture became event-driven
The first implementation was fundamentally polling-driven.
A timer fired.
The application collected everything.
The model analysed everything.
The improved version still polls some sources because the real world stubbornly refuses to provide a beautifully designed event stream for every API I would like to consume.
But polling is now a mechanism for discovering events, not a reason to run analysis.
That gives us a much more useful conceptual pipeline:
Poll → Detect → Classify → Invalidate → Schedule → Analyse → Persist
A new injury report does not necessarily require every league to be reanalysed.
It affects leagues containing that player.
Perhaps it affects only decisions involving that player and their replacements.
A roster transaction invalidates the appropriate roster analysis.
A trade-value movement may matter only if it crosses a decision threshold.
This starts looking suspiciously like traditional distributed systems engineering.
Idempotency matters.
Event keys matter.
Deduplication matters.
Retries matter.
Backoff matters.
Observability matters.
Failure handling matters.
The application needs to know whether an event has already been processed.
It needs to avoid running an expensive analysis twice because a worker timed out after the model responded but before the result was committed.
It needs to distinguish retryable provider failures from permanent failures.
It needs enough metadata to reconstruct what happened afterwards.
None of those requirements disappeared because I put an LLM into the architecture.
If anything, they became more important, because the operation in the middle is now slower, more expensive and less deterministic than most of the functions surrounding it.
“Agentic” does not mean “ask an LLM everything”
This is probably the biggest lesson I have taken from the whole project.
There is a temptation at the moment to equate “agentic” with putting an LLM inside a loop.
Get some information.
Call model.
Get more information.
Call model.
Call another model.
Ask the first model what it thinks about what the second model thought.
Perhaps introduce a third model to adjudicate.
Eventually somebody receives an invoice large enough to qualify as an architectural artefact.
Useful agency does not require constant inference.
A good agent should know when not to think.
It should know when to wait.
It should know when nothing has changed.
It should know when a deterministic rule is more reliable than generated reasoning.
It should know when its previous answer remains valid.
It should know when it does not have enough information.
It should know which state has been invalidated and which has not.
Nobody is going to produce an exciting conference demo in which an AI agent examines a state fingerprint and proudly announces:
“I have decided to do absolutely nothing.”
But in production systems, doing nothing can be the correct behaviour.
And correct behaviour is generally more useful than an impressive animation.
The model is not the architecture
One of the traps in AI development is becoming so obsessed with model selection that everything around the model becomes secondary.
Which model?
Which context size?
Which benchmark score?
Which reasoning mode?
Which provider?
Those questions matter.
But a better model does not rescue an application that sends stale information, duplicates records, loses state, cannot explain why an answer changed and repeatedly asks expensive probabilistic systems to perform work that could have been handled by a database query.
The surrounding software determines:
- which facts reach the model;
- whether those facts are current;
- whether they are duplicated;
- whether they are relevant;
- what previous state is available;
- what changed;
- why analysis was triggered;
- whether the output can be explained later;
- whether failures can be retried safely;
- and whether the economics still work when usage increases by a factor of 100.
This is why AI engineering increasingly looks to me like a mixture of software engineering, data engineering and probabilistic systems engineering rather than an entirely new discipline in which we throw away everything we learned over the previous fifty years.
You still need schemas.
You still need queues.
You still need caches.
You still need indexes.
You still need logs.
You still need metrics.
You still need regression tests.
You still need to understand failure modes.
And occasionally you need to remember that a SQL query has been sitting quietly in the corner for decades, waiting patiently for everyone to finish talking about agents.
The irony of building AI software
The more I have worked on this project, the less impressed I have become by applications that demonstrate their sophistication by making lots of AI calls.
Calling an LLM is easy.
Knowing when to call one is architecture.
Knowing when not to call one is architecture.
Knowing what context to give it is architecture.
Knowing what state to retain between calls is architecture.
Knowing what invalidates an existing answer is architecture.
Knowing which decisions should remain deterministic is architecture.
Knowing why an answer changed is architecture.
Knowing how the application behaves when the model is unavailable, slow or confidently wrong is architecture.
The model matters enormously, but it is still a component.
The software surrounding it determines whether you have built a useful product or an expensive demonstration with a very persuasive loading spinner.
My fantasy football application still uses AI.
It just uses considerably less of it.
And the result is cheaper, faster, more deterministic, more observable and easier to debug.
Most importantly, the model is now being used for judgement rather than bookkeeping.
That is probably the most important lesson I have taken from building it.
The objective should not be to find as many opportunities as possible to invoke an LLM.
The objective is to identify the relatively small number of places where probabilistic reasoning genuinely improves the system, engineer everything around those points properly, and then have the discipline to leave the model alone when it has nothing useful to do.
Which, somewhat inconveniently for anyone measuring AI adoption by token consumption, means that using less AI has made this a considerably better AI product.
As for whether it can finally explain what the Chicago Bears are doing?
Some problems may simply be beyond artificial intelligence.
Although, to be fair, I have not yet tried giving it a larger context window.
This is an expanded version of an article originally published on Medium.
If you enjoyed this, you can read the original article and follow my other writing on AI, software engineering, agentic systems and the occasionally questionable things I build with them on Medium.






























