When Excel Stops Listening

Excel enforces an RTD contract it never wrote down — and when a server breaks it, the penalty is silence. The host's side of the bridge, at production scale.

The side I thought was solid

The last article (Keeping the Dashboard Current) was about the far side of the bridge: the TWS connection that drops for a dull catalog of reasons and has to be rebuilt without the user touching the workbook. This one is about the side I expected to be easy — Excel itself. The host. The half with a documented Microsoft interface and decades of corporate software behind it. I assumed that once data reached my server from TWS, the last few inches into the Excel grid were the solved part of the problem.

Excel spent months teaching me otherwise. The host keeps rules it never wrote down, and when you break one, it doesn't throw an exception. It goes quiet.

The bargain Excel actually offers

To see how, you need the shape of the protocol in your head. RTD is polite, but the politeness goes in one direction. The RTD server never pushes a value into a cell. When new data arrives, it is allowed exactly one move: call UpdateNotify — a knock on the door, carrying no payload. Excel decides when to pick up the new data. When it does, it calls RefreshData, and the RTD server hands over everything it wanted to deliver: topic IDs, values, and a count of how many it delivered. All of it through COM, on Excel's one main thread, in a single-threaded apartment.

On the other side sits the TWS API, which pushes continuously and waits for no one. My RTD server buffers between a party that never stops talking (TWS) and one that only listens when it feels like it (Excel). This article is about the second.

Excel answers when it chooses

The first host-side behavior isn't hidden — it's documented, but easy to underestimate. Interactive Brokers' own Excel RTD page states it plainly: "by design, Microsoft Excel gives precedence to the UI. Updates are ignored when a modal dialog is displayed, a cell is being edited, or Excel is busy." If a user is editing a formula or format, Excel waits until they're done, no matter how long that takes.

The TWS data firehose waits for nothing. A naive RTD server that treats each notify-and-pull cycle as its unit of delivery loses whatever happened while the workbook was blocked. So the server's own cache — not the notify pipeline — has to be the source of truth: every value lives in the engine, updated continuously off the feed, whether or not Excel is taking calls. When Excel finally requests the data refresh, it only wants the current value, not a replay of everything it missed.

For the trader: a dashboard that pauses while you type is the host's design, not a defect. The defect would be showing you a pre-edit number as if it were live afterward.

Never call back from inside a call

The second behavior is the classic COM trap, and it's the reason the notify path is a component rather than a line of code. The temptation is natural: a tick arrives from TWS, so call UpdateNotify right there, from whatever thread, at whatever moment. But some of those moments are inside one of Excel's own calls to you. Respond to Excel synchronously from inside its own callback and you can put it into a state where it silently stops listening to your server for the rest of the workbook session. No error. No exception. Just a workbook that never updates again.

So the engine never notifies inline. The notify path is its own small state machine (idle → pending → notified) that schedules the actual COM call asynchronously, collapses concurrent requests into a single knock, fires at most one notify per pull cycle, and runs a watchdog that re-arms the knock if Excel doesn't come to the door within a configured window. That is what knocking politely costs.

The day Excel stopped calling back

Now the behavior that earned this article its title. One production day, the workbook simply stopped. Every RTD cell frozen. No error dialog, no crash, nothing in Excel's demeanor to suggest anything was wrong. The server was running. TWS was connected. Data was flowing into the RTD server, which was dutifully updating its cache. Excel had stopped calling RefreshData — entirely, permanently. The only thing that recovered it was restarting Excel.

A failure with no exception gets the treatment every undocumented behavior gets here: reproduce, characterize, document, then fix. The characterization (my write-up carries three sequential "CRITICAL UPDATE" headers, each a same-day discovery-fix-validation note) comes down to this: if RefreshData delivers fewer topics than were pending when the server knocked — a count fixed on the server's side at notify time, since the knock itself carries no payload — Excel enters what my notes call a broken trust state, and stops calling RefreshData for the rest of the session. From the write-up:

pending at the knock:   155 topics
RefreshData() returns:  154 topics  ← MISMATCH
Result: Excel broken until restart

One topic short, once, and the workbook is done until restart.

This was a discovery, not officially documented behavior. The countermeasure lives in my RefreshData implementation and is guarded by unit tests. I'm not calling this a bug in Excel; for all I know it's a deliberate defense against misbehaving RTD servers. What I can say is that the behavior is real in my environment, the trigger is a count mismatch, and the consequence is permanent for the session.

How does a careful server come to under-deliver by one? A race, naturally. The count Excel is promised gets fixed at notify time; the set of pending topics keeps living after that. In my case the TWS connection dropped in the middle of a refresh cycle, the disconnect handler swept through the pending queue flipping connection-dependent topics to #N/A, and the number promised drifted apart from the number delivered. Neither side did anything wrong. The mismatch fell out of the seam between them.

The fix is an invariant: RefreshData returns exactly the topics that were pending at the moment of the knock — regardless of what connections drop, what data arrives, or what handlers fire mid-flight. Snapshot at the knock; deliver the snapshot; everything after the snapshot waits for the next cycle. The penalty for breaking the rule is silent and terminal, so the rule bends for nothing.

Many cells, one subscription

The third behavior is Excel's own bookkeeping around duplicate formulas, which is helpful right up until it isn't. Enter a formula, and Excel assigns it a topic ID and calls ConnectData. Enter the exact same formula in a different cell and Excel doesn't call the server at all: it recognizes the duplicate and copies the value itself. Anything short of a perfect character match — "Bid" in one cell, "BID" in another — gets its own topic ID and its own ConnectData call. It's up to the RTD server to detect when two seemingly different topics are requesting the same piece of data. And the RTD server has to deduplicate on the other side because the TWS API bundles some data in what it considers a single subscription. For example, request a realtime quote for AAPL and TWS will send prices, sizes, and other metadata even if you only wanted the Bid. A typical dashboard displaying quotes for AAPL will have separate cells (with separate RTD formulas) for Bid, Ask, Bid Size, Ask Size — Excel see that as four distinct topics; TWS sees them as one. Fail to deduplicate at the RTD server layer and the system can burn through the API's pacing budget. Fail to account for the fact that multiple Excel topics are depending on a single TWS subscription and deleting one formula in Excel mysteriously breaks others. Internally, each logical topic owns the full set of Excel topic IDs that reference it. Those references are counted, and the upstream subscription is cancelled only when the last cell lets go.

The bug this structure had to survive is the one I think of as the abstraction lied. Early code, at several places where updates fan out to Excel, asked a topic for its topic ID — singular. The topic answered with one of them. The result: the first cell that subscribed to a quote kept updating; its later twins froze at whatever value they last saw — a cell that looks current and isn't, the exact failure this series exists to prevent. The fix was cross-cutting, at every collection site: always ask for all the IDs. Twenty-seven regression tests across seven hundred lines now pin it, because a bug that silently freezes some cells and not others is the kind you write seven hundred lines against.

Noticing before the trader does

None of this can be patched from my side of the COM boundary — Excel is going to keep being Excel. What the RTD server can do is threefold.

1. Hold its own side of the contract more strictly than the host holds its own. (That's the exact-count invariant, kept as if a spec demanded it, because something enforces it either way.)
2. Never depend on the host's timing. The RTD server's cache is authoritative and continuously current. Notification to Excel are asynchronous and rate-limited. The data that get sent to Excel are always the latest.
3. Instrument the conversation. The engine counts its notify attempts and outcomes and runs watchdogs on the update cycle, so a knock unanswered past its window is a log line and a re-arm.

The first article introduced the rule the whole system turns on: Fail Loud — "stale data is worse than no data." The host side adds the corollary this article has been circling: failing loud presumes someone is listening. Excel's quiet failure modes — the paused pulls, the reentrancy sulk, the broken-trust freeze — are all ways the listener leaves the room without saying so. So the discipline extends one step: don't just fail loud; keep proof, in band, that the host is still hearing you.

Next: sending orders from a spreadsheet — an apparent anti-pattern, given that RTD is built for streaming data into Excel. And the one discipline that keeps it safe.