Predicting updating

Published: 2026-04-06 Edited: 2026-04-21 Reading time: 3 min Tags: everblue Draft: Yes Part of: everblue notes

Joe Carlsmith’s piece, “Predictable Updating about AI Risk,” is an excellent foray into the topic of how people should think about updates they expect to make in light of evidence they might expect to receive on AI risk. But I’d like to think more generally about to best orient to the anticipation of updating my credence on some event — for example, in cases where it’s hard to predict how your credences will change in light of evidence, or in cases where you have a distribution of possible updates you might make (rather than just two), or in cases where various different pieces of evidence might interact with each other in complicated ways.

As one example: say we’re at time t_0, and there’s some event A which will either happen or not happen at time t_10. there’s some other event B, which will significantly raise your probability that A occurs, which will either happen or not happen, with some uniform distribution spread evenly between t_0 and t_10. as each step from t_0 to t_1 passes and B hasn’t happened, your credence that A will happen should decrease commensurately. then, of course, if B were to happen — say, at t_5 — your credence that A will happen should shoot up, and then predictably not move again. Of course, if B never occured, then your probability that A occurs just continues to go down with each time-segment in which you don’t see B.

You could also, for instance, have some C, which functions in precisely the opposite respect to B: that is, you have a uniform distribution over C occuring between t_0 to t_10, and C’s occurence provides quite a lot of evidence against A happening at time t_10. Let’s stipulate for simplicity that C would change the odds you’d place on A by exactly the inverse factor as B would change your odds on A; that is, C might change your odds on A by a factor of x, and B would change your odds on A by a factor of 1/x. So, each time-segment that passes without either of B or C occuring, your odds on A predictable won’t change (B-not-happening should make the odds of A go slightly down each time segment, but C-not-happening should also make the odds of A go slightly up each time segment, in a way which by stipulation cancels each other out). Then, if at t_5, both B and C occur simultaneously, your odds on A should… continue remaining the same.

But if instead, at time t_5, after your probability in A has remained steady all through t_1 through t_4, only B occurs, then: your probability in A should jump up (after all, we stipulated that B provides a lot of evidence for A), then start slowly decreasing with each time segment that C doesn’t occur.

(As a quick note: this image looks different from the one above labelled the same thing, because the first one was not thinking about the event C, but the second one was.)

And if only C occurs at t_5, then your probability in A should jump down at t_5, then slowly rise with each additional time segment.

Some questions to make this a bit less clear:

What if seeing C happen changes your odds on B happening?
What if you won’t “see” B or C happen, but you might encounter evidence that changes your odds on B or C happening?
What if there’s some event D that acts on some mix of A, B, and C? What if there’s many, many other such events, that each act in their own mix of ways on their own subset of other events?
What if B or C aren’t uniformly distributed across t_1 through t_10? Maybe they have some weird trimodal power-law distribution across time segments, and maybe ‘not seeing B and t_1’ will change the shape of the underlying distribution of seeing B at future time segments.
etc.

After writing all this, I’m still confused, but a little more confident about where I’m confused, and a bit more clearly thinking through the problem.

See Poisoned water & anthropic effects for an interesting edge case of this kind of problem.

graveyard:

As one particular example of a case where Carlsmith’s piece simplifies: he seems to say “look, the median direction of update that you anticipate making isn’t the one you should make — it’s the expected update that you should make. it may be that you anticipate in most worlds your credence rising, but you need to take into account both the direction and the magnitude of that update. ‘updating all the way’ should only happen to your expected updates, not to your median direction of update.”

So: what about more complex cases of distributions of possible updates?