What’s Up With the CDC Nowcast?

How’s it going?

The CDC nowcast last week was 2.7% Omicron. That seemed like a reasonable guess.

The CDC nowcast this week is 73% Omicron, and last week’s nowcast got revised from 2.7% to 12.6%.

That’s two retroactive extra doublings last week, and then four more in the following seven days relative to Delta, for a doubling time of less than two days.

That report came out right after I put out Omicron Post #8, and I quickly noticed I was confused.

One possibility is, hey, large error bars, so I guess there’s that?

This conversation provided potential factors, but did not clear up the confusion.

This provides a clear alternate hypothesis, and I trust the source quite a bit.

Censoring the past ten days is quite a high price to pay given how fast things are moving. That’s a lot of data to give up, and it’s worth noting that 10 days later the difference between 40% and 73% isn’t all that big in context. But if Trevor is right about the speed of submission, and the Nowcast isn’t adjusting, it’s going to give out a nonsense answer.

Which is exactly what it is giving. There are three huge problems with the nowcast’s answer, on top of the revision being rather large and not inspiring confidence.

  1. Where are all the cases?
  2. The regional numbers make even less sense.
  3. Also, it’s averaging over a week so it’s implying even higher rates now.

The third problem I only noticed later, but if it’s 73% for the whole week, and 12% for last week, where did we enter the week, and therefore how high did we have to get to balance out the first few days? Not that this problem is necessary to notice the flaws.

The first issue jumped out at me right away. We know that there wasn’t a dramatic rise in the number of cases overall. We know there wasn’t a rise in the positive test percentage. Yet the claim is there was suddenly, over the whole week, three times as many Omicron cases as Delta. Does that mean that the number of Delta cases was down by more than half inside of a week? Does that seem remotely possible?

Testing capacity is a limiting factor, but if it was having a big effect, we’d presumably be seeing a much bigger jump in the positive test rate. I can imagine a world where that’s not true, but it doesn’t match the data from earlier in the pandemic.

This gets far more extreme if you go to the regional level, and the implications get bonkers.

This was a good visualization of the nowcast by region.

This means that in two regions covering ten states, we had more than 95% Omicron cases, so twenty Omicron cases for every Delta case, whereas a week ago Omicron was a clear minority of cases. Does that possibly live in the same world as our case counts?

This is another good explanation of some of the reasons the data doesn’t add up. This is how one should approach the situation when algorithms produce obvious nonsense. As he notes, this isn’t a knock on the CDC. I can be harsh on the CDC, but this isn’t the time and place for that. It is an example of them doing their best to be helpful, and failing to notice that their algorithm had produced nonsense because they didn’t have a human look at it.

And sure, given how many eyes were on the forecast they should have had a human do a sanity check and put in a warning note. But they also shouldn’t have had to. Everyone else should have also noticed they were confused and that the number didn’t make sense, rather than reporting an obvious nonsense projection with huge error bars as if it were a fact. If I were at the CDC, I’d fix the ‘no human sanity check’ issue but also would be muttering about how this is why we can’t have nice things.

It would be better to not give obvious nonsense as an output, but that’s a lot to ask here. It’s really really hard to create an algorithm that always produces an answer and never produces obvious nonsense. That’s especially true in new situations with different properties that are warping your data. I have a bunch of experience trying to create models of much better-understood situations without producing obvious nonsense, and of watching others make their own attempts, and yeah. It’s hard.

Sure, sure, use the try harder, Luke, sanity check it for five minutes and all that, it’s not like this requires Shutting Up and Doing the Impossible, at most you need to Make an Extraordinary Effort, but this was at least them making a perfectly reasonable ordinary effort to do exactly the right thing. It isn’t the usual situation of our authorities Trying to Try.

There’s even a button to turn off the nowcast.

It also means that when statements like this one from the Stat News article are made, they are an honest misunderstanding, in that they are categorizing the CDC as making a much stronger claim than they actually are.

According to the CDC, in the week ending Nov. 27, the Delta variant accounted for virtually all transmission in the country. The week ending Dec. 4, Omicron made up 0.7% of Covid infections; it climbed to 12.6% in the week ending Dec. 11. In the week ending Dec. 18, it reached 73.2%.

As opposed to at most something like this rewording I’d offer, which really should then end ‘While the growth in Omicron is dramatic and faster than expected, the full nowcast projection does not actually seem plausible give our other data sources’ or something similar:

According to the CDC, in the week ending Nov. 27, the Delta variant accounted for virtually all transmission in the country. The week ending Dec. 4, Omicron made up 0.7% of Covid infections. According to the CDC’s automated nowcast projection, it then climbed to 12.6% in the week ending Dec. 11, and 73.2% for the week of the Dec. 18, with very large error bars.

Trevor’s projections seem far more sane, including when they are extended into the future. The loss of the last ten days is unfortunate and limits their usefulness, but not that much since one can extrapolate. We lose out on noticing any changes in behavior and therefore spread, but the ratio of Omicron to Delta should be mostly as predicted.

Why Does It Matter?

The difference between 40% and 73% Omicron is likely to be something like four days even under relatively conservative projections. It’s all going to be all Omicron soon, and even if the projection is a week ahead all the relevant prediction markets will settle correctly anyway. Why should we care about the current percentage?

Two reasons. There’s the usual reason to always care, and also the implications of the 73% number would be bizarre and force large modifications in our world model.

  1. We should always care to get things right, to learn how to get things right, and to cultivate the virtue of making sure to get things right, and to identify epistemic errors so they can be fixed both individually and systemically.

The next time the media runs similar headlines, you’ll want to notice their conflation of projection and measurement, and also notice you are confused right away, and react accordingly. It’s important to recognize the difference between a measurement and a projection, and have heuristics for which projections have how much credibility.

  1. If we were already at 73% Omicron last week (with 95%+ in some regions) then as was discussed above, where the hell are the cases?

If positive test rates were mostly stable, and cases were mostly stable, but Omicron was three quarters of cases, then that implies a stunning decline in Delta. While Omicron was doubling every two days, Delta would have to be getting cut in half every three.

That in turn implies, in that counterfactual world:

  1. A stunning shift in behaviors, far larger than any previous changes throughout the pandemic. Spread would have been cut, on a dime, by more than half.
  2. This cut in spread somehow still didn’t much slow down Omicron.
  3. The relative R0 of Omicron versus Delta would be even higher than the highest other estimates.
  4. Omicron would be completely unstoppable.

Throughout the pandemic, we’ve had a lot of different sources of information: Deaths, hospitalizations, cases, prevention methods, restrictions and behaviors, variant properties and how much of each is where when, generation times (serial intervals), levels of natural immunity, vaccine effectiveness, methods of spread, population heterogeneity and more.

They all must fit together into a single world with a single set of physical properties. When one measurement is out of whack, if you accept it, it can have implications that throw your entire model off.

This entry was posted in Uncategorized. Bookmark the permalink.

41 Responses to What’s Up With the CDC Nowcast?

  1. Donald Fagen says:

    Omicron may very well be completely unstoppable. And we may come to see this as a good thing once we decide to stop trying.

    I’m predicting gradually more acceptance from health authorities of diminished severity, gradually more recognition of the futility of our measures (including 3 mRNA shots) to prevent cases. Niall Ferguson’s case changed my priors around booster efficacy quite a bit.

    Incidentally, I hope you are your family are well.

  2. Fermion says:

    Re ‘Where are all the cases’: We have a similar mystery in the UK where, according to Alastair Grant’s UKHSA plots [1] we reached 50% Omicron on 13th of December (and this seems to be not very controversial). The cases (by specimen date) [2] went up around the same time (roughly double of the pre-Omicron plateau).

    But since then, nothing really has happened. None of the cases charts is rising exponentially beyond the initial doubling (i.e. 50% Omicron). Daily reported cases (by reporting date) have been 80k – 100k since 16th of December (5 days!), by specimen date also around 100k daily. Initially I thought it might be the weekend but it still holds. I don’t think it’s testing capacity, there seems to be free capacity according to the gov dashboard [3].

    [1] https://twitter.com/AlastairGrant4/status/1473397894971600902/photo/1
    [2] https://coronavirus.data.gov.uk/details/cases
    [3] https://coronavirus.data.gov.uk/details/testing

    I read a suggestion on Twitter that people are disincentivized from testing / reporting positive results as they might have to (legally) isolate over Christmas. But I find it hard to believe that this is the cause. Also a strong behaviour change seems hard to believe. I notice I am confused too.

    • TheZvi says:

      Yeah, I follow him now and find it as puzzling as both of you do. I do buy that the Christmas effect is real because I see it happening around me, as we do our best to try and not lose the family Christmas (given it looks like we had a normal cold rather than Covid), even before we got symptoms we’d decided to take much less risk this week than we would have otherwise. And I can totally believe a ‘well we’ll just not test for it then’ attitude, but as usual the secondary facts should differentiate. Have to look closer.

      • Fermion says:

        Note that outbreak.info [1] says 18-20% Omicron on 13th of December — so 2-3 days behind Alastair Grant. This might be the UK lagging behind London, but even then, I would expect an overall growth from 15th. And also nothing to be seen in London case numbers either — these are new cases by specimen date for “London” region from gov.uk [2]

        2021-12-20 21802
        2021-12-19 19497
        2021-12-18 18131
        2021-12-17 22812
        2021-12-16 26503
        2021-12-15 27650
        2021-12-14 26574
        2021-12-13 20846
        2021-12-12 11357

        I believe numbers from 1 week ago should be reliable, as per Alastair Grant again [3]

        [1] https://outbreak.info/situation-reports?pango=BA.1&loc=GBR&selected=GBR
        [2] https://coronavirus.data.gov.uk/details/download
        [3] https://twitter.com/AlastairGrant4/status/1473325350956732418

        It seems John Schilling [4] was somewhat right in saying ‘And whenever you see someone saying “this is increasing *literally exponentially* and that’s terrible”, you’re usually one to two doublings away from the exponential beginning to tail off’ — it seems to have worked for most previous variants as well (e.g. Delta in India).

        A perfectly straight line on a log plot for 2 weeks, and the it falls off as soon as it becomes a big thing. So the cause might be related to it being a substantial fraction of people? E.g.
        a) People notice and adjust their behaviour, and we’re still vastly underestimating people (official decisions matter less than what people think, as per Lockdown Effectiveness: Much More Than You Wanted To Know)
        b) Our trivial modelling breaks down when we reach ~ 0.1% population? People always say it’s bad but why would it be so bad?

        [4] http://slatestarcodex.com/blog_images/schilling_comment.png, from ACX open thread 202.5

      • TheZvi says:

        I mean I had the “And That’s Terrible” going when London cases were single digits, so there have been a bunch of doublings since then! Not that it changes the central point.

        But… yeah, I believe there was a big behavior change. I do not believe it was a sustainable one. We’ll see. Still investigating.

      • jimv says:

        I would definitely have put a lot of weight on the theory that people (in the UK) are locking themselves down / taking fewer risks, in an attempt to save their own Christmases in the absence of much government tightening.

        But I’ve just looked at the Google Community Mobility Report, and the national levels aren’t immediately shouting signs that the last week or two have seen massive drop-offs in movement, which is surprising to me.

        https://www.google.com/covid19/mobility/

        Greater London area’s charts do look a bit more down though, so perhaps some effect of people there recognising how much COVID there is about?

        Today’s (22 December) Omicron daily overview is just out, and the proportion of cases that are Omicron (well, SGTF) still rising, now at around 76%. And doubling times are ~1.9 to 2.5 days, depending on region.

        https://www.gov.uk/government/publications/covid-19-omicron-daily-overview#history

        Overall case counts continue to rise. I think today’s 106,122 is the highest daily number the UK has yet reported.

        https://coronavirus.data.gov.uk/details/cases

    • Jason says:

      That’s interesting, we’re seeing the same flattening of cases in Denmark. Doubling time of ~1.8 days until Dec 15, and then suddenly almostly complete flat. No fall in Delta cases and no significant rise in Omicron cases. The only plausible theory I have for this is mentioned elsewhere in this thread: that Omicron has huge numbers of asymptomatic, untested cases and is already beginning to saturate. Almost 40% of Danes have got a booster, so it makes sense that this would happen at a relatively low number.

      See the chart/numbers here: https://docs.google.com/spreadsheets/d/1SrPl5MDq7csY8WJ6id5SBmMqURlAxHGvbqFH5uUHrcU/edit?usp=sharing

      • Jason says:

        Also SSI has announced that they will only test for variants on a random sample of PCRs going forward rather than all positives, so I’m hopeful we will have more timely data in the coming variant reports (released every day at 2pm).

  3. jes5199 says:

    How about: an unstoppable, mostly-invisible Omicron is conferring immunity so quickly that Delta is disappearing?

  4. Dave says:

    >>Does that mean that the number of Delta cases was down by more than half inside of a week? Does that seem remotely possible?

    Could it be the result of viral interference? I’m sure that you can’t simultaneously have an Omicron infection and a Delta infection, for the same reason that you can’t have a cold and the flu at the same time.

    The way this works is not well understood by virologists, but it’s a real phenomenon.

  5. Jonathan Monroe says:

    There has been a significant behaviour change in London. My parents (triple jabbed and pretty nonchalant about the risk to their own health since getting the second dose) wanted to visit the grandkids and decided to stay outdoors in order to minimise the risk of having to quarantine over Christmas Day. They arrived early because an 8-mile drive that normally takes 45 minutes to an hour only took 25 minutes.

  6. Graham Blake says:

    There are some proxy measures that we can look at to get a handle on things and update our intuition. My informal analysis of some data in Canada suggests some truly eye-popping changes in the past few weeks. Using data from Alberta, for instance (which provides daily updates for new/active/hospitalized cases and their vaccination status) we can do a simple screening method estimate of vaccine effectiveness. We can then take the change in this VE estimate over time to infer what kind of impact Omicron is having.

    Around October 31 in Alberta, the new case rate for the vaccinated population was ~3.46/100k and for the unvaccinated population it was ~18.52/100k, for a VE of ~81.3%

    By December 5, the new case rate for the vaccinated population was ~6.14/100k and in the unvaccinated population it was 21.84/100k for a VE of ~71.9%

    The December 21 numbers from yesterday… they almost beggar belief…

    New case rate in the vaccinated population: 18.88/100k
    New case rate in the unvaccinated population: 21.35/100k
    VE of ~11.6%

    Obviously a simple screening method like this has all kinds of confounders, but as a rough guesstimate I think it paints a fair picture of just how dramatically VE has fallen, and how quickly. I believe it is a fairly reasonable proxy estimate for just how prevalent Omicron has become, and how quickly.

    In terms of “Where are the cases?”, in Canada at least the cases are most definitely showing up. We are rapidly running into testing capacity limits. On December 13, the 7 day moving average of new cases was 4,097 or about 10.7/100k. Yesterday’s new case numbers were a new daily record for Canada at 12,813 or about 33.5/100k. Now we are at our testing capacity limit in many large centres and it doesn’t appear that new case numbers are going to be very accurate going forward.

    None of this provides a direct measure of how rapidly Omicron has grown as a share of all cases, but it does allow us to believe it has become a very large share very quickly.

    • Graham Blake says:

      And now today’s numbers out of Alberta are even more disturbing. This is going to be fun to have to explain to people. Vaccination is now showing negative apparent effectiveness vs. being unvaccinated.

      New case rate
      Vaccinated: 34.63/100k
      Unvaccinated: 21.35/100k

      That works out to being unvaccinated being 38.3% more effective than being vaccinated, in preventing new detected cases.

      There may be some health-seeking behaviour bias in this, but I think it’s going to be very hard to escape the conclusion that immunity from previous infection is holding up significantly better than prior vaccination. In just over 2 weeks, the new case rate for the unvaccinated is practically unchanged (21.84/100k on December 5 vs 21.35 December 22) and but almost 6x higher for the vaccinated (6.14/100k on December 5 vs 34.63/100k December 22)

      Naturally, these numbers are skewed by a higher rate of previous infection in the unvaccinated cohort, and if we were able to compare the new case rate in the vaccinated to the unvaccinated with no previous infection, I am sure vaccination would be at least be on par (I sure hope!), but still. This is a dramatic.

      It would require the unvaccinated cohort to have a rate of previous infection of ~22.2%, and for all new infections in the unvaccinated to be in the 77.8% that had no previous infection for the new case rate among the vaccinated to be on par with the the unvaccinated without prior infection. Since infections in those previously infected are certainly happening, even if the rate of previous infection were significantly higher than 22.2% in the unvaccinated cohort, it seems like an inescapable conclusion that previous infection provides much more robust protection against symptomatic Omicron infection than two-shot vaccination.

      • Graham Blake says:

        I think I committed a copypasta error, the actual new case rates in Alberta today are:

        Vaccinated: 34.62/100k
        Unvaccinated 26.86/100k

        So there is actually some growth in the unvaccinated population, but it is very modest indeed. Today’s rate is 123% of the rate of December 5 in the unvaccinated population, and 564% higher today in the vaccinated population compared to December 5.

        I wonder if the “extreme transmissibility” of Omicron is actually only applicable to the vaccinated population, but it’s similarly or only slightly more transmissible than Delta in the unvaccinated population.

    • Dav says:

      Thank you for bringing up the situation in Canada. Here in Quebec we went from 1100 cases/day in the first week of December to over 6000/day and that is limited by testing capacity. They were actually closing drive through testing centres because cars lined up for kilometers were blocking side roads and there were too many instances of road rage.
      The actual number of cases is probably closer to 10000/day.
      All along our governments have talking about controlling the spread of the disease and it has more or less worked.
      BTW, Canadians wear masks in schools and stores since the summer of 2020.

      • Graham Blake says:

        It looks like Quebec actually damned near hit 10,000 test positive cases today. I don’t think most regions in Canada ever developed the testing capacity of some other regions of the world, so I think we have probably hit the wall in terms of cases per day we can test for. Ontario hit 18% positive today.

  7. Tom W says:

    I think no matter what else could be wrong, misinterpreted, or insane about this data, it’s clear that Omicron has been in the US (and in places like the UK too) for far longer than anyone would like to admit before anyone noticed anything.

    There’s simply no other way to explain so many cases spread across such a wide area–once again, much like March 2020, we weren’t looking and we were finding nothing, then we started looking and finding plenty, and the correct conclusion to draw is not that this thing appeared everywhere overnight, it’s that it’s quietly been everywhere for a while.

  8. JS says:

    Why are we rejecting the hypothesis that Omicron is undetected and/or unreported at higher rates than previously? It has been assumed all along that less than a third of COVID cases are caught in official case counts, with low estimates running as low as one in ten. If Omicron is milder than delta and is further mitigated by existing immunity (vaccine or natural), then I find it quite plausible that an even larger portion of Omicron cases are undetected. That in turn supports the hypothesis that Omicron is continuing to double as expected, but reported case numbers are not tracking because a larger and larger proportion of actual infections are undetected.

    • Humphrey_Appleby says:

      Yes this.

      If you keep piling up disincentives to testing, then you should not be surprised if people, in fact, stop testing. Particularly since the expected severity (conditional on vaccination or prior exposure) should now be well below literally seasonal flu levels.

      • Brett Bellmore says:

        I continue to not understand, (Maybe I don’t want to understand?) why we fail to do random sample testing of the population, so that we can have decently reliable numbers to work from. Were I Covid Czar, this would have been like the very first thing I would have instituted.

        If we’d been doing that all along, we’d have no questions at all about the numbers.

      • Humphrey_Appleby says:

        @Brett Bellmore Probably because we don’t have a good way to compel random samples of people to get tested. We can do screening of this kind in restricted settings (higher ed, international travelers), and we can do wastewater surveillance, but we can’t easily pick e.g. a random 1% of the population every week and force them to get (and report) a test.

      • Brett Bellmore says:

        Who needs to compel people when you can bribe them? Pick a random tenth of a percent of the population, or whatever, and offer them money to be tested.

        Pretty soon you’d have people treating it like winning the lottery, not avoiding it.

    • BH says:

      We have plenty of situations like this, probably the best example is screening testing within higher ed, even where asymptomatic cases are all caught. At least for beta and Delta we did not see a significantly higher proportion of cases in those environments relative to the general public, indicating that asymptomatic spread was not wildly underestimated.

  9. Rage against the R0 says:

    Dear Zvi, two critically important questions:

    1. How likely am I to avoid the omicron infection under the following precautions? There are 4 people living in my apartment, in a block with 8 apartments and ~20 people. I will seal all the gravitational ventilation vents (grills) in my apartment with a couple layers of duct tape and paper/foil (in principle, the air goes from the apartments to the common duct and should not return, but there’s such a risk). There will be two air purifiers working 24/7 + constantly, very slightly opened windows. Nobody will leave the apartment over the course of the next 8-12 weeks, except for one person in an N95 mask and visor, going through the common staircase 3x a week (2*1 minute each time) to take the trash out. Any parcels will be received by the same person in an N95 mask and protective visor on the staircase (~20-second contact each time), and we will strive to minimize their number.

    2. Given the current knowledge, how would you roughly update the Microcovid model – e.g. if weekly shopping had a 5,000 microcovid budget under delta, how would it likely look like now and in the upcoming weeks, given our knowledge on omicron’s prevalence, its basic reproduction number, resistance to the vaccines, etc?

    • Dave says:

      Re #1: If you’re thinking of going to *that* much trouble, why not a P100 instead of an N95? The difference in filtration isn’t a big deal, but the quality of the P100’s skin-tight seal is much better.

      • New Matrix sucks says:

        Would it make sense to frighten the neighborhood with a P100 mask (instead of wearing an N95) if it’s a matter of spending 48-72 minutes total over the course of the 8-12 weeks on the empty staircase?

      • Dave says:

        I don’t disagree, but they’re talking about using face shields, which I assume would already have a similar effect.

  10. bruh says:

    It’s important to note that this is not an automated system that nobody is reviewing that the media has picked up and is running with on their own.

    Walensky is repeating this on her twitter (https://twitter.com/CDCDirector/status/1473088297039761411) and in TV inteviews (ex. https://www.cnbc.com/video/2021/12/20/omicron-is-now-about-73-percent-of-the-virus-sequenced-in-the-u-s-says-cdc-director-walensky.html)

    (sorry if you included this in your commentary and I missed it, it’s all running together tbh)

  11. Teddy Weverka says:

    Use a sudden rise in case counts as a proxy for when Omicron takes over in prvalence.
    We expect the total to case count behave like Delta when it is dominant and behave like Omicron when it becomes dominant. The high growth rate makes is easy to pick out on a graph. Look at the case count for District of Columbia and you see that a week ago omicron became dominant. Look at the case count for NYC, by week ,in cases/100,000 per week The last four weeks are 122, 191, 274, 872. You don’t even need a graph to see it. For D.C. it is 68, 173, 199, 935. For Puerto Rico, the case count by week, in cases/100,000 per 7 days is 14, 24, 21, 252.
    San Francisco: 346, 621, 581, 1383
    Miami-Dade: 1630, 1985, 2657, 11543
    Honolulu: 315, 385, 857, 4125
    Cases more than tripled in three dozen large counties nationwide.
    But this can’t be the case nationwide. If 73% of the cases nationally were omicron, we would be seeing this change in most counties.

  12. Teddy Weverka says:

    From the CDC website:
    “*The median time from specimen collection to sequence data reporting is about 3 weeks. As a result, weighted estimates for the most recent few weeks may be unstable or unavailable. CDC’s Nowcast is a data projection tool that helps fill this gap by generating timely estimates of variant proportions for variants that are circulating in the United States.”

    This thing doubles about every 3 days. A small error in estimating the growth rate, projected through 7 doublings leads to a large error in the projection.

  13. Brian Lui says:

    Would be super interested in discussion on whether Omicron will displace Delta, or if Delta will be active alongside Omicron (i.e. Omicron wouldn’t confer much protection against Delta). Don’t really have a good understanding of the mechanisms there.

  14. Pingback: Get Set, Also Go | Don't Worry About the Vase

  15. Pingback: Omicron Post #10 | Don't Worry About the Vase

  16. Pingback: Omicron Post #11 | Don't Worry About the Vase

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s