Sunday, 12 Jul y 2009 - 02:30 UTC

I just got turned down for a grant. That’s nothing new, you win some and you lose some, and every senior professor has gotten used to that over time. This time, however, I cannot find it in myself to just say “oh well” and let it go at that. This time, I think I need to go public, because I think what happened shows an endemic problem with the US National Science Foundation and, I hope, points out some things they could do to fix it.

The proposal was for an NSF “Expeditions in Computing” grant, and we put together a team of which the final panel review said “the research team is among the best in the world” (as well as me, the PI team included three winners of MacArthur “genius” awards and one of the top researchers in theoretical computer science). They said our vision was “compelling,” and that the proposal “outlines an ambitious research program to attain this vision, while at the same time with realistic ambitions as to what can be done within the resources of the expedition.”

There are in fact, no real criticisms in the review until the summary which after saying more nice things about the team and the vision and the technology (a potential new breakthrough technology for the Web) said
The panel was concerned with the management plan, in particular how to ensure collaboration among the diverse research communities, especially given the number of strong intellectual leaders among the team members.
Or, in my words, they were worried how a team of such mavericks could pull together.

In the end, NSF told us it had been “very very close” but that they were just not quite willing to deal with the risk and they could not fund us. But here’s the thing — this is supposed to be “high risk, high reward” research. In the end, NSF simply could not bring itself to take the risk!!

OK, now if it just ended there I would expect, rightly, for you to say “sour grapes” – but I hope if you’re a US taxpayer, and looking at NSF turning down what the reviewers said could be breakthrough research in new ways of using the Web, you’re annoyed at this. “What,” you might ask, “could be done to improve the situation?”

Here’s some thoughts I have based on having been both a researcher and a funding agent at different times in my life:
1 – Many funding agencies in this situation would be able to come back to the team and work with us to craft a management plan that would make sense to both sides – we wanted enough freedom to use our judgment in which paths to follow, they wanted some better assurance we would pull together. In feedback from some NSF PDs they have ideas of exactly what we “should have said,” but they couldn’t give us this feedback until after we were rejected. In short, NSF cannot work with a “near miss” to craft a winner, they can only say yes or no.
2 – Having been turned down, we can, of course, submit again next year. But the amount of effort that went into a proposal of this magnitude (we had 36 researchers in 11 different fields at 8 universities) is quite high, and next year we would be started again from scratch. NSF has no means where we could say “last year we almost made it and this year we fixed the management plan” – so we’d have to do preproposals, proposal if invited (no sure thing), site visit if we made it that far, etc (and all for different review panels who would have different questions and issues). There would be no advantage to having made it so far in the first round. Most funding agencies have a way of giving some sort of extra consideration in these sorts of cases, NSF doesn’t. Starting from scratch with the high likelihood of the same outcome (because this team wouldn’t be interested in doing anything if it isn’t high risk) is not a compelling idea, and we won’t be doing it, and NSF has nothing to offer to encourage us to do so.
3 – NSF reviewing is now all done by panels. In the old days, it was all done by external reviewers (as many countries still do it). There is a lot of anecdotal, and some formal, evidence that panels tend to be more conservative. On the other hand, the workload of external review was way too high. The solution might be, particularly in situations like this, to solicit outside review to be compared with the panels (or to comment on specific matters such as “does the management plan meet what you know of these reviewers”). The panel who were at the site visit only included one person who has done published research directly relating to our proposal, and a couple more who knew parts of it. Getting some people who would have pointed out that many of the people in the proposal had coauthored with each other, and we were all known for working in large groups, might have helped alleviate NSF’s fear.

I know NSF does a great job, putting up with constantly changing oversight from Congress, dealing with a public that doesn’t understand science, and trying their best to fund the best work while keeping the bureaucrats happy. Still, I must tell you that I am bitterly disappointed that they turned down a grant that had, as the reviewers themselves described it, “the potential to transform the use of widescale, networked computing” with _"a dream team of Web researchers and also top researchers in foundational computer science and social science." I would have loved to work with this team, and I’m sure many of us will now go and find other ways to pursue pieces of our vision. Knowing the team, this setback will delay this research getting into practice, but won’t stop us — we’re too "unmanageable"to give up. But I sure hope that the agency can understand why this turn down is more of a condemnation of them than us, and I hope the constructive solutions above can lead to NSF’s rethinking their approach to funding high-risk research.

addendum July 16, 2009
I was beginning to enter the “acceptance” stage of this rejection when I came across a 2005 report by the National Research Council (the organization that coordinates the National Academies in the US – the highest level scientific societies we have) entitled “Facilitating Interdisciplinary Research.” Their number one recommendation with respect to funding was

Funding organizations should recognize and take into consideration
in their programs and processes the unique challenges faced by
IDR with respect to risk, organizational mode, and time.

exactly what NSF did not do in this case – in fact, the exact reason they said they rejected it. Ahh well, they didn’t trust our research team which included a number of members of the National Academies, why should I expect them to listen to the NRC. Sigh – back to denial, anger and depression…