NOTE: A version of this post was originally published at Tamarack’s Communities Collaborating Institute Online.
Here at Taylor Newberry, when we do evaluation work, we try our best to create surveys and tools that are thoughtful, accessible, and useful. We try to avoid creating mindless paperwork. I have just finished reading The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureaucracy, by David Graeber, and it makes me think perhaps we have to be even more vigilant to guard against the creep of bureaucracy. The book is a dense, difficult, sweeping, and sometimes incisively brilliant book on the way in which bureaucracy has permeated every aspect of our lives and constrained the way we think about democracy and about capitalism. Although Graeber does not comment on program evaluation directly in his book, I suspect he might say that evaluation work is often bureaucratic. And if evaluation is bureaucratic, then he would argue that it is oppressive. His argument goes something like this. Bureaucracy is about trying to make the world fair and transparent by replacing human judgement and open ended social negotiation with a set of clearly explained and consistently enforced rules. It is about trying to make society run like a formally organized sport. In sports, what counts as a goal or a touchdown is objective. What counts as a penalty is agreed-upon by everyone. If you play by the rules and practice hard, you will win. For the author, bureaucracy is the idea that we can solve social problems and build a fair and just society if only we can create a set of rules and procedures that are up to the task.
The problem, Graeber argues, is that this is a utopian vision. Like all utopian visions, it can never be sustained in the real world. Social problems are much more complex than the problems one faces in a hockey game. They can’t be fixed by creating a better set of rules. In any modern bureaucratic society, those who have power and privilege will always have the discretion to bend or break the rules. Those who are treated unfairly will always seek to find loopholes within the rules, or opt out of participation in the game entirely. The ultimate form of bureaucratic utopia, the author argues, is when we come to love the system of rules more than the people those rules are intended to help. We buy into the utopian idea that the rules are capable of creating a society that is fair and just. If only, we say, you had filled in the form correctly, or read the small print, or completed your letter of intent on time, the system would have worked properly. It would have been fair and transparent, and you would have gotten the money, the support, or the consideration you deserved.
Of course, no system of rules is ever perfect. We have all seen corruption, selective use of the rules, and lack of transparency in the system from time to time. Paradoxically, we typically try to solve these problems by creating additional layers of bureaucracy. Eventually, Graeber argues, we create nested systems of rules so arcane that no-one really understands them, and then we blame the marginalized for failing to understand.
Evaluation work might be seen as having contributed some uniquely perverse flourishes to our society’s system of rules for addressing social issues. We have said that social interventions must be both innovative and evidence based. Not too many strategies can meet both those standards, and getting better at one of them often means you lose ground on the other. Funders and evaluators have also said that programs must serve those with complex needs, but that the intended outcomes should be focused and measurable. We have said that evaluations should not be undertaken without adequate resources, but that all agencies should be required to do them. We have said that social change is complex and requires time, but that evaluation projects should be completed quickly and efficiently. These stances could all be seen as examples of rule sets that no nonprofit can fully obey, in a game that the house always wins.
For Graeber, who is one of the key intellectual leaders of the Occupy movement, truly democratic interactions are those in which we set aside the rules and enter into open-ended, unstructured and non-hierarchical social negotiation. In these interactions, we step together in the unknown. We name and address power dynamics directly, and don’t pretend they can be legislated away using a system of rules.
So – is evaluation inherently bureaucratic? Is it necessarily and always an attempt to use surveys and forms as a way to avoid real, honest, open ended interaction with people? I don’t believe I have fallen in love with my rules. If you are engaging people in the process of design, and collecting data that you intend to use, and if you are asking questions that really matter both to you and to your participants and to other stakeholders, then the process isn’t mindless paperwork. Done properly, I’ve always thought evaluation can actually be a kind of social negotiation. It can be a way of starting conversations and building understanding across different groups of people – a process that can lead to real learning and action.
Even so, this book is a great reminder of how easily the good intentions of evaluators can degenerate into actions that are inadvertently bureaucratic and potentially oppressive. Even if I don’t fall in love with the rules I create, it is possible that other people somewhere in the system will. In my practice, I need to do all I can to prevent that from happening, by building evaluation designs that ensure data collection will always be linked to real, person to person communication.