I recently heard Robert Sutton speaking at Stanford. He and Jeffrey Pfeffer have written a book, Hard Facts, Dangerous Half-Truths and Total Nonsense, in which they champion what they rather unimaginatively call "evidence-based management."
Their central argument is that most companies, especially at the management and administrative levels, make decisions based more on tradition and intuition than on experimentation and evidence. We do things because…we have done it for years. Or, I used 6 sigma at the last company I managed, so I'm going to use it here. They argue that a stronger emphasis on testing will profit companies quite a bit.
This dovetails with an organizational communications class I took. We reviewed a basic decision-making model. It went something like this:
- Identify the problem
- Talk about possible solutions
- Research the possible solutions to narrow them down a bit
- Test some of the solutions
- Implement the best solution
This was a nice, neat, clean theoretical model for decision making that, organizational communications researchers found, real companies didn't follow. I guess they hadn't taken the org com class in college.
An example:
Sutton talks about a top-level manager in a Vegas casino-owning company (Harris? Harrison? - I forget the name of the company). This manager took an evidence-based view of management. He said, Let's test the assumptions of casinos in Vegas. They may be correct, but we should know if they are or not. Then we can move forward, knowing that we're doing the right thing. The assumptions are that we'll make the most money by:
- Focus on high-rollers, who are our big sources of income
- Attract gamblers with free or discounted hotel rooms
- Make the environment child-friendly to attract families
- Make the hotels glamorous and glitzy to attract people.
After some experiments, they found that every one of these assumptions has something wrong with it. It turns out that most of a casino's money comes from people who live nearby and come frequently, spending a small amount each time they come. So that invalidates the first assumption. It also invalidates the second - the best way to attract these people isn't cheap rooms (they live nearby), but in fact free meals and snacks.
Making a casino child-friendly isn't much good, since families with children don't have much discretionary money or time.
And instead of spending on glitz, money can be better spent on training staff well so that customers, who are spending most of their time inside the casino, not outside looking at it, will feel well-treated.
The lesson, of course, is that there are lots of things that seem to make sense, both in business and out, that are wrong. If you have the opportunity to test, you should.
Comments
The casino example is one a stunning illustration of their argument. What seems very intuitive and sensical to many (me included!) is sometimes so, so wrong. I keep this example in mind whenever I'm swayed by what sounds like a good argument - I always try to ask, Where's the evidence?
Now, the book itself may be a bit hand-wavy. I haven't read it - it's on my book list. For a negative review (it's short), visit this Niblettes review.
If you've read it, by the way, please add some insights.
Counter-intuition: Examples
In the field of medicine
I recently read in Scientific American (I think - fill in if you know the correct source) that a lot of medical techniques have actually never been tested. Doctors think they are useful, but a meta-study found that a significant portion in common usage today are actually found to be completely ineffective - they neither help nor harm. The thing is that doctors don't generally trust studies. They KNOW that something works, and don't listen if a study tells them something otherwise.
To be fair, there is an argument in the doctors' favor. Gordon C S Smith, a professor, and Jill P Pell, a consultant, published a paper titled Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. The study asked the question: Has anyone tested whether parachutes actually increase survival when jumping from a plane? They found, much to their surprise (inject sarcasm here) that no controlled experiments had been conducted.
To properly test the effectiveness of parachutes, the authors wrote, experimental subjects should jump out of a plane after randomly being assigned to jump with or without a parachute. The experimenters should compare the survival rates between the parachute and non-parachute groups, and only then draw conclusions about the effectiveness of parachutes. Perhaps they aren't necessary after all!
It is, of course, tongue in cheek. The point they go on to make is that many medical techniques cannot really be tested, because it would be inhumane to try to withhold treatment from someone who is dying or injured.
Baby Einstein
Another study compared cognitive skills between babies who watched Baby Einstein, and babies who watched other TV shows. It found, surprisingly, that the Baby Einstein watchers did worse on the cognitive tests! Psychologists explained the results by pointing to the fact that babies learn best by hearing normal speech. Regular TV shows give them that experience. Baby Einstein, on the other hand, purports to ch to a baby's level by using simpler words and sounds. This prevents a baby from learning at its full potential.
Satisficing
Decision-makers in real life don't choose by examining all the options, testing them out, then instituting the best one. One good reason is that all this takes resources. Managers don't have the time, money or personnel to test out every option (called optimizing). Instead, they satisfice: they set a threshold of acceptability, and any solution they find better than that threshold is accepted.
Imagine a manager looking to hire a new secretary. An optimizer would search high and low, put out job wanted ads in newspapers and on craigslist, and spread the word through employees. The interviews would go on and on, until the manager was sure she had the best secretary on the job pool. Clearly it makes more sense to interview a few people, and hire someone who is "good enough."
