Aaron Swartz discusses the possible problems of relying too much on scientific decision-making in Do I have too much faith in science?:

If you’re struggling with a decision, we’re taught to approach it more “scientifically”, by systematically enumerating pros and cons and trying to weight and balance them. That’s what Richard Feynman would do, right? Well, studies have shown that this sort of explicit approach repeatable leads to worse decisions than just going with your gut. Why? Presumably for the same reason: your gut is full of tacit knowledge that it’s tough to articulate and write down. Just focusing on the stuff you can make explicit means throwing away everything else you know—destroying your tacit knowledge.

My initial reaction was probably similar to yours. Something like this:

As expected, many commenters on Aaron’s post vehemently disagrees with him. Joe Blaylock asks:

You seem to take a narrow view of what science is and how it’s done. Is this rhetorical? Are you representing an extreme reductivist worldview to try to make a point?

gwern tells an interesting story to make his/her point:

‘One day when I was a junior medical student, a very important Boston surgeon visited the school and delivered a great treatise on a large number of patients who had undergone successful operations for vascular reconstruction. At the end of the lecture, a young student at the back of the room timidly asked, “Do you have any controls?” Well, the great surgeon drew himself up to his full height, hit the desk, and said, “Do you mean did I not operate on half the patients?” The hall grew very quiet then. The voice at the back of the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the visitor’s fist really came down as he thundered, “Of course not. That would have doomed half of them to their death.” God, it was quiet then, and one could scarcely hear the small voice ask, “Which half?”’

To bring this debate over into the world of design, I like Dmitry Fadeyev’s description of the different approaches as Primal (intuitive) vs. Cerebral (scientific). He concludes:

The best work is probably a combination of the two forces: restraining the primal force enough to yield a useful product that performs, but not ignoring it altogether so that the more basic human element is satisfied too, both in the creator and in the user.

Dmitri explores this theme more in his essay The Cerebral Designer:

Likewise, primal and cerebral design instincts are complements, not opposing forces. They are concerned with disparate goals which is why neither is better at achieving what the other sets out to do. If the design is driven only by the cerebral creative instinct, it will be too plain. If it is fully primal, it will not be very good at fulfilling its function for it would be more of an illustration or an ornamental piece than a design. Instead, if the primal is restrained by the cerebral but not yet fully killed, we arrive at a design that is functional, structured, pleasing to the eye and a joy for the designer to create.

I guess as designers we’re lucky. Instead of having to pick extreme points of view in an argument, in many cases the easy way out (calling for a middle ground) is also what’s best for our work. That is certainly the case here. We can combine things like A/B testing (within limits) with an intuitive humanity to design memorable, usable experiences.