I joined this list only recently, and was rather nonplussed that some
early comments resulted in challenges to submit my personal
observations to ABX testing. I demurred with as much good humor as I
could muster, even after a suggestion that I was “afraid” to do so.
Several references to Boston Audio Society papers were made, and I have
noticed that other posters have gotten responses that reference the
same test protocol, and that more posts have recently commented on it.
While this list is far more civil than a Rec Audio newsgroup back when,
it was evident that proposing an ABX test remains a convenient
rhetorical device to dismiss an opinion with which one disagrees.
My undergraduate degree was in (Experimental) Psychology, so I have
some conceptual understanding of and respect for the problems of test
design. I was also a dues-paying member of the BAS back in the day (so
was CJ, I think), so I am familiar with their fascination with ABX
testing. My most recent encounter with such a test was at The Home
Entertainment (parasite) Show at CES in Las Vegas last month. The tests
were run by, believe it or not, the Wall Street Journal. The sound from
either A/B choice was so a-musical I walked out, chuckling in
amusement. But it was probably not the worst sound on hand in Vegas
that week, either.
First, let me suggest that anyone who demands an ABX test as “proof” of
another’s personal experience, but who has never actually designed or
even participated in one is simply blowing smoke, and should refrain
from doing so here.
Second, for anyone on the lists unfamiliar with ABX tests but given to
intellectual curiosity, I can suggest a simple, free illustration of
the process. ABX testing involves exposing a subject to sample A, then
sample B. Sample X is then provided, and the subject decides whether it
is identical to either A or B. If it is a double blind test, neither
the subject nor the tester knows what A, B, or any X is. If you Google
ABX, you will find a lot of angry audiophile bickering, but an ABX test
can be designed for any situation that involves the cognitive processes
of perception, memory, and identification. So, with apologies for the
added bandwidth, a challenge follows:
Go to a paint store. As we want to avoid “coloration”, go to the rack
of “Neutral” paint patch strips, and choose one where the differences
in tint, hue, and saturation are minimal. (Of course, to be truly
”blind”, you can have someone else do all this). While the clerk is
distracted, swipe 11 strips. After you get past mall security, pick the
two patches that are the closest. Let’s say they are called Pewter Fog
and Pearl Mist. Cut out the patches (eliminating the names) and label
the backs of Pewter Fog A and the Pearl Mists B. Cover 10 sets of A and
B labels with opaque but removable tape. Set aside the remaining pair
labeled A and B, and dump the rest into a bowl deep enough that the
contents cannot be seen or counted.
Place the two labeled A/B patches in front of the subject, label side
up. Place one empty deep bowl labeled A to the left, and another
labeled B to the right. Put the bowl of taped samples where they can be
reached but not seen.
The subject (it can be yourself) can turn over A and examine it but
must replace it face down to look at B, and vice-versa. Placing both
face down, a sample is drawn from the bowl, and the subject must decide
if it is identical to A or B, and place it in the appropriate bowl. The
process is continued until all the samples are sorted. You can then
peel back the tape and see how many were correctly ID’ed and sorted.
Any more than 10 correct and you are better than chance. Score 20 and
you can be referred to on this list as “Golden-Eyed”.
While this seems a pleasant enough parlor game, it would not cut the
mustard in even an intro course in Experimental Psych. The result above
is just a single data point; one needs another variable for the test to
have meaning. Repeat the test several times to see if the skill can be
learned. Test 100 people and sort by age, gender, religion, etc. Change
the lighting from incandescent to fluorescent. Compare putative experts
(artists, house painters) to the overall population. Choose different
patches and correlate to the pigment differences. Etc. And of course,
you can have lots of fun twisting the statistics, but we won’t go there
If very few subjects can reliably tell the difference between Pearl
Mist and Pewter Fog, they are perceptually identical, according to the
statistics typically applied. If the new CEO of Benjamin Moore was
hired from Crayola, he may decide to "dumb down" the variety of colors
available and eliminate one (or both) colors from the line in order to
impress the stockholders with his bold vision. Of course, the test
actually tells us nothing about the qualities of paint that determine
consumer preference – like durability, fade resistance, hiding power,
etc. Or which shade your mother would choose for the dining room. And
the marketing dept would never let such decisions be made “blind”,
anyway. The use of evocative names acknowledges that some buyers would
prefer a rich, luminous shade like Pearl Mist rather than a cool
metallic one like Pewter Fog, even if the two shades can’t be
distinguished by ABX testing. And of course, the putative “Golden-Eyes”
like artists or housepainters would do no better than average folk
because their background involves no special training in remembering
subtleties of dried paint chips, even if they splash around in wet
paint every day.
I realize of course, that resorting to “analogies” might rile up some
of the more “scientific” types on the list, but perhaps actually
participating in a simple double blind test can illuminate some of the
grey (or is it gray?) areas in such discussions.
More to come.