Hi, Bruce,

I concur with Richard, as usual. Allow me to add a few points from 
someone who has never participated (much less designed or evaluated) an 
ABX test (but has casually been part of many a blind test, not to 
mention blind wine tastings ;-)). This is probably my last posting on 
the subject, as I fear that some seem to hold rather religious beliefs 
on either side of the issue.

1. All tests have limitations. I have no doubt that some use ABX testing 
as "a convenient rhetorical device to dismiss an opinion with which one 
disagrees", but it seems that it may have value for some of us. Even 
within that framework, however, there are ABX tests better designed than 
others. For example, the original ABX designers suggest showing a more 
exaggerated effect (say, MP3-encode something five times serially) to 
"train" those who are not "golden-eared". I guess I intrinsically trust 
the idea, but I am sure it can be perfected. Is there a better test out 
there? One that can reliably show that the average listener *can* tell 
the difference between 16 bit and 24-bit encoded audio? There very well 
may be.

2. Engineer members of ARSCList (as opposed to an audiophile list) are 
more concerned with fidelity rather than the perception of sound. Thus 
we are more worried about whether a process or a piece of equipment 
changes the sound at all. To do this, we do A to B comparisons 
constantly, but ideally we would like not to be influenced by other 
factors. For example, I think most audio engineers have had the 
following experience: you are equalising a channel, turn the knob(s), 
think "ah... here we go", and then realise the EQ is not engaged. Can we 
accept that the mind can play tricks on you? Then it seems a 
well-designed blind test is helpful. But perhaps it is not for everyone: 
As I said in an earlier post, seeing an expensive amplifier does change 
the perception of its sound, much as placebos can cure people --actually 
cure them; and people who hear such differences or are cured by placebos 
are no more nor less honest, gullible, or malicious than those who do not.

3. I very much enjoyed your "shades of paint" scenario, but now allow me 
to propose a different one. A salesman comes to my house with a bag of 
exotic crystals that he swears will make a my recordings sound closer to 
the original when placed in a corner. If I fail to hear the differenceof 
crystals vs. no crystals, could I be blamed for not buying them if my 
budget is limited? That is all some of us are trying to do: trying to 
figure out what to spend our money and resources on --and it seems the 
weight of the proof should be on the positive hypothesis. (Also, I would 
submit that if I painted my mother's house with Pewter Fog instead of 
Pearl Mist, she would not be able to tell the difference).



> Go to a paint store. As we want to avoid “coloration”, go to the rack 
> of “Neutral” paint patch strips, and choose one where the differences 
> in tint, hue, and saturation are minimal. (Of course, to be truly 
> ”blind”, you can have someone else do all this). While the clerk is 
> distracted, swipe 11 strips. After you get past mall security, pick 
> the two patches that are the closest. Let’s say they are called Pewter 
> Fog and Pearl Mist. Cut out the patches (eliminating the names) and 
> label the backs of Pewter Fog A and the Pearl Mists B. Cover 10 sets 
> of A and B labels with opaque but removable tape. Set aside the 
> remaining pair labeled A and B, and dump the rest into a bowl deep 
> enough that the contents cannot be seen or counted.
> Place the two labeled A/B patches in front of the subject, label side 
> up. Place one empty deep bowl labeled A to the left, and another 
> labeled B to the right. Put the bowl of taped samples where they can 
> be reached but not seen.
> The subject (it can be yourself) can turn over A and examine it but 
> must replace it face down to look at B, and vice-versa. Placing both 
> face down, a sample is drawn from the bowl, and the subject must 
> decide if it is identical to A or B, and place it in the appropriate 
> bowl. The process is continued until all the samples are sorted. You 
> can then peel back the tape and see how many were correctly ID’ed and 
> sorted. Any more than 10 correct and you are better than chance. Score 
> 20 and you can be referred to on this list as “Golden-Eyed”.
> While this seems a pleasant enough parlor game, it would not cut the 
> mustard in even an intro course in Experimental Psych. The result 
> above is just a single data point; one needs another variable for the 
> test to have meaning. Repeat the test several times to see if the 
> skill can be learned. Test 100 people and sort by age, gender, 
> religion, etc. Change the lighting from incandescent to fluorescent. 
> Compare putative experts (artists, house painters) to the overall 
> population. Choose different patches and correlate to the pigment 
> differences. Etc. And of course, you can have lots of fun twisting the 
> statistics, but we won’t go there right now.
> If very few subjects can reliably tell the difference between Pearl 
> Mist and Pewter Fog, they are perceptually identical, according to the 
> statistics typically applied. If the new CEO of Benjamin Moore was 
> hired from Crayola, he may decide to "dumb down" the variety of colors 
> available and eliminate one (or both) colors from the line in order to 
> impress the stockholders with his bold vision. Of course, the test 
> actually tells us nothing about the qualities of paint that determine 
> consumer preference – like durability, fade resistance, hiding power, 
> etc. Or which shade your mother would choose for the dining room. And 
> the marketing dept would never let such decisions be made “blind”, 
> anyway. The use of evocative names acknowledges that some buyers would 
> prefer a rich, luminous shade like Pearl Mist rather than a cool 
> metallic one like Pewter Fog, even if the two shades can’t be 
> distinguished by ABX testing. And of course, the putative 
> “Golden-Eyes” like artists or housepainters would do no better than 
> average folk because their background involves no special training in 
> remembering subtleties of dried paint chips, even if they splash 
> around in wet paint every day.
> I realize of course, that resorting to “analogies” might rile up some 
> of the more “scientific” types on the list, but perhaps actually 
> participating in a simple double blind test can illuminate some of the 
> grey (or is it gray?) areas in such discussions.
> More to come.
> Bruce