The problem with evaluations

March 28, 2021
 by Paul McGowan

What do you call the lowest-performing student graduating medical school?

A doctor.

In any field, the range from good to great is all over the map. Graduation degrees, specifications, and even reviews only tell us so much.

A power amplifier meeting all the basic requirements of distortion, frequency response, and power output does not—can not—sound the same as a different design with identical specifications.

It’s why we interview our medical providers.

It’s why we read the first chapter of a best seller before committing to the whole.

It’s why we listen to our amplifiers.

Specs, degrees, and reviews are fine for clearing away the cruft of the unworthy.

The rest is up to you.

Subscribe to Paul's Posts

40 comments on “The problem with evaluations”

    1. An interesting group review. And as a triskaidekaphile, among other things, I heartily approve of Simon Price’s choice of names for his YouTube channel: 13th Note HiFi Reviews. Not being an expert on geographic accents, do you know where he hangs his hat? My ears suggest that he would make a good technology reporter for your ABC TV news stream (via YouTube, yet again).

  1. In our system getting the basic medical qualification is the start, not the end. You become a member of the Royal College of Physicians, that has been the regulator for medical practise for over 500 years. Some doctors go straight into primary care as general physicians and others spend many years becoming specialist consultants in narrow fields. It would be quite normal even for a specialist consultant, having identified a specific issue, to refer to a professional colleague with better or sometimes unique experience of particular conditions, often just for a second opinion.

    I was never aware that in the USA, as an individual, you have to interview and choose your medical provider. That sounds utterly horrifying. What criteria can you use? It’s not like you can take them for a demo and if you don’t like the result chose another one.

    We have no choice. We call the surgery, perhaps 5 to 10 doctors, and one of them will give primary advice and dispense medicine. We now do this on Zoom and by email. The doctor will make any referral to a specialist of their choice, based on medical need.

    If there is any comparison, it is that my audio dealer saves me wasting a lot of time listening to endless products, I tell him what I want and he can diagnose the right equipment.

    I suppose if you interview a doctor you may decide based on whether you like their personality – you could call it their brand value. All I’m concerned about is their medical expertise, I want to trust them, not like them, I want good medical advice, I won’t be asking them round for dinner. I take the same approach with audio.

    1. Steven,

      The qualifications to become a MD are a minimum standard. To assume you’ll be referred to a leader or specialist in their chosen field is just naive. It’s incumbent upon the patient to seek out who they feel most comfortable with, and who gives them the highest chance of curing their ailment.

      Socialized medicine may be the greatest thing from your perspective, but not all of us.

      You’re last statement hits the nail square on the head.

      According to you, your system relies on referrals and then you’re in a position to choose. How is that any different than what you perceive the US system to be?

      1. No, we don’t choose. Our general practitioner makes a referral to the specialist and in the unlikely event you are not happy with the advice you can go back to your general practitioner and ask for a referral for a second opinion. Consultants work in teams and often discuss cases as a team anyway.

        I’m 57 and no one in my family has ever chosen who to be treated by. For standard procedures there are hundreds of competent specialists, but if you have something extremely rare and there is one expert, who may be the best in the world, that person will treat you. Clinical decisions and treatment are made solely on the basis of need.

        I have private insurance was well. It’s pretty cheap ($4,000 for a family of 4). It’s only really for non-urgent things, you sometimes get treated quicker, you will get a private room rather than a ward, but treated by the same doctor and often in the same hospital. You still go to the same State doctor for the referral.

        The best system is I’ve seen is in Sri Lanka, where you turn up at the hospital and have a choice between Ayurvedic or western medicine. It’s a but like turning up at your audio dealer looking for an amp and saying you prefer valves to solid state, what have you got? If it transpires you have 82dB speakers, the dealer should insist on solid state, just like if you choose the Ayurvedic clinic and need brain surgery, the doctor will refer you to the western clinic.

        It is very easy to establish trust when medical need rather than profit is the only issue. I rely on the advice of an audio dealer, who knows my likes and price level. He doesn’t try and over-sell. I suspect there are some dealers who do or try and sell the products with the biggest profit margin. That’s an easy way to lose the trust of the customer.

        Maybe Paul’s analogy has merit, but not for the reasons he thinks.

        The difference, where it breaks down, is that with audio there is room for trial and error.

        1. Ok Steven. I’ll agree the analogy may not be perfect. But ignoring the two or three statements concerning the medical world, Paul’s premise makes sense to me. Some will agree with it, some won’t.

          Choices for all is not a bad thing. One size doesn’t fit all.
          Different strokes for different folks?

            1. Possibly, Or a quack? That term may be construed as derogatory, but then again as with any profession there are those who excel. Know your limitations and defer to others when necessary.

            2. Off at a tangent but The Beatles LP ‘Yesterday and Today’ with controversial ‘butcher cover’. Anyone got one of those? Rare and collectable.

          1. I agree. I was just a bit overwhelmed about the idea of having to choose a doctor. I’d probably go for the one with the cleanest shoes, as it demonstrates attention to detail.

            The last consultant I saw was for a shoulder injury, lovely guy from Sri Lanka, as it happens, but over 9 months didn’t fix it. That was done by an American acupuncturist in 2 treatments and I was fixed in 10 days. Wonderwoman.

            The answer is that I have a dealer who cares firstly for my audio health, so I trust him, rather than just making money. Doctors require patient consent to perform a procedure and often they advise against. Audio has the advantage if you can arrange a trial or sale/return.

            1. The two most common questions that Americans ask when assessing a physician are:

              1. Do you accept my insurance?

              (If yes then)

              2. Do you take new patients?

    2. If I don’t like my doctor, dentist, mechanic, or fruiterer I’m not going back ever – my emotional connection or lack thereof affects my perception and trust level. Once went to a new town and booked an appointment with one of 3 local doctors. I walked in and saw a hugely overweight and sweaty man who smelt of cigarettes – it was a short and never repeated visit. He was eventually busted for selling Amway in seasons.
      In the 80s I to have a very trusted Hi Fi dealer I used for years. He had a large range, many years experience and knew my tastes and past buying history. After 20 years something happened, he cut his range to 7 or 8 brands and started to push me to buy those ones only. They were great, well respected brands but nothing he had ever turned me on or sat right with me – I moved on to another dealer eventually.
      All this leads me to say that personally while I take the technical side of any thing or discipline into account to quite a large degree, if ‘it’ or ‘they’ don’t gel in other ways (social, emotional, personal) then I fond I cannot fully meet them to make a full commit.

  2. Paul,

    Back to the audio topic of the day. (Not the focus on the analogy)

    You also hit the nail on the head. Audio is an individual choice, based on many individual requirements and perceptions.

    For those in the land O’measurement who base decisions on that fact alone, they will be happy. That is until they take the opportunity to hear differences in sound just by changing, in this case amps, in their set-up and hear a difference. The differences may be small and may require years of training or experience to pick out, but they are there.

    For those in the land O’ears only, they could be quite content not ever giving a damn about the measurements. They may loose an argument with a measurementalist based on the measured facts, but will live on with the assurance that what they hear is wonderful.

    Then there’s a group in the land O’middle who pay attention to it all. Realizing that what they perceive from a sound system is based on many different things.

    We’re back to the floating boat. There’s room for all.

    1. Yes, there’s room for everyone’s boats, and, with the amazing advances in audio over the last 50 years everyone benefits from the R & D.
      “A rising tide lifts all boats”.

  3. I would change the post title to “The Benefits of Evaluation.” When we have choices, evaluations are an opportunity to assure best value, performance and personal satisfaction.

    We start by investigating the field of offerings; we study the specs to be sure minimum qualifications are complied with and even exceeded (degrees, experience, honors, awards, affiliations, technical data); we look at the package (photos or in person); we look at online reviews of customer satisfaction (yes, even individual doctors are rated by customers); we then interview or audition in person; if we like what we see and hear, and feel good about it, we go with it–if not, we can decline and try out another model.

    In medical, most insurance companies even the ones with the most affordable HMO options let us select from dozens, if not hundreds, of primary care physicians within their networks, each of whom routinely refers us as needed to any of hundreds of in-network specialists, which we also have the opportunity to evaluate. My health care provider is UCLA Health which has a huge network of top-rated doctors in all specialties that accommodate most insurance plans. UCLA Health has great online support and automatically sent me an invitation for my COVID-19 shots–now completed. I can email any of my doctors at any time, and get a personal response within 24 hours. It’s like going with PS Audio products. You know everything under that same roof is going to be reliable and top-notch with excellent communication and personal attention. It gives the best chance that evaluations are going to be positive and there will be a coordinated synergy between products and services.

    For over twenty-five years my PCP has been the same, even though I have switched between four different insurance carriers and am now in Medicare Advantage. I typically research the specialists suggested by my PCP and request changes if for any reason I am not comfortable with ones he selected. I scrutinize the specialists as much as I did my PCP. I evaluated my cardiologist long before my first heart attack and angioplasty. I researched and insisted on the most experienced hernia repair surgeon and waited an extra month to get the best.

    Evaluation is a privilege indeed.

  4. “It’s why we read the first chapter of a best seller before committing to the whole.”

    The thing is the first chapter isn’t always the best chapter.
    I don’t know if anyone else has this experience but when listening to a new album I often find the first track is weak song and not representative of the album as a whole. A classic example of this would be The Eagles ‘Hell Freezes Over’. That first track ‘Get Over It’ has got to be the worst track on the album, one of the worst Eagles tracks ever.
    Perhaps that’s what I need to do, get over it.

    1. Interesting observation Richtea. When previewing an album on line I often find the 3rd or 4th tracks are “the dogs”, and if I sorta like them I move around the rest of the tracks. I’ll respectfully disagree about your example. I don’t skip one track on that album.

      To go a little €€€€ OFF TOPiC €€€€
      Do you miss not getting the e-mail notification when someone replies to your comments?

      1. Mad Dogs and Englishmen was a great lil band and short US tour assembled in the 11th hour led by Leon Russell after Joe broke up with his “Grease Band”. The back story is they were all fueled up on MDA that Saturday evening March 28th 1970 at Fillmore East as witnessed by Joe’s banter “Don’t get hung up on Easter” and then proceed to break out a cover of The Beatles “She Came In Through The Bathroom Window”. Good show, check out the DVD.

        I was not there but chances are either Jay Jay French or Genez may have been in attendance.

    2. Sorry to learn that. The DVD opens with Hotel California and the Eagles Hell Freezes Over tour witnessed at Shoreline Amphitheater Mountain View CA in June 1994 was definitely the best Bay Area show of the year by a country mile.

      It was great to see Joe Walsh back on earth with two feet firmly planted on the stage playing his ass off singing great vocal harmonies with Frey & Henley before they mutually tossed Don Felder out of the band again. (insert sad face emoji)

  5. “in any field, the range from good to great is all over the map”
    I’ve seen my share of live music but, since joining this PSA community, I’ve been paying more attention to all things musical lately. Last night my friends and I enjoyed one of our favorite local bands. They are a six piece New Orleans inspired very funky brass band. Their energy is contagious and rhythms have everyone up dancing. During the live “experience” of listening, seeing, and dancing – they sound amazing or to the “map” reference of this post – they sound GREAT! But the morning after, I watch some of the videos I took on my phone and they only sound “GOOD”. And I am not talking about the recording quality. I’m talking about hearing imperfections in their execution that went unnoticed in the heat of the live show. SO I guess I’m trying to articulate how complex our relationships with music can be. Sometimes the imperfections of a GOOD band make the experience more fun and real and human. And somehow more SPECIAL because only those who were there can relate since the reproduction can never do justice to the experience. For these reasons, I prefer studio recordings for critical listening at home. And more often than not, I enjoy live bands that are a bit rough around the edges than the ones that are polished. Call me a “blue collar” audiophile 😉

  6. I guess listening skills could be a medical qualification when dissecting or evaluating a specific piece of equipment. From now on I would call my ears “the Surgeon Generals.” 😉

  7. Paul,

    There are so many issues that I disagree with your point of comparison of medicine and amplifiers.

    You say that you interview medical providers. What assessments do you use to determine what is a good provider? How much do you know about medicine or related sciences to determine if this person is going to “work” with you? What specific questions are you asking this potential provider? How do you assess the answers? Are you going to be as irrational as checking if the shoes are polished? It may be as rational to use that “metric” as stating that a blue light in the amp makes a better sound. Or cable elevators. Most people have no idea or an extremely vague idea of how to choose doctors, besides bedside manner. But if you base everything on bedside manner, you may end up with a charming doctor holding your hand to your grave. Specifics matter, generalities are for pub conversations.

    Amplifiers are literally much easier to choose. As long as the SINAD is reasonably low, let’s say under 90-95dB, are capable of driving your speakers to a target level, the frequency response is flat, even at various impedances, and SINAD is not affected much by power, you will get a transparent amplifier. You may want to send more that $10k for a tube amplifier with horrible specs that are clearly within listening range because you “like” the sound of tubes, but that doesn’t make it a good amp or a transparent amp. You may even want to spend $100k in an amp so you can brag with your friends, but you don’t need to do this as a modern Class D will probably be more transparent. You may want to target specs to the maximum possible level, but a Benchmark will be impossible to tell apart fro a good Class D design. It may appear irrational to chase specs beyond what we can hear but it becomes more expensive.

    Modern well designed amplifiers when auditioned at well controlled levels with proper speakers are impossible to tell apart. It is what we “like” and not what we hear.

    I just want to know what you “like” in doctors.

    By the way, I have worked in the medical field since 1985, so I am familiar with systems in many countries besides the USA. They all have their strengths and weaknesses.

    1. That’s hilarious when you write “there are so many issues I disagree with”. Me too! Luckily that’s what makes it fun to have a conversation with people.

      I am certain your questions when you interview doctors (if you do) would be very different than mine. I interview doctors in the same way I interview potential team members for PS Audio. I look for brightness and intelligence. I find I have quite a knack at sussing it out in people – but every personality is different so the questions I ask have to fit the situation and the person.

      As you point out, I have no medical training so my questions do not involve much of that. I want someone who is bright and can independently think outside the box. Further, it really depends on what I am there for. If I go to my local doc in the box urgent care for a fever I am not that interested. I just need a competent provider and I trust the place I go to to provide that for me.

      When I went through my bout with a Gleason’s 9 case of prostate cancer you can bet I interviewed more than my fair share of surgeons and radiologists. It took me nearly three months of interviews to find the sharpest tool in the pack, but I did and the results worked out fine. Those interviews involved my intelligence test along with their track record of results after surgery. If they didn’t know their track record (many surgeons don’t have much in the way of a follow up with patients past a few months), or if their track record for zero leakage after 6 months was less than 95% they were rejected.

      1. G9 is pretty scary. I hope it all went well and that follow up remains uneventful. It is for those tough situations when a systematic and intense assessment is worth it. Especially with surgeons who are very poor in following up with patients. Importantly, they are also quite poor in “controlling” their studies. They don’t like to do “sham” surgeries or randomizations in appropriate types.
        I wish you well.

        1. Thanks. It’s coming up on 1 year now and I am back to “normal”. The surgeon was extremely skillful and saved all the nerves as well as got all the little nasties out of there. PSA score zero! Thanks for the kind words.

          1. I am pleased for you!

            You will probably not be surprised to know that the majority of the population does not know how to assess physicians and how poor is their science understanding. Also, in many cases, physicians are very reluctant to talk to patients. Nowadays, many show up with reams of Google data and think that this is enough. It becomes very difficult for the relationship. People have a very hard time figuring out what is proper scientific information and what is quackery online. I wish they taught kids how to assess sources of information.

            Your 5-10 year prognosis should be excellent. As a good friend of mine who researched this field used to tell me, if you do post mortem on all old men, you will find that they all had some kind of prostate cancer. But a G9 is not a play thing. Just don’t get run over while e-biking….

  8. Good points Paul. The room and the recording quality are huge variables. Facts and specs can’t reveal the effects of a bright apartment nor a compressed recording.

    Also, written reviews cannot either, but specs, facts, and reviews can narrow choices down. Trying components at home is the best way. Coodos to companies like PS which offer in-home trial.

  9. Very true and the only way to get the right system. This advice has been given for years but anyone who has little or no experience with how different sound systems sound will have to listen to someone else usually a dealer or a friend a reviewer etc. Quite natural. Of course if such a person,over time becomes a critical listener the the original system will be history as is the case mostly.Money can be a deterrent unfortunately.Regards.

  10. “Specs, degrees, and reviews are fine for clearing away the cruft of the unworthy”
    To a large degree yes, but, your mileage may vary because of factors outside those parameters.
    Good start though.

  11. Love the analogy. Folks that is an old adage. Oldie butt a goodie. The “audiophiles” who live by specs probably look at the ingredients of food to decide whether it tastes good or not. There’s another old adage..”beauty is in ear of the belistner”. Music is tactile. For many of us the greatest moments in music was in a car with a AM radio played over a metal speaker plopped in the dash. Baffling. And static..oh god…lots of beautiful static.

Leave a Reply

Stop by for a tour:
Mon-Fri, 8:30am-5pm MST

4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

Stop by for a tour:
4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram