Word to the Wise, All CPA Exam Questions Are Not Created Equal

As we all know, CPA exam advice is a dime a dozen these days. There are handfuls of sketchy blogs spitting out generic - and sometimes completely wrong - advice that appears to be written by English-challenged spambots, new CPA review programs popping up all the time and all manner of "resources" that are really no more than transparent attempts to get reciprocal backlinks from the resources mentioned.

In my eternal quest to piss in other people's Cheerios in the most bitter way possible, I came across yet another "advice" article packed with tips and tricks to tackle the CPA exam. Most of the tips are spot on; spend about 500 hours total studying for all 4 parts of the exam, budget your time, know what kind of ID you need to bring to the test center... all great tips. I did take issue with suggesting one could theoretically skip the written communication in BEC assuming one does exceptionally well on the MCQ in that section but the AICPA admits it is technically possible so we'll let that one go. But seriously, don't do that. Even if you are a horrible writer, it doesn't take much to do well in the written communication section assuming you can write at a 7th grade level and know where to put the right punctuation.

But then there's this:

Cherry-Picking is Acceptable: There’s nothing wrong with grabbing that low-hanging fruit first. Within every testlet, answer easy questions first and come back to the more difficult ones later. They are all worth the same amount of points anyway, and this plan of attack may just bolster your confidence. This technique can be particularly helpful in the TBS testlets. Scan quickly through each of the 6 or 7 simulations and decide which ones you can knock out of the park quickly and effectively.

Now, on its face this strategy is a good suggestion. Instead of getting stuck on one particularly difficult question, it's a better idea to skip it, move on to easier questions and return before you close out the testlet. However, questions are most definitely not all worth the same amount of points. This is completely wrong. In fact, harder questions are worth more than medium difficulty questions.

As we all know, everyone starts out with a medium difficulty testlet. If you perform well on that first one, your next testlet will be considered difficult. The good news is that because difficult questions have a higher point value, you don't have to answer as many correctly to pass as you would medium difficulty questions. But even a "difficult" testlet contains both medium and difficult questions as each question is not quantified as a category ("moderate" or "difficult") but as a numeric value along a scale. A testlet is considered medium or difficult based on the average difficulty of the questions within that testlet. Make sense?

If all questions were worth the same amount of points, it would be easier to ballpark your score at the end of your exam based on the ones you thought you got right. But we all know that there's also some magical AICPA formula thrown in there that basically means candidates who get all medium difficulty testlets, a medium and two difficults or two mediums are scored fairly and comparably.

From the AICPA's own mouth [PDF]:

Does that mean difficult testlets can have easier questions and medium testlets can have difficult questions? Yes. All testlets have questions ranging in difficulty. Questions in difficult testlets just have a higher average level of difficulty than those in medium testlets.

What do you mean when you say “statistical characteristics” of test questions? There are three statistics used to describe the questions: Difficulty – whether the question is generally easier or more difficult for candidates, Discrimination – how well the question differentiates between more able and less able candidates, and Guessing – the chances of candidates answering the question correctly just by guessing. The statistics are generated when the questions are administered as pretest questions and used in the scoring when the questions are operational. The formulas for generating the statistics and scoring the Exam come from a scoring approach commonly referred to as ―Item Response Theory.‖ Item Response Theory is currently being used or has recently been adopted by nearly all of the large licensing examination programs in this country and also by many of the moderate-sized and smaller examination programs.

Can two candidates answer the same number of questions correctly, but get different scores? Yes, because scores depend on the characteristics of the questions, not just how many are answered correctly. For example, all else being equal, a candidate who correctly answers 10 difficult questions would score more than a candidate who correctly answers 10 easy questions.

The lesson here, as always, is to familiarize yourself with the official material from the AICPA before you start basing your test-taking strategy on things you've read on the Internet (even here), things your colleagues have told you, things your cousin's mother's babysitter's doctor's friend said... If you know how it actually works, you can take advice articles with a grain of salt or at least separate the good advice from the bad. Sadly, there seems to be a lot more bad out there these days than good. Don't let that scare you too much as there are also great non-AICPA resources out there but exercise a little professional skepticism when wading through all that CPA exam info floating around out there in the big scary Internets.

Comments