« Previous | Main | Next »

Test Analysis

Little Hover Commission 2004

Regulation of Acupuncture: A Complementary Therapy Framework

For this study, a comprehensive review and psychometric analysis of the state and the NCCAOM examinations was conducted by consultants from California State University, Sacramento and RAND. The analysis found that both tests meet professional psychometric standards. However, the California examination was determined to be more rigorous than the NCCAOM examination. The analysis also found that the test items on the California examination are on average more difficult than the NCCAOM examination. Further, California's underlying documentation - upon which examinations are built - was stronger. The chart on the following page provides a comparison of the two exams.101 49

Evaluation Summary of the State and National Examination Programs
Portion of Examination ProcessCaliforniaNCCAOM
Occupational AnalysisForm expert panelsConsistent with professional standardsConsistent with professional standards
Identifying tasksConsistent with professional standardsConsistent with professional standards
Identify knowledge elements underlying the tasksConsistent with professional standardsNot explicitly documented
Organizing the job contentConsistent with professional standardsConsistent with professional standards
Sampling respondents for job surveyConsistent with professional standardsConsistent with professional standards
Statistical analysis of important job componentsAcceptable but less than best practiceConsistent with professional standards
Reliability of the job analysis questionnaireRater reliability: not computedRater reliability: very high
Internal consistency: very high for tasks; very high for knowledge elementsInternal consistency: very high for tasks (no knowledge elements were on the survey)
Developing the examWeighting of the content areasConsistent with professional standardsConsistent with professional standards
Crafting itemsConsistent with professional standardsNot documented
Administering the exam Consistent with professional standardsConsistent with professional standards
Quality assessmentTest reliabilityConsistent with professional standardsConsistent with professional standards
Item difficulty levelsModerateRelatively easy
Point-biserial correlationsConsistent with professional standardsConsistent with professional standards
Comparability of language versionsSeems likelyNot documented
Pass point setting Process usedConsistent with professional standardsConsistent with professional standards
Passing rateAround 55%75%+ except for lower Korean pass rate
Maintaining same minimal competency standardsYesNot documented
Source: Lawrence S. Meyers, Ph.D., Department of Psychology, California State University, Sacramento. June 21, 2004. "The Acupuncture Regulation Project: Evaluations and Comparison of California's License Examination and NCCAOM's Certification Examinations. Report to the Commission.

Among the issues analyzed were the following:

Meeting standards of exam development and security. One concern raised in the course of the study, and allayed by the consultant's analysis, was that only the national examination has met external quality review standards and the California exam has not. This is of particular importance because of long-standing concerns that both examinations are subject to corruption.102 To determine whether an examination meets professional standards, externally validated criteria and constant refinements to examination security are needed. California's examination is developed by the State's Office of Examination Resources within the Department of Consumer Affairs using what are nationally recognized as the highest professional standards. Similarly, the NCCAOM examination has maintained accreditation by the National Organization for Competency Assurance as meeting nationally-recognized high standards for validity, reliability and other factors.103 Both exams were found to meet high standards of security, but it was noted that this issue is one that requires constant vigilance for all professions because incentives for corruption are high.

Examination difficulty. Some California professionals raised the concern that the NCCAOM examination is easier. This has been confirmed by the independent consultant, who found higher passing rates and easier questions on the national exam. However, the goal of the examination is to test for competency to practice safely, not to exclude competent practitioners. So the question of relative difficulty of examinations is not the central issue, but rather whether an examination has sufficient rigor to demonstrate minimum competency for safe practice. Notably, other states that use the NCCAOM examination are not known to be experiencing safety problems at a greater rate than California.



Also On This Topic

Managed Hosting by BlueCliffHosting.com