Test Analysis
Little Hover Commission 2004
Regulation of Acupuncture: A Complementary Therapy Framework
For this study, a comprehensive review and psychometric analysis of the state and the NCCAOM examinations was conducted by consultants from California State University, Sacramento and RAND. The analysis found that both tests meet professional psychometric standards. However, the California examination was determined to be more rigorous than the NCCAOM examination. The analysis also found that the test items on the California examination are on average more difficult than the NCCAOM examination. Further, California's underlying documentation - upon which examinations are built - was stronger. The chart on the following page provides a comparison of the two exams.101 49
| Evaluation Summary of the State and National Examination Programs | |||
|---|---|---|---|
| Portion of Examination Process | California | NCCAOM | |
| Occupational Analysis | Form expert panels | Consistent with professional standards | Consistent with professional standards |
| Identifying tasks | Consistent with professional standards | Consistent with professional standards | |
| Identify knowledge elements underlying the tasks | Consistent with professional standards | Not explicitly documented | |
| Organizing the job content | Consistent with professional standards | Consistent with professional standards | |
| Sampling respondents for job survey | Consistent with professional standards | Consistent with professional standards | |
| Statistical analysis of important job components | Acceptable but less than best practice | Consistent with professional standards | |
| Reliability of the job analysis questionnaire | Rater reliability: not computed | Rater reliability: very high | |
| Internal consistency: very high for tasks; very high for knowledge elements | Internal consistency: very high for tasks (no knowledge elements were on the survey) | ||
| Developing the exam | Weighting of the content areas | Consistent with professional standards | Consistent with professional standards |
| Crafting items | Consistent with professional standards | Not documented | |
| Administering the exam | Consistent with professional standards | Consistent with professional standards | |
| Quality assessment | Test reliability | Consistent with professional standards | Consistent with professional standards |
| Item difficulty levels | Moderate | Relatively easy | |
| Point-biserial correlations | Consistent with professional standards | Consistent with professional standards | |
| Comparability of language versions | Seems likely | Not documented | |
| Pass point setting | Process used | Consistent with professional standards | Consistent with professional standards |
| Passing rate | Around 55% | 75%+ except for lower Korean pass rate | |
| Maintaining same minimal competency standards | Yes | Not documented | |
| Source: Lawrence S. Meyers, Ph.D., Department of Psychology, California State University, Sacramento. June 21, 2004. "The Acupuncture Regulation Project: Evaluations and Comparison of California's License Examination and NCCAOM's Certification Examinations. Report to the Commission. | |||
Among the issues analyzed were the following:
Meeting standards of exam development and security. One concern raised in the course of the study, and allayed by the consultant's analysis, was that only the national examination has met external quality review standards and the California exam has not. This is of particular importance because of long-standing concerns that both examinations are subject to corruption.102 To determine whether an examination meets professional standards, externally validated criteria and constant refinements to examination security are needed. California's examination is developed by the State's Office of Examination Resources within the Department of Consumer Affairs using what are nationally recognized as the highest professional standards. Similarly, the NCCAOM examination has maintained accreditation by the National Organization for Competency Assurance as meeting nationally-recognized high standards for validity, reliability and other factors.103 Both exams were found to meet high standards of security, but it was noted that this issue is one that requires constant vigilance for all professions because incentives for corruption are high.
Examination difficulty. Some California professionals raised the concern that the NCCAOM examination is easier. This has been confirmed by the independent consultant, who found higher passing rates and easier questions on the national exam. However, the goal of the examination is to test for competency to practice safely, not to exclude competent practitioners. So the question of relative difficulty of examinations is not the central issue, but rather whether an examination has sufficient rigor to demonstrate minimum competency for safe practice. Notably, other states that use the NCCAOM examination are not known to be experiencing safety problems at a greater rate than California.