Computerized Testing YouTube Lecture Handouts Computer Based, Computer Assisted & Computer Adaptive Testing for NTA (UGC)-NET 2019

Download PDF of This Page (Size: 153K)

Get video tutorial on: https://www.YouTube.com/c/Examrace

Watch Video Lecture on YouTube: Computer Based Evaluation - Computer Based, Computer Assisted & Computer Adaptive Testing

Computer Based Evaluation - Computer Based, Computer Assisted & Computer Adaptive Testing

Loading Video
Watch this video on YouTube

Computerized Test

  • Computer Based Testing

  • Computer Assisted Testing

  • Computer Adaptive Testing

  • Computerized tests are now one of the most important ways to judge learning, and, selecting tailored questions for each learner is a significant part of such tests. The basic notion of an adaptive test is to mimic what a wise examiner would normally do.

  • Computer-assisted testing is an assessment model in which learners answer questions or complete exercises that are part of a computer program. In many cases, computer tests also include automatic scoring. Computer-assisted testing is used for assessment in E-learning, standardized tests, for psychological and skill assessment in classrooms, and may even be used by individuals who wish to test themselves.

  • This testing model is especially common on college campuses where students have the option to take online classes.

  • Those who are preparing to take standardized tests, for example, might use practice test software for self-assessment.

  • Computer-assisted testing is the use of computers to support assessment and testing processes. Computer-assisted testing began in the early 1950s when optical scanners were adapted to read special answer sheets and score tests. Thus, in addition to scoring tests, computers began to interpret test scores and analyze test data.

  • Examples are Minnesota Multiphasic Personality Inventory and the Strong Interest Inventory. CAI in the 1960s and 1970s consisted of computers functioning as “page turners,” with very basic branching logic to support the instructional process.

  • A screen was presented to the student, the student made a response, and rudimentary computer software determined the next screen to present to the student. Computer-based testing, using the same page-turning approach, was a natural result of this process.

  • Computerized adaptive testing (CAT) is a form of computer-based test that adapts to the examinee's ability level.

Why Computerized Test?

  • varied learners with different abilities are involved, a simple test cannot serve the purpose of assessment.

  • test items should mimic the ability; a test item bank should be ready to satisfy all possible constraints.

Types of Computer Assisted Approach

  • Conventional Testing

  • Branched or Response-Contingent Testing

  • Partially Adaptive Testing

  • Fully Adaptive Testing

  • Sequential Testing

    • Conventional Testing:

      The simplest application of computers in test delivery is the administration of conventional tests in which all examinees receive the same test questions in the same order, usually a question at a time. Although this seems like a trivial advance over paper-and-pencil tests, it has a number of advantages. First, all instructions are presented by computer, prior to the examinee receiving the test questions, typically along with some practice questions. This insures that each examinee has read and understood the instructions. Second, scores can be made available to the examinee or test administrator immediately after completion of the test. Furthermore, all examinee responses are recorded electronically, thus eliminating the need to optically scan test answer sheets.

    • Branched or Response-Contingent Testing:

      Branched or response-contingent testing is useful in measuring variables that can be evaluated through a problem-solving scenario or sequence of steps. In this approach, a problem situation is presented to the examinee with a number of alternatives. Each alternative “branches” to a different second stage in the problem-solving process. Subsequent branches for each subsequent question continue to lead to different changes in the situation presented to the examinee. As a consequence, each examinee can follow a different pathway through the problem-solving process, some of which lead to an appropriate solution to the problem whereas others do not.

    • Partially Adaptive Testing:

      Partially adaptive tests operate from a bank of questions that is structured by difficulty. The simplest of these tests consists of subsets of questions grouped into short tests, or testlets, comprising questions of differing average difficulty levels. A testlet of medium difficulty is administered, one question at a time, and immediately scored. Examinees who score high on the testlet then receive a more difficult testlet. Those who score low are then administered an easier testlet.

    • Fully Adaptive Testing:

      Fully adaptive testing, based on a family of mathematical models called item response theory (IRT), is currently the most used approach to adaptive testing.

    • Sequential Testing:

      Sequential tests are typically used to make a classification decision (e.g., to hire or not to hire, to graduate or not to graduate, or whether someone is or is not depressed) using one or more prespecified cutoff scores. Typically, the questions in the test are ranked in order of how much precision they contribute to making the classification decision.

Benefits of CBT

  • Multiple-Test Administrations

  • Dynamic And Individualized Assessments

  • Immediate Grading

  • Helps With Open-Ended Assessments

  • Feedback: Voice feedback tools, like Kaizena, allows instructors to provide voice feedback

  • Vertically And Horizontally Aligned Assessments

  • Value-Added Growth Measures

  • Uncover Student Thinking

  • Engaging

  • Analytics For The Instructor And Learner

  • Greater Amount Of Test Items

  • Help Learners With Disabilities

  • Incorporate Other Types Of Technology

  • Improves Writing

  • Can Secure Testing

Fully Adaptive Test – Characteristics

  • A fully adaptive computerized adaptive test (CAT) has the five following requirements and characteristics:

  • It uses a question bank in which all questions have been calibrated by an appropriate IRT model. The IRT family includes models for questions that are scored in two categories (e.g., multiple choice scored as correct or incorrect, true or false, yes or no) and rating scale questions that are scored as multiple categories.

  • Preexisting information about each examinee (e.g., his or her school grade) can be used as a starting point for selecting questions.

  • Questions are administered one at a time, and the examinee’s score is estimated after each question is answered.

  • After each question is administered, the entire question bank is searched and the question that will provide the most precise measurement of that examinee (given the examinee’s score at that point in the test) is selected for administration.

  • This process of selecting and administering a question and rescoring is repeated until a suitable termination criterion is reached. Fully adaptive CATs can be terminated when the examinee’s score reaches a prespecified level of precision, when there are no more useful questions in the bank for measuring a given examinee, or when the examinee has been reliably classified with respect to one or more cutting scores.

Important Approach

  • The critical and foremost important component in composing computerized intelligent testing is test item selection. Though many approaches were devised to select proper test items to compose intelligent test sheet

  • The test sheet parameters (Test Sheet Difficulty and Test Sheet Discrimination) need to be optimized with the expected objective values while framing the Test Sheet.

  • multiple criteria like concepts to be covered in the test, its proportion, association between test item and concept, number of test items and exposure frequency needs to be satisfied while generating the Test Sheet

  • the questions in a test are randomly selected from the item bank or selected based on a simplified optimization model.

Personalization in Computer Adaptive Tests

  • Normal test cannot always satisfy the need in discriminating the learner‘s knowledge

  • Each learner has different learning status and hence different items should be used for precise evaluation

  • Adaptive Item sequencing is needed (Extreme difficult items may frustrate learners and extreme easy items may lack any sense of challenge)

  • To provide a platform for evaluating students‘ learning status and compose high quality test because quality of test items significantly affects accuracy of the test

  • Key to the quality test depends not only on subjective appropriateness of items, but also on the way the test sheet is constructed

  • Well-constructed test sheet not only helps evaluation of learning status, but also facilitates diagnosis of any problems within learning process

  • Test item attributes are hard to be controlled

  • In E-Learning, it is necessary to conduct long-term continuous assessment of each learner and hence, series of self-adaptive and reliable test sheets with multipart criteria needs to be composed

  • Accurate and efficient evaluation of trait level of examinees

Method in CAT

  • CAT successively selects questions for the purpose of maximizing the precision of the exam based on what is known about the examinee from previous questions. From the examinee's perspective, the difficulty of the exam seems to tailor itself to their level of ability. If an examinee performs well on an item of intermediate difficulty, they will then be presented with a more difficult question. Or, if they performed poorly, they would be presented with a simpler question. Compared to static multiple choice tests that nearly everyone has experienced, with a fixed set of items administered to all examinees, CAT require fewer test items to arrive at equally accurate scores.

  • The basic CAT method is an iterative algorithm with the following steps [Alan, 2012].

  • The pool of available items is searched for the optimal item, based on the current estimate of the examinee's ability

  • The chosen item is presented to the examinee, who then answers it correctly or incorrectly

  • The ability estimate is updated, based upon all prior answers

  • Steps 1–3 are repeated until a termination criterion is met

  • Nothing is known about the examinee prior to the administration of the first item, so the algorithm is generally started by selecting an item of medium, or medium easy, difficulty as the first item. As a result of adaptive administration, different examinees receive quite different tests [Leung, 2003]. The psychometric technology that allows equitable scores to be computed across different sets of items is item response theory (IRT). IRT is also the preferred methodology for selecting optimal items which are typically selected on the basis of information rather than difficulty [Alan 2012].

CAT Models

  • A Computer adaptive testing application is a system that presents questions to a student and infers the student’s knowledge level from his answers.

  • Item Response Theory (IRT): In psychometrics, item response theory is also known as latent trait theory, strong true score theory, or modern mental test theory, is a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables. It is based on the application of related mathematical models to testing data. The name item response theory is due to the focus of the theory on the item, as opposed to the test-level focus of classical test theory, by modeling the response of an examinee of given ability to each item in the test. In this approach the probability that the student answers correctly to a question depends on the student’s knowledge level.

  • The higher the student’s knowledge level, the bigger the probability of correct response to a question is. Knowing the shape of the probability distribution of the questions, it is possible to predict the user’s knowledge level, given the set of responses to the previous questions.

  • In IRT model the answer given by a student to a question can be probabilistically predicted given the student’s knowledge level and the item characteristic curve of the question. The Item Characteristic Curve (ICC) of a question expresses the probability that a student answers correctly to the question given the student’s knowledge level.

  • Bayesian Network Model: is directed acyclic graph whose nodes represent variables and links represent the dependence relationships between the variables. In BN, the probability that a student answers correctly to a question is conditioned to the student’s responses to the previous questions.

  • Sequential Probability Ratio Test Model: In theory, CAT can dramatically reduce the testing time while maintaining the quality of measurement as compared to the fixed-item type of tests in either pencil-and-paper or CBT format

CAT Components

  • Among the CAT components, estimating the initial ability of the learner as a starting point and the test item selection strategies are the foremost important components.

  • Learner’s ability

  • Test item selection strategy

  • Scoring procedure

  • Termination criteria

Multiple Criteria

  • Item difficulty

  • Item differentiation

  • Association between test item and concepts

  • Concept weightage

  • Exposure frequency

  • Completion time

  • Type of test item

  • Type of test sheet

  • Serial test sheets

Issues in CAT

  • Item pool size and control

  • Item removal and revision

  • Item response model

  • Adding item to item pools

  • Item selection strategy

  • Using multiple item pool

  • Test entry procedure

  • Test termination criteria - when the predefined amount of information about proficiency is achieved.

Intelligent Water Drops Algorithm

  • Metaheuristics especially nature-inspired swarm-based optimization algorithms are being increasingly used for solving optimization problems. Naturally, the IWD algorithm is appropriate for combinatorial optimization problems.

  • If we put ourselves in place of a water drop flowing in a river, we would feel that some force pulls us toward itself, which is the earth’s gravity. This gravitational force pulls everything toward the center of the earth in a straight line. Therefore with no obstacles and barriers, the water drops should follow a straight path toward the destination, which is the shortest path from the source of water drops to the destination, which is ideally the earth’s center.

  • The real path is so different from the ideal path such that lots of twists and turns in a river path are seen, and the destination is not the earth’s center but a lake, sea, or even a bigger river.

  • Prefer an easier path than a harder path

Get unlimited access to the best preparation resource for UGC - Get detailed illustrated notes covering entire syllabus: point-by-point for high retention.

Developed by: