ELA ITEM TYPES Item types Assessing item quality

ELA ITEM TYPES Item types Assessing item quality

ELA ITEM TYPES Item types Assessing item quality 4 BUILDING BLOCKS Learning Progressio n Item Design Assessme nt Quality Outcomes

/Scoring ITEM TYPES Classification of item types: Objective Performance based Item types (SBAC, 2012, pp. 27 32): Selected response (SR) Constructed response (CR) Extended response (ER) Technology-enhanced item (TE) Performance task (PT) CLASSIFICATIONS OF ITEMS Objective Selected Response : True/False Multiple choice Constructed (restricted )

response: Short answer Technology-enhanced Subjective/ Performance based Extended response: Essay Performance task: Group/individual Project/portfolio Class participation Peer assessment SELECTED RESPONSE (SR) Contain a set of options from which to select correct response(s) Checklist: Does the item assess an important aspect of the standard?

Does the stem ask a direct question or set of specific problem? Is the item based on a paraphrase rather than words lifted directly from a textbook? Are the vocabulary and sentence structure at a relatively low and nontechnical level? SELECTED RESPONSE (SR) Checklist (continued): Is each alternative (stem) plausible so that a student who lacks of knowledge of the correct answer cannot view it as absurd or silly? If possible, is every incorrect alternatives based on a common student error or misconception? Is the correct answer of the item independent of the correct answers of the other items? Is there only one correct or best answer to the item?

SR: MULTIPLE CHOICE Advantages Disadvantages Dos and Donts Ease and objectivity in scoring Can assess a number of learning targets Focus on reading and thinking Less chance of randomly guessing answers than in T/F items Incorrect response(s) may indicate

misconceptions May only assess lowerlevel thinking, if poorly written Discourage students from expressing own solutions Limited sampling of content Difficult to construct distinct and meaningful incorrect responses Write the question or task clearly by focusing it on specific learning target(s) Write the correct answer along with concise and unambiguous incorrect

responses Write plausible incorrect responses (i.e. not obviously wrong responses) Avoid using all/none of the above or a combination of correct answers as alternatives Avoid using negative or double negative statements EXAMPLE 1: POOR AND BETTER Read this sentence from the article: Rainfall data from ancient cypress trees shows that the region's worst drought in 800 years peaked in 1587, the year the 120 men, women and children of the Roanoke colony were last seen by Europeans. What does peaked mean in this sentence? Poor: A. was sharp

B. was at its height C. was wet D. was dry Wrong answers are too obviously wrong Better: A. was sharp B. was at its height C. was mountainous D. was rising

Distractor s are more plausible. EXAMPLE 2: POOR AND BETTER Poor Which of the following is least dissimilar in meaning to concur ? Double A. misalign negatives B. assent C. withdraw D. sanitize Better Which of the following is most similar in meaning to concur ? A. misalign

B. assent C. withdraw D. sanitize CONSTRUCTED RESPONSE (CR) Require student to generate a response in one or more words Checklist: Does the item assess an important aspect of the standard? Is the item based on a paraphrase rather than sentence copied from a book? Is the item worded clearly so that the correct answer is a brief phrase, single word of single number? CONSTRUCTED RESPONSE (CR) Checklist, continued: Is the blank or answer space toward the end of

the sentence? Are there only one or two blanks? If the item is in the completion format, is the omitted word an important word rather than a trivial word? If appropriate, does the item (or the directions) inform the appropriate degree of detail, specificity, precision, or units that the answer should have? Does the item avoid grammatical (and other irrelevant) clues to the correct answer? CR: SHORT ANSWER Advantages Disadvantages Dos and Donts Easy

construction of question May not be able to anticipate all possible answers Carefully choose correct wordings of question so that the required answer is both brief and specific Ease and objectivity in scoring May only assess lower-level thinking

Harder to measure Less chance complex learning given to students outcomes in guessing the answer Do not form a question based on a direct copy of statements from textbook Give enough indication of type of answer wanted CR : RESTRICTED RESPONSE ESSAY Advantages Easy construction of question Can measure complex learning skills

Assess and improve highorder thinking Can positively influence students studying strategies Disadvantages Timeconsuming and error in scoring Limited sampling of content Dos and Donts Construct questions that can assess behavior specified in the learning outcomes Phrase the question so that the students task is clearly indicated

Indicate an approximate time limit for each question Avoid the use of optional question EXAMPLE: POOR AND BETTER The Greek mythical hero Hercules was given twelve great tasks to complete as penance. One such task was the cleaning of the enormous royal Aegean stables in a single day, which Hercules was able to complete in part due to his great strength. Poor Explain herculean task Unclear wording Better The modern phrase herculean task comes from this myth explain, in everyday language, what it means

and how it is related to the myth. EXTENDED RESPONSE (ER) Require student to provide more elaborate answers and explanations of reasoning Checklist: Does the item assess an important aspect of one or a set of standards? How does the item tap into the level of depth of knowledge as defined by the taxonomy? Does the item require students to apply their knowledge to a new or novel situation? EXTENDED RESPONSE (ER) Checklist, contd: Does the item define the tasks with specific directions and focus? Is the prompt worded in a way that students can interpret the intended task?

Does the prompt give clear information on the length of answers/writing, purpose, time needed, and basis on which the answers will be scored? If the item is on a controversial matter, does the prompt make it clear that the assessment will be based on the logic and evidence supporting the argument, rather than on the ER: EXTENDED RESPONSE ESSAY Advantages Easy construction of question Can measure complex learning skills Assess and promote high-order thinking Can positively influence students studying

strategies Disadvantages Time-consuming and error in scoring Unlimited sampling of content Possible multiple answers unreliable Dos and Donts Construct questions that can assess behavior specified in the learning outcomes Phrase the question so that the students task is clearly indicated Indicate an approximate time / word limit for each question Avoid the use of optional

question EXAMPLE: POOR Sisyphus was the son of King Aeolus of Thessaly, and was avaricious, deceitful, and murderous. In life, Sisyphus believed himself more clever than the gods. In some versions of the myth, after his death he was sent to Tartarus (a deep abyss of torment, as far below Hades as the earth is below the heavens). Once there, Sisyphus deceived Thanatos, the personification of death, into chaining himself and escaped. While Thanatos was trapped in Tartarus, no human could die. As punishment for his trickery and hubris, Sisyphus was made to endlessly roll a huge boulder up a steep hill. Each time Sisyphus neared the top of the hill, the boulder would roll back to the bottom, and he would have to begin again. In plain words, what is a Sisyphean task? How is it different from a herculean task? In common usage, what aspect of the story of Sisyphus is most important for how the term sisyphean task is used commonly today?

Vocabulary is unnecessarily complicated SAME EXAMPLE: BETTER Sisyphus was the son of King Aeolus of Thessaly. He was greedy, dishonest, and murderous. He thought he was more clever than the gods. After his death he was sent to Tartarus (a deep abyss of torment, worse than Hades). Once there, Sisyphus deceived Thanatos, the personification of death, into chaining himself, and escaped. As punishment for his trickery and arrogance, Sisyphus was forced to roll a huge boulder up a steep hill, forever. Each time Sisyphus neared the top of the hill, the boulder would roll back to the bottom, and he would have to begin again. Based on this story, what does it mean when something is described as a sisyphean task? How is it different from a herculean task? What aspect of the story of Sisyphus is most

important for how the term sisyphean task is used today? PERFORMANCE TASK (PT) Require students to do an activity to integrate their knowledge and skills across multiple content standards Must use clearly defined assessment criteria to evaluate how well a student has achieved the expected standards Checklist: Does the item assess an important aspect of one or a set of standards? Does the item require students to use curriculum-specified thinking process? PERFORMANCE TASK (PT) Checklist, contd: Is the item feasible to be done within the allocated time?

Does the item define the tasks with clear directions and focus? Is the prompt worded in a way that students can interpret the intended task? Does the prompt give clear information on the expected product? Does the item allow for multiple points of view and interpretations, as intended? EXAMPLE ITEM Which is a better material for packing fragile items to mail them, styrofoam packing peanuts, cornstarch foam packing peanuts, or crumpled paper? Which is cheaper, which is more effective, which is better for the environment? To answer these questions, follow these steps 1. Find three different sources of information about the different packing materials (can be books, newspapers, or websites). 2. Read the information carefully, and find the evidence that the authors give. Based on this information, decide which

packing material you think is better. 3. Write a paragraph summarizing your position. Be sure to explain the evidence for your position, as well as any evidence against your position. Explain in detail why you think your position is better 4. Provide a bibliography of the information sources you used PT: PROJECT/PORTFOLIO Advantages Demonstrate students best work Assess educational growth of each student Demonstrate evidence of subject-master mastery Disadvantages Time-consuming and

error in scoring Dos and Donts Use and hand out a scoring rubric or performance criteria to students well in advance Obtain more than one rater/marker, if possible PT: PEER ASSESSMENT Advantages Assess students real capacity in day-to-day basis Improve student engagement Enhance academic selfbelief and motivation in learning

Disadvantages Error in scoring Dos and Donts Use and hand out a scoring rubric or performance criteria to students well in advance Obtain more than one rater/marker, if possible TECHNOLOGY-ENHANCED (TE) Require student to use a technological tool to provide answer May not be feasible for day-to-day classroom assessment Item types can be of selected response or constructed response with short or brief answers ASSESSING ITEM QUALITY

Blueprint/LP/content review Alignment between standards and proposed items Predicted item difficulty versus actual difficulty Item panel Validity evidence ITEM BLUEPRINT For each item given, teacher should be able to: explain its relationship to the framework, justify that it is appropriately expressed for the students, generate the sort of information as intended, and ensure that the sorts of responses it elicits can be scored using the scoring guide.

Sample test blueprint Standards Item/ Task Standard Standard Standard 1 2 3 Item 1 v Item 2 Item 3 v v v

ITEM PANEL Definition Judgment or review of the quality of items by samesubject teachers or subject-matter expert(s) Activities Check if the item has measured the intended learning/assessment target (standard) at appropriate level of difficulty and depth of knowledge Review the proposed item location along the LP map VALIDITY EVIDENCE Think-aloud cognitive interviews (with example) Ask students to verbally identify their thinking process while doing the item Evaluate if students understanding of the item are as expected

Exit interview Collect students opinion about the set of items Typical questions: Which item do you find it the least difficult? Why? Which item do you find it the most difficult? Why? Can you give a constructive feedback on how to improve the test? These questions may also be posed at the end of every test/learning session (change item into topic) BIBLIOGRAPHY Nitko, A. J., & Brookhart, S. (20 07). Educational a ssessm ent of students. Upper Saddle River, NJ: Pearson Education, Inc. McMillan, J. H. (2007). Classroom assessment. Principles and practice for effective standard-based instruction (4th ed.). Boston: Pearson - Allyn & Bacon.

Oregon Department of Education. (2014, June). Assessment guidance. Wihardini, D. (2010). Assessment development I. Unpublished manuscript. Research and Development Department, Binus Business School, Jakarta, Indonesia . Wilson, M. (2005). Constructing measures: An item response modeling approach. New York: Psychology Press, Taylor & Francis Group. Wilson, M., & Sloane, K. (2000). From principles to pra ctice: An embedded assessment system. Applied Mea surement in Education, 13 (2), pp. 181-2 08. Smarter Balanced Assessment Consortium. (2012, April). General item specifications. CREATIVE COMMONS LICENSE I t e m Ty p e s ( E L A ) P P T b y t h e O re g o n D e p a r t m e n t o f E d u c a t i o n a n d B e r ke l e y E v a l u a t i o n a n d A s s e s s m e n t Re s e a rc h C e n t e r i s l i c e n s e d u n d e r a C C BY 4 . 0.

Yo u a r e f r e e t o : Sha re cop y an d red ist rib u te th e ma te rial in any med iu m o r forma t Adapt remix, transform, and build upon the material Under the following terms: A t t r i b u t i o n Yo u m u s t g i v e a p p r o p r i a t e c r e d i t, p r o v i d e a l i n k t o t h e l i c e n s e , a n d i n d i c a t e i f c h a n g e s w e r e m a d e. Yo u m a y d o s o i n a n y r e a s o n a b l e m a n n e r , b u t n o t i n a n y w a y that suggests the licensor endorses you or your use. N o n C o m m e r c i a l Yo u m a y n o t u s e t h e m a t e r i a l f o r c o m m e r c i a l p u r p o s e s. Sha reAlike If you rem ix, tran sfo rm, or b uild u p on th e m at erial, yo u m ust distribu te you r contributions under the same license as the original. Oregon Department of Education welcomes editing of these resources and would g r e a t l y a p p r e c i a t e b e i n g a b l e t o l e a r n f r o m t h e c h a n g e s m a d e . To s h a r e a n e d i t e d version of this resource, please contact Cristen McLean, c r i s t e n . m c l e a n @ s t a t e . o r. u s.

Recently Viewed Presentations

  • Curriculum development is a plan or blueprint for structuring ...

    Curriculum development is a plan or blueprint for structuring ...

    Curriculum development is a plan or blueprint for structuring the learning environments and coordinating the elements of personnel, materials, and equipment. ... Behavioral-Rational ApproachBy the Taba and Tyler Models, 1969. Most Recognized Technical . Scientific Models. The Ralph Tyler Model...
  • Dynamic Memory Allocation

    Dynamic Memory Allocation

    Abstract data types Data structures are usually implemented as abstract data types - stacks, Queues, Vectors, linked list, trees Stacks First In Last Out (FILO). Insertions and removals from "top" position only Analgoy - a stack of cafeteria trays. New...
  • Physical Exams - images.pcmac.org

    Physical Exams - images.pcmac.org

    Main types of physical exams. General, or complete, physical: all areas of the body are examined; blood, urine, radiographs, & electrocardiogram may be done.An EENT and/or GYN exam may be done during this exam.
  • The Wizard Who Did Not like Magic Once

    The Wizard Who Did Not like Magic Once

    The spell I accidentally cast cleaned my whole lab from head to toe faster than you can say 'wizards love stars!'" "OK," the wizard said. And he went home. He tried some spells and said, "This is the most fun...
  • Bab 5 (tambahan)

    Bab 5 (tambahan)

    Bab 12 Pengujian dan Permodelan Pengguna Objektif Menerangkan bagaimana melakukan ujian pengguna Membincangkan perbezaan di antara ujian pengguna, ujian kebolehgunaan dan ujikaji dalam penyelidikan Membincangkan peranan ujian pengguna dalam ujian kebolehgunaan Membincangkan bagaimana merekabentuk ujikaji mudah Menerangkan tentang GOMS, model...
  • Workplace Liability

    Workplace Liability

    Italian Legal Framework. Chinese Legal Framework. Working injury - any damage caused by an injury related to the working activity shall be compensated by the company and social fund unless intentional or due to a criminal action.. Civil tort liability...
  • HEALTH POLICY FELLOWSHIP Competition Guidelines  2017-A Intake October

    HEALTH POLICY FELLOWSHIP Competition Guidelines 2017-A Intake October

    Conny H. Lin, M.Sc. Ph.D. Candidate Neuroscience, University of British Columbia. Research Scientist | Rosetta Stone, Fit Brains Testimonial. This experience has: changed my perception of how government works (it's not what I thought at all) gained valuable connections with...
  • Universidad Autnoma del Estado de Mxico Facultad de

    Universidad Autnoma del Estado de Mxico Facultad de

    En tanto la primera corresponde al nivel institucional, la planeación táctica se desarrolla en el nivel intermedio. La planeación estratégica se transforma en varios planes tácticos, y estos, a su vez, en planes operacionales que se deben ejecutar.