Month: March 2023

The purpose for introducing the constructed response question (CRQ) in middle school is to prepare students for this kind of assessment later in their education. Ideally, the task should lay the groundwork for the habits of mind that promote success and should accustom students in a practical way to the assessment itself, its common form and its vocabulary. Experience teaching this to eighth graders shows that one of the first major obstacles is to get students to move away from the reflex ingrained in elementary school: to respond to a text by stating what it says. The second major obstacle to teaching this is that students coming out of elementary school are wholly unfamiliar with the idea that some materials they may be given are quite possibly not reliable. In addition, they lack the vocabulary to manage the concepts of text reliability.

It is difficult for upper elementary students to address primary source material for a variety of reasons. First and foremost is the text complexity. Secondly, a limited ability to comprehend given their severely limited background knowledge (class lessons should remedy this). The middle school CRQ needs to be accessible to most students while still preserving the “primary source” characteristic of the task; the opportunity to see what people of the past had to say in the way they said it. It is often a further revelation to people at this age that the English language has not always existed or that it has existed in variant forms they would find incomprehensible. An appreciation of language change and variety plays an important role in addressing primary sources for this age not only for a deeper understanding but to appreciate reliability concerns of translation and excerpts and secondhand accounts. The documents for analysis in the middle school CRQ will be carefully devised in the following ways:

• An image of the source’s original format and language will be provided for purely observational purposes. This may be merely an incomplete image or fragment.
• A standard translation of the source will be provided despite that it is at a text complexity above the grade level band. This is also purely for observational purposes, though some students may make the attempt to analyze it.
• A translation of the source into a Lexile range of 800-1000 will be provided if necessary. This is the document on which students are to work.
• There will be a citation of the source in the simplified version of the citation format used by genealogists. Students should consider the source in their analysis.

One task will consist of two pairs of documents. Students will give the historical or geographical context of the first document in each pair. Students will assess the reliability of the second document in each pair. In addressing the reliability of the source, students will need more support, naturally, than their compatriots at the high school level. The second document in each 2 pair will ask the student to address reliability in multiple-choice format. This will habituate the student to the typical phrases used in addressing reliability. First, students will be prompted in multiple-choice format to identify the document’s bias, point of view, audience, or purpose. Secondly, students will be asked to identify the best use of the document for a historian or anthropologist. Thirdly, a multiple-choice format question will ask the student to conclude how the reliability factor affects this document’s use as a reliable source of evidence to prove something specific. This latter point is important to help students understand that sources may have different reliability depending on what the historian wishes to do with them.

The last part of the task calls upon the student to synthesize a relationship between documents 1 and 2. This will also be in multiple-choice format. Students will be prompted to use both documents in one of three possible ways: (a) state a similarity or a difference between the documents; (b) explain one change associated with a turning point in history that the documents reflect (the turning point will be identified for the student); (c) explain how some development or idea is the cause of some event, idea, or historical development reflected in both documents.

Gradually toward the end of grade eight, students will move toward short answer format CRQ’s as they will see in high school. Having seen the same wording each unit across grades six through the first part of eighth, the idea of context, reliability, and turning point should be well established. A multiple-choice version for students who are still developing the skill could be offered for a reduced maximum score.

Task Models

Borrowing from the “task model” concept used in developing the New York State Global history and Geography Regents examination part one (stimulus-based multiple-choice), the following are the task models for the multiple-choice version of the middle school CRQ. It will be important to use similar language in constructing the questions for consistency.

• Which of the following [statements | titles] best represents the [historical | geographic] context of the [document | map]? 
• Correct answer will be historical background information not present in the document 
• One incorrect option will be of the type “This document is about…” 

• Correct answer will be geographic background information that explains the origin of the map’s information 
• One incorrect option will be of the type “This map is showing is about…” 
• Which of the following statements best represents the geographic context of the map? 
• Which of the following would be the best use of this document for a historian? 
• Which statement best describes the [point of view, intended audience, purpose, bias] of the document? 

• When point of view is asked, one incorrect option will be “first person’ or “third person”. This is to teach the student to distinguish between how that term is used in English and how it is used in social science. 
• Which of the following factor(s) would [weaken | strengthen] the reliability of this source for the purpose of __. 
• The reliability factors taught are: authorship, format, point of view (objective or biased), time and place, intended audience, purpose. 
• These will often have more than one correct answer. 
• The factors are listed, followed by a colon and a description. Example: 

• Point of view: The author is very biased. 
• These two sources are artifacts from a turning point in history. Which would be the best title for that turning point? 
• Which statement best describes a [similarity | difference] between the two sources? 
• These two sources are artifacts from historical events. Which statement describes a cause-effect relationship of the historical events the sources represent? 

Assessment Task Comparison Across Three Assessments 

The purpose of developing this task is to create a logical early training step for students in middle school working toward the assessment tasks they will see in high school. 

Teachers use video lessons a lot these days, some they create themselves, and other videos that they find on Youtube. Not everyone has time to build a set of cued comprehension questions for each video lesson. Some teachers assign their students to summarize what they saw in the video. This is a fantastic way to keep kids engaged in the video, but it’s a lot of work scoring them.

The Innovation AI grading assistant is the perfect solution. In this post, I would like to show you how to generate a video lesson in Innovation and then quickly train the AI to help you score the summaries very rapidly.

Quick Links to Short Video Tutorials

• Add a video for students to summarize
• Student access to the video and summary space
• Part 1, Train the AI grading Assistant
• Part 2, Train the AI grading assistant
• The AI can coach students to write better summaries.

A Word about the Innovation AI

There are two broad categories of AI: those that work using a complex algorithm and those that work using large language model learning. The Innovation AI is of the former type. The latter, the machine-learning AI, is exemplified by ChatGPT and the Bing AI under development right now. They “learn” by analyzing vast amounts of data across the internet. The Innovation AI is trained on five to seven models that the teacher provides for comparison.

I developed the Innovation AI to help me grade summaries and short answer tests. When I taught social studies, I often assigned summaries of texts instead of comprehension questions. It works by comparing the student text to a number of models and scoring the comparison on a dozen features. These features include measures of similarity like cosine and Jaccard, as well as readability, number of words, level of text complexity, and so forth. The scoring rubric was designed using 500 of my students’ work submissions that I had scored manually so that the AI essentially grades as I would.

The Innovation AI is highly effective for helping you score summaries and short answer responses where the range of possible answers is fairly limited. The AI does effectively recognize different ways to say the same thing using natural language processing algorithms.

Training the AI

When you train the AI, you give it model answers to use in the comparison algorithm. For short answer tasks, the limit is presently five models. For writing samples such as summaries and compositions, you can store up to seven model answers.

The process, in summary, is this: (1) Compose your own summary of the task or let the Innovation AI generate a summary from the source text for you; (2) Manually score the work submissions of your students who usually get full credit. When you find a submission to which you would award full credit, ask the AI to score it. If the AI cannot recognize it as a full credit answer, you “add it to the corpus” of model answers. The next time you ask the AI to score a student submission, it will compare it to each of the models in the corpus and award the highest score earned by the student in those comparisons.

Since many of us reuse our assignments from year to year, you really only have to do this once. I trained the AI on most of my Global 9 and Global 10 assignments in 2018 and just continued to score with those for several years.

Creating an Inbox for a Task with Embedded Video

1. Select the Inbox button from the new course playlist element dashboard.

2. Enter the title and some optional attributes. Paste in the embed code from the video you want students to watch and summarize.

3. Once created, you can click and drag the element to its right position in your class playlist.

Students Save their Summaries

When students access the task from the course playlist, they will see the video you embedded and the space underneath to compose their summary.

AI can coach students to write better!

The AI grading assistant can be engaged to coach students along the way. As they compose their summaries, they can periodically click the “Coach” button to get an estimate of their grade so far. In my experience, this promoted student prolonged engagement for a better work product.

The stimulus-based multiple-choice test item was introduced into the New York State social studies Regents examinations starting in 2019 for Global History and Geography II and for United States history. The task poses challenges for students such that it merits some regular, focused training throughout the year.

In a stimulus-based task, the student is directed to respond to a document, map, or image using their ability to analyze and their knowledge of historical context. In the case of the New York State exams, there are eighteen “task models” used when designing questions. For example, a student may be asked to evaluate and classify (identify) best use of a source or to respond based on knowledge of historical context. Principles of reliability assessment are applied here, such as when students are asked to identify point of view, purpose, context, bias, format of source, location of source in time and/or place, and/or intended audience of sources using background knowledge.

Click here to shop stimulus-based tasks at my store for grades seven through eleven social studies.

The first important habit of thought to train students to engage is to think beyond the document. Habit since their first reading lessons has asked them to find the answer in the text somehow. It takes a lot of practice and reinforcement to get students to activate their schema on the topic; to think of the story of which the document is but a fragment. The question cannot be answered without background knowledge.

• part I of the Global and US History Regents
• 25-30 questions
• primary or secondary source documents
• M-C questions are always paired with stimulus
• primary or secondary source
• maps
• charts
• cartoons
• may have more than one stimulus tied to it
• no more stand-alone questions
• estimate 30-45 minutes for this part of the exam

I used almost exclusively primary source documents for my stimulus-based tasks. This can be challenging for weaker readers, but with practice in skills for addressing difficult texts, this obstacle can be addressed.

My custom was to assign a stimulus-based multiple-choice at the end of every unit starting in October. At first, students find these very difficult. I use a z-score standardization procedure to adjust the scores so as not to bomb out their GPA while they are just learning. Click here to read up on standardized scoring. It is a great way to score students in tasks they are not yet proficient at.

• Practice! Students are generally not good at these at first. 
• Read the question first. be certain you know what it is asking.
• Remind yourself that the answer is rarely found in the document itself.
• Identify the historical time period the documents go with.
• Consciously call to mind the historical context of the document before you read. Try to recite to yourself who, what, when, where, why of the time period.
• Use process of elimination to narrow down the options.

Some of my students always would wonder why ask questions this way. If the test writers want to know whether a student knows something, why not just ask? I don’t have a good answer for this. I strongly support instruction in social studies that calls upon students to think critically and make meaningful connections with knowledge. I also think that students should actually possess knowledge. This assessment method was no doubt inspired by AP history exams. It remains a question in my mind as to whether this level of complexity is necessary for an instrument for secondary school evaluation. If we want to know whether the student knows what caused the French Revolution, for example, maybe we should just ask them that?

The updated New York State Regents examination in United States History and Government, part II, is a short essay task designed to measure students’ ability to work with historic documents. It is a mature version of the “CRQ” found on the tenth grade Global Regents. Students are called upon to understand text, engage it with historical context, and assess a text’s reliability.

In document set 1, students describe the historical context surrounding two documents and identify and explain the relationship between the events and/or ideas found in those documents (Cause/Effect or Similarity/Difference or Turning Point).

Turning point is always the most challenging for students, mainly because it demands a strong knowledge of historical context which only the higher performing students usually possess. In stating similarities and differences, it is important to stress to students that this should be a substantial feature of the two texts, not trivial. For example, some students may respond something like this: “Document one is a cartoon and document two is a newspaper report”. This is trivial and should be discouraged. For cause and effect analyses, remind students that some events may lie outside the documents at hand, so they may need to rely on their historical knowledge.

Click here to short for short essay prompts at my TeachSimple store.

Document set 2 asks students to describe the historical context surrounding two documents and (for one identified document) analyze and explain how audience, or purpose, or bias, or point of view affects the document’s use as a reliable source of evidence.

A good strategy for ensuring students possess the skills to address this task is to be certain to assign one every month or so throughout the year, followed by a debriefing where the class can study their classmates’ work (anonymously) and develop strategies for improvement.

Weaker readers are particularly disadvantaged in this task, although since the test items are field tested before administration it is likely the field testing will mitigate some issues with the difficulty of reading some primary source texts. Students can be taught compensatory reading strategies to help deal with difficult texts.

As always, the challenge is to ensure that students have learned a strong body of historical context. That is, the best marks are reserved for those who actually recall the history and who can analyze it (cause-effect, turning point, etc.) This is best achieved by regularly administering quizzes on historical knowledge. I like to give students time in class to study for these. The apps here at Innovation Assessments are especially suited to that end. A lot of social studies assignments can tend to be just look-it-up and transfer kinds of exercises without real demands on students to remember. This is an easy instruction error to remedy.

Teaching factors affecting the reliability of sources is another matter. This takes a great deal of time and practice and, I would argue, is of upmost importance for a person’s education in this day and age. Students, for the most part, do not intuitively analyze and explain how audience, or purpose, or bias, or point of view affects a document’s use as a reliable source of evidence. They tend to take what we give them on face value. It is important to teach students to think critically and approach all historic documents with a healthy skepticism.

Click here to shop my store at TeachSimple.com

I think teaching reliability should begin young, down in middle school. Engaging students with documents that have very vague reliability weights is a good practice. In a debriefing after the task, it is useful to anonymously display some student reliability evaluations for all to see and to discuss. It is important to do this regularly, starting off right at the start of the school year. There are really no stock phrases that students can learn by rote for this, given the variety of context and source material. I had the benefit of working in a small school where I had the same students grades seven through ten or eleven, so I could implement reliability assessments early in my program. In larger school districts, it would be good to consider a commitment to reliability factor training from an early age. I assigned one longer primary source to analyze each topic (so about once a month). This short essay included an extended analysis of reliability in a conclusion paragraph. The training paid off and when my eleventh graders were preparing for the Regents in US History and Government, they had little difficulty with reliability factors.

A good piece of advice on this is to assign students to do this every month in grade eleven. I suggest assigning it as a test each time. Coach students on the historical context they have to memorize in advance. Lots of teachers assign these for homework, but this entirely misses the point of such training. Student independent work practices are highly efficient in applying the minimal effort to a task, including copying their colleagues’ work or copy-pasting from a source. If they are not doing these without notes, they’re not really practicing.

If you are afraid to assign your students this as a test because they are not likely to do well at first and don’t want to bother their GPA, I recommend using standardized scoring. You can use the z-score calculator here at Innovation Assessments. Use 78 as your standard mean and 14.8 as your standardized standard deviation. Read more about standardized scoring here and where I got those figures. The beauty of this system is you can apply this to their grades every month and as the class improves, as the class average approaches the standardized mean (78 in this case), then the algorithm affects their scores less and less.

Once you have scored their papers, select out some problem responses for class discussion in a debriefing. Keeping the responses anonymous, review how to improve the answers next time.

The short essay is 14% of the Regents score, so for passing the text it’s a good practice. But I would suggest that this kind of work as a regular lesson is extremely valuable as an educational tool. Studying primary sources rightly should take center stage in our social studies lessons. It is not just teaching to the test to do this. It is developing a critical thinking skill set and insisting on recall of historical context that are the values here. Remember, that the highest valued performance is that which is based on a substantial recall of historical context!