Evaluation strategies at the Teaching Learning Center of IUCAA
The teaching Learning Centre (TLC) at IUCAA conducts courses and workshops on various topics related to astronomy and astrophysics. Here, we describe the ways in which our managing of courses and participant evaluation process has evolved in the past couple of years as we adapt to changing to various constraints and make progress towards finding out best ways to manage our courses.
Here’s a small sample of the kind of courses (of relevance to learning management and assessment) we had organized at the TLC in the last two years:
- Refresher Course in Astronomy and Astrophysics
A yearly course covering basic, advanced, and special topics in astronomy and astrophysics for faculty members. The course contains about 75–80 hours of lectures and online quiz-based assessment. - Introduction to optical and infrared interferometry
A course based on pre-recorded lectures given by Prof. Jean Surdej covering theoretical fundamentals in the field of optical and infrared interferometry. This course involved weekly live interactions between the course lecturer and the participants and regular quizzes and assignments. - An advanced course in pulsar astrophysics
A course based on pre-recorded lectures given by Prof. Dipankar Bhattacharya covering a wide range of advanced topics on pulsars including theoretical, numerical, and observational results. This course involved weekly live interactions between the course lecturer and the participants and regular quizzes and assignments. - Introductory course on astronomy and astrophysics for college teachers
This online course on Introductory Astronomy and Astrophysics (organized in collaboration with Tezpur university) was meant largely for college and university teachers who may wish to teach the subject as part of their regular course work. The course contained about 30 hours of lectures and online quiz-based assessment.
Most of these courses, with the exception of the refresher course, see about a 100 participants register and about half of them participate regularly and complete at least some assignments and quizzes. The refresher course sees larger number of participants.
So, in this post we will try to outline how we manage participants sizes between 50–100 and try various strategies that support their learning.
The fundamental goal of assessment is to measure learning. Although measuring learning accurately is not a simple task, aspirationally we try to design assessment prioritizing learning and engagement of the participants to course material. The following are some of the strategies we adopt for learner assessment.
1. Open book, extended time quizzes
These are short objective-type quizzes with 10–15 questions for every 2–4 lectures which can be completed over the period of a week. The primary goal of these quizzes tends to be highlighting important ideas from the relevant lectures by testing how well the participants have understood them and the grade itself of secondary importance.
Benefits and drawbacks
+ Emphasizes learning rather than performance by allowing participant to go back to source material to figure out the answers.
+ Greater flexibility for participants to choose when they want to attempt the quiz. Quiz can be accessed multiple times before final submission.
+ Easy managing the quiz since scheduling is flexible and an open-book test removes the need of proctoring
– No mechanism to avoid someone blindly copying off someone else
– Requires extra effort to create sufficient hard objective type questions
How to make these quizzes effective?
(a) Quality of questions
An open-book test requires questions that cannot be answered simply by searching for the question online or in a book. It must require understanding concepts to a certain degree or able to apply a concept to make inferences about or applying it to a new problem. Simple numerical problems can always be included since they require one to apply the knowledge rather than do a simple recall.
Here’s an example of a relatively simple question (from the Introductory course on optical and infrared interferometry) the answer to which cannot simply be found by searching online or from a book. It is fairly straightforward when one understands the concept which is discussed in the lecture.
We cannot measure the electric field at optical wavelengths because the (Select all the options that apply)
a. energy is too low
b. wavelength is too large
c. intensity is too high
d. frequency is too high(b) Format of questions
The chosen format of the question can also ensure that the questions are challenging for the learner and requires deep understanding of the material. Instead of simple multiple choice question with a unique correct answer and a few detractors, we preferentially use multiple selection questions with negative marking and numerical answer questions.
The following examples, again from the introductory interferometry course, illustrates the point
::Q8:: The visibility in a Fizeau-type interferometer is equal to the complex amplitude of mutual coherence when which of the following conditions are satisfied? {
~%-50% The light is monochromatic
~%25% The source is point-like and infinitely far from the interferometer
~%25% The apertures are infinitesimally small
~%25% The amplitude of the electric vectors is constant over the wavelength range
~%-50% The coherence length is much smaller compared to the difference in path lengths
~%25% The light is quasi-monochromatic
}The question above shown in gift format is a multiple selection question with 6 options to choose from, four of which are correct and they carry partial credits distributed evenly, and two incorrect options which carry negative credit of 50% each. The “negative credits” are not truly negative markings since they can deduct score only up to zero and cannot allow negative grade for any question.
::Q7:: Estimate the frequency of amplitude modulation of the electric field when observing a source using the V-band filter ($$\lambda\=551\pm44$$ nanometers) through a Young two-hole interferometer. (Enter only numeric value in Tera-Hertz ($$10^\{12\}$$ Hz) up to two significant digits) {
#43:4
} THz .The question above is a numerical question which accepts only a numerical value withing the range 87 ± 5 as the correct answer. Below one can preview the questions in the form it is presented to the participant for review after the completion of the test.
(c) Quiz rules
The quiz rules make it clear to the participants that questions may contain negative marking which discourages random guesses and selections and encourage careful, serious answering and double checking their understanding.
We explicitly forbid participants from collaborating to answer the quiz questions since we prefer them to use lectures and books to arrive at the answers individually. Posting of direct answers to quiz questions is also forbidden in course forums.
We however do allow participants to ask any questions related to concepts that may be required to answer these questions, during weekly meeting and in forums, as long as they are not asking for direct solutions.
2. Assignments
We all would agree that assignments are absolutely essential for efficient learning since it gives the learner opportunity to work on problems by themselves based on what they have learned, use textbook or other external resources to strengthen their understanding, apply and make inferences on a well-posed problem.
In an introductory course, the assignments can be simple nearly textbook style questions that require adapting some formulae to a new problem or making numerical estimates to understand a physical system. Our course on interferometry did use mostly textbook style questions in assignments for participants to work out and submit their worked out solutions.
Advanced courses, however, cannot be very effective with simple textbook style problem solving as they often cover topics that cannot be found in textbooks. Here, we had to take a different approach.
(a) Step-by-step derivation of theoretical results from key papers
The goal here is to make it easier for learners to re-derive key results presented in the course lectures. The challenges usually are (i) papers do not always give a detailed derivation of theoretical results (ii) limited time available during the course may not be sufficient for participants to derive the result unguided.
We provided exercises with step-by-step instructions for deriving various results. A part of an assignment sheet is shown in the figure below where we re-derive results from the Goldreich-Julian model of the magnetosphere for the advanced pulsar astrophysics course. The text in blue are omitted when offering the assignment and is for the learners to work through.
(b) Data analyses using real astronomy data
Astronomy learning experience is incomplete without working with actual data. As often as possible we try to create hand-on data analysis tutorials simplified enough that they can be worked through in a relatively short period of time and get a much personal feel for various observational results.
Offering such exercises to audience with a fairly wide range of background or expertise requires that
(i) the data are reduced to simple forms that are straightforward to use
(ii) participants are not limited by computing resources for which online computing resources such as Google Colaboratory are useful
(iii) Basic parts of the code such are initializing libraries, loading data, and perhaps even plotting routines are setup beforehand so that the learners can focus on the relevant science-related sections
A couple of examples are shown below, the first one is a Colab python notebook based code for reading publicly-available radio polarization data and viewing the pulsar’s position angle traverse as part of the pulsar astrophysics coursework.
Another example below shows part of the code used in the Radio astronomy winter school where raw voltage data from the Ooty Radio Telescope is used to detect pulsar signal and estimate its dispersion measure and period. The code helped the participants apply concepts related to Fourier transforms, radio observations, and pulsar astrophysics, learned in lectures.
(c) Simplified derivation of textbook results
There are often derivations that are only found in specialized books that are not easily available. Sometimes a complete picture of a concept is realized only by combining information from multiple sources. Assignments are useful for such scenarios as well.
The pulsar astrophysics course included such assignments where learners are prompted to derive concepts such as ‘Condition for coherent emission’ and ‘Beta equilibrium proton-to-neutron ratio’ (see figure below) which are not commonly found in books dedicated to pulsars.
An obvious question arises when planning such assignments for a course and that is grading the assignment submission. Individually grading the assignment submissions may be feasible for class sizes less that about 20–30 but can be time-consuming for large class size. In order to simplify the assignment grading process, we created ‘assignment assessment quiz’.
3. Assignment assessment quizzes
The idea behind assignment assessment quizzes is to determine if the assignment problems were fully solved and possible give partial credit for partially solved problems. This requires questions related to the final results of the exercises, their inferences, and some intermediate steps. Even data analysis assignments can be graded using this method.
Shown below is a ‘match the options’ question that has a higher probability of being answered correctly if pulsar polarization data has been analyzed as part of the exercise.
Assignment with derivations from Goldreich-Julian model of pulsar magnetosphere (Pulsar astrophysics course) can be assessed using questions such as the ones shown below. The first question asks the form of the electric field which is derived in an intermediate step of the exercise and is not directly found in the paper or books. The next question asks the learner to make an inference based on the exercise results.
Another example taken from deriving the proton to neutron ratio under beta-equilibrium in interiors of neutron stars, are numerical evaluation questions that can correctly answered upon deriving the expression as part of the assignment.
Important note: It is important to balance the assignment assessment quiz and the assignment submission itself. A mixed grading approach was adopted by us, where, the assignment assessment quiz takes into account the accuracy of the assignment exercises and the assignment submissions themselves are also graded with a concession. The grading of assignment submissions can now be done swiftly without paying too much attention to the accuracy of the final result (which is tested by the accompanying quiz) but only to make sure the participant has performed a valid derivation. One can easily check if the submitted assignment is identical to that of any other participant or there is sufficient evidence for the participant having worked through the exercises in sufficient detail and by themselves.
For assignments, our policy is to allow discussion with other participants through forums or privately, without directly exchanging answers, and the assignment submission must show that they eventually did the work themselves.
4. Timed quizzes
We do offer timed quizzes where a specific date and time are chosen, commonly applicable to all participants, when they can log onto the course site and take the quiz during the allotted time.
Such quizzes are suitable especially when participant numbers are large and/or when the course topics are so diverse that we do not have sufficient human resource to produce assignments and hands-on material for all those topics.
Our strategies for these involve some of the following:
1. Collecting objective-type (usually multiple choice) questions from the speakers and review/modify them as needed
2. Requiring mandatory integrity pledge to discourage using unfair meant to take the quiz
3. Use Moodle quiz feature to present one question per page and randomize both answer choices and question order for all participants. Only randomizing of questions allowed when using Google forms.
4. Assigning quiz time such that the participants just have enough time to answer the questions and not enough to possibly contemplate looking up some of the answers do help.