 |
| Image source - https://www.utexas.edu/academic/ctl/assessment/iar/images/asmtprocess-planning.gif |
Designing and Managing MCQs:
Appendix C: MCQs and Bloom's Taxonomy
Contents of This Chapter
- C1 Bloom's Taxonomy
- C2 Application of Bloom's Taxonomy to the design of MCQs
C1 Bloom's Taxonomy
Following the 1948 Convention of the American Psychological Association, B S Bloom took a lead in formulating a classification of "the goals of the educational process". Three "domains" of educational activities were identified. The first of these, named the Cognitive Domain, involves knowledge and the development of intellectual attitudes and skills. (The other domains are the Affective Domain and the Psychomotor Domain, and need not concern us here). Eventually, Bloom and his co-workers established a hierarchy of educational objectives, which is generally referred to as Bloom's Taxonomy, and which attempts to divide cognitive objectives into subdivisions ranging from the simplest behaviour to the most complex. It is important to realise that the divisions outlined above are not absolutes and that other systems or hierarchies have been devised. However, Bloom's taxonomy is easily understood and widely applied.C1.1. Knowledge.
Knowledge is defined as the remembering of previously learned material. This may involve the recall of a wide range of material, from specific facts to complete theories, but all that is required is the bringing to mind of the appropriate information. Knowledge represents the lowest level of learning outcomes in the cognitive domain. Examples of learning objectives at this level are: know common terms, know specific facts, know methods and procedures, know basic concepts, know principles.C.1.2. Comprehension.
Comprehension is defined as the ability to grasp the meaning of material. This may be shown by translating material from one form to another (words to numbers), by interpreting material (explaining or summarizing), and by estimating future trends (predicting consequences or effects). These learning outcomes go one step beyond the simple remembering of material, and represent the lowest level of understanding. Examples of learning objectives at this level are: understand facts and principles, interpret verbal material, interpret charts and graphs, translate verbal material to mathematical formulae, estimate the future consequences implied in data, justify methods and procedures.C.1.3. Application.
Application refers to the ability to use learned material in new and concrete situations. This may include the application of such things as rules, methods, concepts, principles, laws, and theories. Learning outcomes in this area require a higher level of understanding than those under comprehension. Examples of learning objectives at this level are: apply concepts and principles to new situations, apply laws and theories to practical situations, solve mathematical problems, construct graphs and charts, demonstrate the correct usage of a method or procedure.C.1.4. Analysis.
Analysis refers to the ability to break down material into its component parts so that its organizational structure may be understood. This may include the identification of parts, analysis of the relationship between parts, and recognition of the organizational principles involved. Learning outcomes here represent a higher intellectual level than comprehension and application because they require an understanding of both the content and the structural form of the material. Examples of learning objectives at this level are: recognize unstated assumptions, recognises logical fallacies in reasoning, distinguish between facts and inferences, evaluate the relevancy of data, analyse the organizational structure of a work (art, music, writing).C.1.5. Synthesis.
Synthesis refers to the ability to put parts together to form a new whole. This may involve the production of a unique communication (theme or speech), a plan of operations (research proposal), or a set of abstract relations (scheme for classifying information). Learning outcomes in this area stress creative behaviours, with major emphasis on the formulation of new patterns or structure. Examples of learning objectives at this level are: write a well organized theme, gives a well organized speech writes a creative short story (or poem or music), propose a plan for an experiment, integrate learning from different areas into a plan for solving a problem, formulates a new scheme for classifying objects (or events, or ideas).C.1.6. Evaluation.
Evaluation is concerned with the ability to judge the value of material (statement, novel, poem, research report) for a given purpose. The judgments are to be based on definite criteria. These may be internal criteria (organization) or external criteria (relevance to the purpose) and the student may determine the criteria or be given them. Learning outcomes in this area are highest in the cognitive hierarchy because they contain elements of all the other categories, plus conscious value judgments based on clearly defined criteria. Examples of learning objectives at this level are: judge the logical consistency of written material, judge the adequacy with which conclusions are supported by data, judge the value of a work (art, music, writing) by the use of internal criteria, judge the value of a work (art, music, writing) by use of external standards of excellence. There is a great deal of information on Bloom's taxonomy on the Web. Some starting points are:- http://www.coun.uvic.ca/learn/program/hndouts/bloom.html
- http://cleo.murdoch.edu.au/gen/aset/confs/edtech98/pubs/articles/abcd/dalgarno.html
- http://152.30.11.86/deer/Houghton/learner/think/bloomsTaxonomy.html
- http://amath.colorado.edu/courses/7400/1996Spr/bloom.html
- http://www.stedwards.edu/cte/blooms.htm
- http://quarles.unbc.edu/lsc/bloom.html
- http://www.dlrn.org/library/dl/guide4.html
C2 Application of Bloom's Taxonomy to the design of MCQs
C2.1 Level 1: Knowledge
At this level, one simply requires the recall of acquired knowledge. WARNING! A test at this level can easily become a "Trivial Pursuit" exercise! Example C2.1.1Which one of the following persons is the author of "Das Kapital"? 1. Mannheim 2. Marx 3. Weber 4. Engels 5. MichelsNote that the responses are internally consistent - they are all the names of Germans whose written work have been major contributions on social issues. Example C2.1.2
In the area of physical science, which one of the following definitions describes the term "polarization"? 1. The separation of electric charges by friction. 2. The ionization of atoms by high temperatures. 3. The interference of sound waves in a closed chamber. 4. The excitation of electrons by high frequency light. 5. The vibration of transverse waves in a single plane.Simple recall of the correct definition of polarization (#5) is required. Internal consistency and plausibility are maintained in that all responses are actual physical phenomena. Example C2.1.3
According to the microgenesis of perception concept, the threshold of awareness consists of a hierarchy of thresholds. Which one of the sequences shown below is correct? 1. Recognition thresholds > physiological thresholds > detection thresholds. 2. Physiological thresholds > detection thresholds > recognition thresholds. 3. Physiological thresholds > recognition thresholds > detection thresholds. 4. Recognition thresholds > detection thresholds > physiological thresholds.In this example, nothing more is required than the recall of the order of certain pieces of related information. The correct answer is #2.
C2.2 Level 2. Comprehension
At this level, knowledge of facts, theories, procedures etc. is assumed, and one tests for understanding of this knowledge. Example C2.2.1Which one of the following describes what takes place in the so-called
PREPARATION stage of the creative process, as applied to the solution
of a particular problem?
1. The problem is identified and defined.
2. All available information about the
problem is collected.
3. An attempt is made to see if the proposed
solution to the problem is acceptable.
4. The person goes through some experience
leading to a general idea of how the problem
can be solved.
5. The person sets the problem aside, and gets
involved with some other unrelated activity.
In this question, the knowledge of the five stages of the creative process
must be recalled (KNOWLEDGE), and one is tested for an understanding
(COMPREHENSION) of the meaning of each term, in this case, "preparation".
Note that this question violates the rule that the answer and distractors
should all be of about the same length. It is difficult to get around this
one here, so the text is edited so that each line is about the same length.
C2.3 Level 3: Application
In order to classify a question into this group, ask yourself if prior knowledge of the background to the question is assumed to be both known and understood, and whether one is merely expected to apply this knowledge and understanding. Calculations based on known formulae are good examples of this, as shown in the example below: Example C2.3.1Which one of the following values approximates best to the volume of a sphere with radius 5m? a. 2000m³ b 1000m³ c. 500m³ d 250m³ e. 125m³In order to answer this question, the formula 4[pi]r³> /3 must be known (recall of knowledge) and the meaning of the various symbols in the formula understood (comprehension) in order to answer this question. The correct answer is #3. Example C2.3.2
Which one of the following memory systems does a piano-tuner mainly use in his occupation? 1. Echoic memory. 2. Short-term memory. 3. Long-term memory. 4. Mono-auditory memory. 5. None of the above.This is clearly a case of testing for the application of previously acquired knowledge (the various memory systems), which is also understood, as the meaning of each term must be clear before the student can decide whether it is applicable to the given situation. The correct answer is #1. Note that students may not necessarily know what a piano- tuner is or does. Watch out for cultural bias! The next example is more difficult to classify: Example C2.3.3
You are the sole owner and manager of a small enterprise
employing 15 workers. One of these, Alfred, (who has been
working for you for the past year and has somewhat of a
history of absenteeism), arrives late for work one Wednesday
morning, noticeably intoxicated. Which one of the following
actions is the most appropriate in the circumstances?
1. You terminate Alfred's employment on the spot, paying him
the wages still due to him.
2. You parade Alfred in front of the other workers, to teach them
all a lesson.
3. You give Alfred three weeks' wages in lieu of notice, and
sack him.
4. You wait until Alfred is sober, discuss his problem, and give
him a final written warning, should it be required.
5. You call Alfred's wife to take him home and warn her that this
must not happen again.
Note that this this question is classified as APPLICATION as in order to
answer it, the relevant labour legislation should be known and understood.
One could made a case for it to have a higher classification such as EVALUATION,
on the grounds that one is asked to evaluate which one of the proposed actions
is the best in the circumstances, or ANALYSIS, on the grounds that in order
to select the most appropriate answer, one should analyse the possible outcomesof
each decision. For both these levels, one would expect a greater amount of
information as to Alfred's situation, the relationship between Alfred and
his co-workers, union involvement in the enterprise etc., and have a more
sophisticated set of distractors. Here, option #4 is clearly the best both
on legal and human terms. Note that the figure of speech "on the spot" may
not be understood by second- language students. Use suitable language!
C2.4 Level 4: Analysis
Example C2.4.1"The story is told of the famous German Organic Chemist
Auguste Kékulé who was struggling with the problem of how
the six carbon atoms of benzene were linked together. He was
getting nowhere with the problem, and one day fell asleep in
front of the fireplace while he was pondering on it. He
dreamt of molecules twisting and turning around like snakes.
Suddenly, one of the snakes swallowed its own tail and
rolled around like a hoop. Kékulé woke up with a start, and
realized that his problem could be solved if the six carbon
atoms of benzene were attached to each other to form a ring.
Further work showed that this was entirely correct."
The above passage illustrates a particular phase of the
creative process. Which one is it?
1. preparation
2. incubation
3. orientation
4. illumination
5. verification
In the above example, the student is expected to know and understand the
five stages of the creative process, and to apply this knowledge to an important
factual example of creative thinking (the elucidation of the chemical structure
of the benzene molecule). The ability to analyse the data (i.e. the given
text) in terms of each of the five stages is what is being tested. The correct
answer, by the way, is #4.
Example C2.4.2 (Assume the question below is asked in a
philosophy test.)
Read carefully through the paragraph below, and decide which
of the options 1-5 is correct.
"The basic premise of pragmatism is that questions posed by
speculative metaphysical propositions can often be answered
by determining what the practical consequences of the
acceptance of a particular metaphysical proposition are in
this life. Practical consequences are taken as the criterion
for assessing the relevance of all statements or ideas about
truth, norm and hope."
1. The word "acceptance" should be replaced by "rejection".
2. The word "often" should be replaced by "only".
3. The word "speculative" should be replaced by "hypothetical".
4. The word "criterion" should be replaced by "measure".
This question requires prior knowledge of and understanding about the concept
of pragmatism. The paragraph, seen in this light, contains one word which
vitiates its validity, and the student is tested on his/her ability to
analyze it to see whether it fits with the accepted definition
of pragmatism. With this in mind, #2 is correct. Option #1 would degrade
the paragraph further, while #3 and #4 would simply result in changing to
acceptable synonyms. Note that this question does not address Level 6
(Evaluation), as one is not asked to pass a value judgement on the text.
This must be considered as a very difficult question, and will obviously
require a high level of reading skills. Bear in mind that there will be a
significant time factor involved.Example C2.4.3
Look at the following table and indicate which countries' statistics are being reported in rows A, B and C.
| GNP per capita 1991 ($ USA) | Growth rate of GNP per capita p.a. 1980-91 | Population growth rate 1980-91 | Structures of total employment 1980-85 (percentages) | |||
| Agriculture | Industry | Services | ||||
| A | 500 | 2,5% | 1,5% | 51 | 20 | 29 |
|---|---|---|---|---|---|---|
| B | 1570 | 5,8% | 1,6% | 74 | 8 | 8 |
| S.A. | 2560 | 0,7% | 2,5% | 17 | 36 | 36 |
| C | 25110 | 1,7% | 0,3% | 6 | 32 | 32 |
Choose your answer from the following list of possible
answers:
1. A is South Korea; B is Kenya; C is Canada.
2. A is Sri Lanka; B is Germany; C is Thailand.
3. A is Sri Lanka; B is Thailand; C is Sweden.
4. A is Namibia; B is Portugal; C is Botswana.
In order to answer this question, students must be able to recall the relative
economic rankings of various countries (KNOWLEDGE) and understand the basis
for such a ranking (COMPREHENSION). They must be able to apply these concepts
when information is supplied to them (APPLICATION), and they must be able
to ANALYZE the given information in order to answer the question. Students
did not like this question when they were faced with it in a class test,
as their immediate reaction was that "it was impossible to remember the
statistics for all the countries that were discussed in class and given to
them in handouts". They were surprised when told that such detailed knowledge
was in fact not expected of them, but that they were to examine the table
and perform a ranking on the basis of concepts that they should have mastered.
The correct answer is 3.
C2.5 Level 6: Evaluation
At this level, one is asked to pass judgement on, for example, the logical consistency of written material, the validity of experimental procedures or interpretation of data. Example C2.5.1A student was asked the following question: "Briefly list
and explain the various stages of the creative process".
As an answer, this student wrote the following:
"The creative process is believed to take place in five
stages, in the following order: ORIENTATION, when the
problem must be identified and defined, PREPARATION, when
all the possible information about the problem is
collected, INCUBATION, when there is a period where no
solution seems in sight and the person is often busy with
other tasks, ILLUMINATION, when the person experiences
a general idea of how to arrive at a solution to the
problem, and finally VERIFICATION, when the person
determines whether the solution is the right one for the
problem."
How would you judge this student' s answer?
1. EXCELLENT (all stages correct in the right order
with clear and correct explanations)
2. GOOD (all stages correct in the right order, but the
explanations are not as clear as they should be).
3. MEDIOCRE (one or two stages are missing OR the
stages are in the wrong order, OR the explanations
are not clear OR the explanations are irrelevant)
4. UNACCEPTABLE (more than two stages are missing
AND the order is incorrect AND the explanations
are not clear AND/OR they are irrelevant)
In the above question, one is expected to make value judgment on the content
of the given text (KNOWLEDGE of the subject is required), the meaning of
the terminology used (COMPREHENSION of the subject matter), and its structure
(ANALYSIS of the answer for the right order of events. The correct answer
here is #1, but suitable modification of the putative student answer could
provide a small bank of questions with other correct answers
Example C2.5.2
Another example is the "Assertion/Reason" question, in which two statements
linked by "BECAUSE" have to be evaluated in the light of certain criteria:
Judge the sentence in italics according to the criteria
given below:
"The United States took part in the Gulf War against Iraq
BECAUSE of the lack of civil liberties imposed on the Kurds
by Saddam Hussein's regime."
a. The assertion and the reason are both correct, and the
reason is valid.
b. The assertion and the reason are both correct, but the
reason is invalid.
c The assertion is correct but the reason is incorrect.
d. The assertion is incorrect but the reason is correct.
e. Both the assertion and the reason are incorrect.
The correct answer is "b", since while it is true that the United States
took part in the Gulf War, it is also true that the Kurds in Iraq did not
(and still do not) enjoy an abundance of civil liberties, but the threat
to the US's oil supply as a result of Iraq's invasion of Kuwait was a much
more pertinent reason for the United States joining in the fray. A knowledge
and understanding of Middle East politics is assumed. What is tested here
is the ability to evaluate the between cause and effect in the sentence in
terms of predefined criteria.
Link source - http://web.uct.ac.za/projects/cbe/mcqman/mcqappc.html
More links to assessment examples:
https://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-scenario.php
https://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice.php
FAQ on Outcomes Assessment
Source of PDF below is -
https://www.utexas.edu/provost/planning/assessment/iapa/resources/pdfs/Outcomes%20Assessment%20Frequently%20Asked%20Questions.pdf
Powerpoint Presentation about Outcome-based Education
Source - http://bit.ly/1w1reW3
