Teaching
Undergraduates at U of T's Economics Department
(Our Intranet
is a complementary resource)
|
Course sites (including Quercus) |
||
|
Teaching and
Learning Community of Practice (CoP), Department of Economics, 2016-2023 |
Last
updated: April 25, 2023; Created and maintained by: Jennifer
Murdock
|
A&S
Academic Handbook for Instructors (html) |
Center
for Teaching Support & Innovation (CTSI) Faculty
of Arts & Science: Teaching Technology Support Accessibility Services &
Accommodated Testing Services (ATS) |
|
Undergraduate
Programs, St. George Economics St
George ECO timetable & syllabi (fall/winter) |
Policies, Guidelines &
Best Practices (Provost) University
of Toronto Provostial Guidelines on the Use of Digital Learning Materials
(i.e. asking students to pay to access online materials, iClickers, etc.) |
|
LSM: Learning Space Management
(formerly ACE, Academic and Campus Events, and before that, OSM, Office of
Space Management) Report classroom
malfunctions (LSM (formerly ACE, and before that, OSM), St. George) |
|
|
A&S Digital Teaching and
Learning (Quercus site, Online Learning Academy) Teaching and
Learning Community of Practice (CoP), Department of Economics |
Current A&S Timetable
(regular academic year): find your classroom, current enrolment, cap on
enrolment |
A&S
Academic Dates & Deadlines (also A&S Sessional Dates)
A&S
Times and locations for final examinations
Accommodations
for Religious Observances (Provost)
A&S
Registration dates for students
back to top
·
You can view the course outlines of your colleagues on
the undergraduate course time tables at St.
George (and sometimes via a link from the faculty
member's own homepage).
·
***Your course outline must be uploaded to our
departmental website: http://www.economics.utoronto.ca/index.php/index/teaching/coursesForUser***
·
The syllabus and the first day of class are your
chance to communicate your expectations to your students. Make your
expectations clear and precise for a smoother course and improved student
satisfaction.
·
You may require students who miss an assessment due to
illness to provide a U of
T Verification of Student Illness or Injury Form.
·
Visit your classroom before the first class. Room
assignments are posted at Current A&S Timetable
(regular academic year). Visiting in-person is ideal. You can also check
out the room virtually. (Additionally, rooms will be posted
before the first class on our
website.)
Teaching and
Learning Community of Practice (CoP), Department of Economics
EdTech Office for the Faculty of
Engineering (all U of T faculty welcome & often relevant for Economics)
Center for
Teaching Support & Innovation (CTSI) Programming and Teaching & Learning Symposium (annually)
Academic Skills Centre (UTM)
·
You may periodically check The Journal of Economic Education. From a U of T computer, you get
full electronic access directly at http://www.tandfonline.com/toc/vece20/current.
·
CTSI gives seminars and courses to help you in your
professional development as a university teacher. Visit their web site to
sign up. Subscribe to the CTSI listserv
to ensure you receive timely information about upcoming events,
workshops, lunch-time roundtable discussions, etc. To subscribe, you may send
an e-mail to CTSI
asking them to add you.
·
You may choose to include a mid-course review of your
own design. Students typically given more substantive and constructive comments
on these than end-of-term evaluations. CTSI has a booklet entitled "Gathering
Formative Feedback with Mid-Course Evaluations". MS Forms is a useful, easy, quick, and
institutionally supported tool that lets you create a professional looking
online survey and easy-to-review results.
·
You may consider joining (subscribing) to the Society for Teaching and Learning in Higher
Education (STLHE), which is a Canadian association dedicated to the
advancement of teaching and learning, or subscribing to a teaching newsletter
such as The
Teaching Professor, which will help you keep up-to-date with developments
in teaching and learning and keep your courses on the cutting edge without
taking up much time.
Student
evaluations:
Course evaluation framework for
all of U of T
Online searchable database of
undergraduate course evaluation results since 2012/13 on Quercus: click
“View Past Evaluation Results”
“U
of T Hub: A course selection tool for new and returning students” a
September 26, 2021 article in The Varsity discussing https://uofthub.ca/
ASSU Anti-Calendar
(St. George Campus) (published summaries of faculty members' student survey
results; the last publication is Summer 2012)
Copy of blank student survey form (used
until 2012/13) (St. George Campus)
Policies
on assessing teaching:
A&S
Faculty Resources (links for PTR, Continuing Status Review, Tenure Review)
A&S
Guidelines and Procedures for the Assessment of Teaching Stream Faculty (for
Probationary, Continuing Status and Promotion Reviews) (Also, Teaching
Evaluation Guidelines across U of T)
Teaching
Evaluation for Promotion and Tenure (Provost's Guidelines; general for U of
T)
A&S
Guidelines for the Assessment of Effectiveness of Teaching in Tenure and
Promotion Decisions (Research stream; St. George Campus)
A&S
Student Academic Integrity (SAI)
Academic Integrity at the
University of Toronto
Arts
& Science Academic Integrity System (Note: To log in to the SAI to
report cases, you must use VPN even if you are on campus using your U of T computer.
Using VPN, connect to general.vpn.utoronto.ca and then do your UTORid
authentication.)
tests/examS, ETC.:
Important
rules regarding term work (Academic Handbook Sections 2.8 – 2.19)
University
of Toronto Libraries Exams and Course Collections A complete repository of
old final examinations across all three campuses
See Using
multiple choice questions on this website
Academic
accommodations for students with disabilities
University
of Toronto Provostial Guidelines on Online Assessment Invigilation
·
At U of T there is a major distinction between term work (which is anything you
administer during the term such as term tests, essays, etc.) and final examinations (which you must
assist in but you are not in charge of administering it: i.e. you do not
control the date, illness policies, or the reporting of results for final
examinations to students)
·
Scheduling your term tests and obtaining appropriate
testing rooms can be challenging.
o
To avoid student conflicts with other courses schedule
tests during lecture or tutorial time. Note: Section
6.3 of A&S Academic Handbook: “If a student has a conflict between a
course holding a test outside its normal class hours and a test or required
obligation for a class regularly scheduled into that hour, the
regularly-scheduled academic obligation has precedence. The course with the
irregularly-scheduled test must accommodate the student in some appropriate
way.”
o
Avoid dates with religious
observances.
o
If you are teaching a 200-level course, coordinate
with other instructors on term test dates as many students are simultaneously
enrolled in intermediate micro (ECO200/204/206), quantitative methods
(ECO220/227) and intermediate macro (ECO202/208/209). All streams of
intermediate micro should strive to overlap on test dates (as these courses are
exclusions for each other so no student could have a conflict). Similarly, all
streams of intermediate macro should strive to overlap on test dates. We should
strive for minimal overlap in test dates between micro, quantitative methods,
and macro.
o
Typically, your regular lecture room is not
appropriate for testing. To request a room, ask the Economics Department Undergraduate Assistant to
book a room for you (make sure to specify the course, sections, number of
students taking the test, date, and time). Do this early (before classes
start) to ensure that it is possible for you to get an appropriate room at the
requested time: sometimes the University will not accommodate you. You will
receive a confirmation e-mail: bring a copy to the room as proof that you have
reserved the room for that time if there is a conflict. If you have
50-minute-long lectures holding assessments during lecture time can be
limiting. For year-long courses (ECO###Y) you can request time for a midterm
test (2 or 3 hours) during the December examination period by contacting the Undergraduate Assistant
early in the term and the University will schedule your exam and assign a room
(you lose control of the time and date).
·
A&S sets the times
and locations for all final examinations with the schedule published a
couple of months in advance.
·
Final examinations from previous years are publicly
available from the University
of Toronto Libraries Exams and Course Collections. If you wish not to
have your examination made publicly available see “Restricted Exams” in Section
9.4 in the A&S Academic Handbook for Instructors.
·
For term tests, Accommodated
Testing Services (ATS) will contact you (via e-mail) if there are students
in your course that require special accommodations. (In most cases you will
have at least one student requiring accommodation.) For non-online assessments,
they coordinate everything, but you need to upload the details and a copy of
the test to the CIS system.
Writing
at the University of Toronto (an excellent
and comprehensive website for faculty and students)
WIT Program (Writing Integrated
Learning) (no website as of September 6, 2019)
Writing Economics: A
Guide for Harvard Economics Concentrators (August 2021 version)
Percentage marks,
letter marks, and GPA at U of T
80% is an A? (no longer available)
Important
rules regarding term work (Academic Handbook Sections 2.8 – 2.19)
Marks
Distribution Guidelines (Section 10.3 in the Academic Handbook)
·
See the A&S
Academic Handbook for important grading information including the U of
T guidelines on mark distributions.
·
Given that marks can be curved up but not curved down,
there appears an incentive to write assessments that are difficult or
time-pressured to ensure that marks are not too high and then to retroactively
curve the marks up as need be. This is highly unpopular with students.
Even if after the curve the marks are typical, students seem to never forget
(or forgive you for) the initially low un-curved marks. While there are several
possible explanations, one is that students believe that the assessment
reflects your expectations of them and if they do not do well on the assessment
they feel they have fallen short of expectations possibly despite extensive
studying and preparation. The ex-post curve does not change this feeling. Many
students will blame you for this failure because you have not effectively
communicated your expectations to them or are perceived as having expectations
that are impossible for them to meet.
·
Term marks (not inclusive of the final examination)
should give students a good idea of how they are doing in your course. This
means that you should strive to follow the guidelines during the term. It is
not fair to students to give easy term work that is generously graded and then
hit them with a much more difficult final examination that differs from their
expectations based on the term work.
·
Students' official transcripts indicate not only the
student's grade but also the overall course average.
Multiple Choice Questions and Software:
In
2018/19 Crowdmark launched a functional way to include multiple choice questions within a Crowdmark test/exam and have them machine marked, which can replace our
use of the separate A&S bubble forms. This section of this website discusses
our old way using a separate A&S bubble form and having the papers machine
marked.
With
scanned answers, your TA can run the marking software below. I would recommend
having your TA visit this web site (instructions
for your TA (txt)). You will need to provide your TA with a
solution key and a spreadsheet with the list of students registered in your
class. Your TA can set up and execute the code. Your TA needs to have access to
Stata version 10 or higher to do this, which is available at computing labs
around campus. Your TA can give you the automatically produced report
summarizing student performance, a spreadsheet with the marks, and a file that
you can post to your course web site so students can see their answers and
their marks.
Marking
Software: version (April 2014)
I have written software to grade
and analyze student performance on multiple choice questions. This description explains what it can do.
Instructions for your TA that explains
how to do marking
Software to
mark multiple choice question data (STATA do-file) version (April 2014)
Template
for online posting of results for students (Excel 2010 .xlsx)
|
Illustrative example from Fall 2011,
Term Test #1 in ECO220Y Term
Test #1 test paper (form A); Term Test #1 test paper
(form B) |
|
|
Inputs |
Outputs |
|
Term Test #1 solutions (Form A) (pdf
file) Term Test #1 solutions (Form B) (pdf
file) Raw Data from scanning machine (spreadsheet) (note:
names and student numbers are fake) Course List from ACORN (spreadsheet) (note: names and student
numbers are fake) |
Report on student performance on Term Test #1 (text
file) |
This illustrative example uses real
data (but with fake student names and student numbers). For each question, the
report on student performance (sample report)
indicates the percent of students that chose the correct answer, the percent
that picked each of the wrong answers, the percent that chose the correct answer
broken down by quartiles (first quartile, second quartile, third quartile and
fourth quartile based on overall performance on the multiple choice questions),
the Discrimination Index (DI), R-squared (r2) and the slope. The DI, r2
and slope are all statistics that can be used to assess the effectiveness of
each of your questions.
·
The DI is the difference in the percent correct for
the top and bottom quartiles of the class. One measure of the quality of a
particular multiple choice question is the DI: the higher the DI the better the
question is at separating proficient students from incompetent students.
If a question is too easy the DI will be small: proficient students and
incompetent students get it right. If the answer is obvious even if you
don't understand the concept the DI will be small: proficient students and
incompetent students get it right. If the question is way too hard the DI
will be small: proficient students and incompetent students get it wrong. If
students who understand an important concept get the question right but others
get it wrong then the DI will tend to be large: proficient students get it
right and incompetent students get it wrong. If you write a really poor
or misleading question it is possible that you get a negative DI index:
proficient students are misled while incompetent students guess and some get it
right. In general, the larger the DI the better.
·
The R-squared (r2) uses the information in the data
more efficiently than the DI index, which makes it a better measure. The
R-squared for a question tells you what percent of the variation in the
students' overall percentage correct on all other questions is explained by
their performance of this question. A relatively high R-squared indicates that
the question is a good predictor of the students’ performance on the other
questions. This is one sign of a good question. A bad question that
confuses students who know the material would be expected to have a low
R-squared. A hard question where most students guess would again have a
low R-squared.
·
The slope for a question measures the how much higher
on average the percentage correct on all other questions is for students that
got this question correct. A high slope is an indicator of a good
question, whereas a low slope is an indicator of a bad question.
·
If you see a relatively low R-squared and a relatively
large slope that is a sign that the question is relatively easy. Hence
most students got it, but those who missed it really didn't know the material
and also did very poorly on other questions. It is a good idea to try to
include some questions like these to separate the incompetent students (F's)
from the borderline competent students (D's and low C's).
·
If you see a question with a low R-squared and a low
slope, this is a sign that the question might have problems. It means
that this question is a poor predictor of students' success on other questions
and students who got this question right didn't tend to fare better than
average on other questions. This could be consistent with: an overly hard
question where all students (even top students) are guessing, confusing wording
that leads proficient students astray, or an overly obvious question where all
students (even incompetent students) can easily spot the right answer.
You should review the specific question you wrote and also look at which
distracters students selected.
By carefully reading the report,
you can study exactly what happened on each question and potentially identify
questions that did not work well. For well-written questions you can
identify common misconceptions amongst the students if you see numerous
students selecting the same wrong answer.
Basic
tips for writing effective multiple choice questions (pdf)
|
Pros and cons of using multiple choice
questions in your assessments |
|
|
Pros |
Cons |
|
·
You
control marking ·
Ex
post you can decide to accept alternate answers ·
Marking
is objective ·
Marking
is completely consistent across students ·
Marking
is done by machine so that even for hundreds of students it takes only a few
hours total ·
Students
can get quick feedback: each student can have their marked paper
"returned electronically" within hours of the assessment (responses
and marks posted on website) ·
Allows
for more frequent testing: for example, with quizzes ·
Potentially
frees some TA hours for more student contact ·
Gives
very detailed feedback on students’ performance on each question ·
You
"set the agenda" for the assessment so students cannot avoid the
questions (i.e. give long and irrelevant answers) ·
No
opportunity for ex post cheating (i.e. not possible to modify the test paper
and then submit it for a remark) |
·
It
is time consuming to write multiple choice questions ·
Likely
not a useful tool if you have few enough students such that you can mark most
assessments yourself ·
Poorly
written questions lead to extremely frustrated students: see basic tips for writing multiple choice questions ·
Poorly
written questions can "give away" answer: see basic tips for writing multiple choice questions ·
Multiple
choice questions are not suitable for all of the subject matter taught in our
department ·
Multiple
choice questions do not give students a chance to improve their writing
skills ·
Guessing
can impact students marks and create spurious variation across students ·
Some
students complain that there is no partial credit (even though that's not
true in expectation) ·
Because
forms are machine read, students that fail to follow instructions could face
catastrophic consequences (failure) |
Quercus
(login with UTORid)
Quickstart Quercus (a 12 minute
introductory video to Quercus)
Arts & Science Quercus Support –
Faculty
Canvas
Video Guides (these are helpful)
IT IS POSSIBLE TO HAVE A PUBLIC
WEBSITE WITHIN QUERCUS: This video
is quite informative about how to make your site public. Also, if you like to
program in HTML, you can do that from within Quercus. Generally, it is easy to
create clean webpages in Quercus: see the Canvas Guide on the Rich
Content Editor. Hence, you can incorporate a stand-alone site into Quercus
(and still have it public).
MAKING YOUR OWN COURSE WEBSITE:
Instructors that choose to may
continue to use self-maintained web pages (i.e. through the Economics
Department server) and possibly in conjunction with Quercus.
Teaching
Assistants' Training Program (TATP)
Graduate
Student Directory (for e-mail addresses/telephone numbers)
·
TA assignments usually happen around the time that
classes begin and sometimes a bit after they have started.
·
At the beginning of the course you are required to
tell your TAs the breakdown by task of the hours they will spend on the
course. Martin Osborne has created a web-based form you need to fill out
for each TA. Please log-in to our Economics
Department Intranet.
·
The Economics Department expects you to evaluate each
of your TA's at the conclusion of your course: you will receive an e-mail with
links from the Economics Department Associate Chair, Graduate Affairs near the
end of an academic session.
·
Duties typically assigned to TAs include: marking,
conducting review sessions, holding office hours, and invigilation. It is often
advisable to have TAs attend your lectures and especially for those TAs that
will have student contact through review sessions or office hours. Even
for 100 and 200 level courses, the assumption that the TA is sufficiently
familiar with the course material to teach it is often a poor
one. Further, TAs that are not in lecture cannot be expected to know what
you covered each week and what was emphasized or possibly how you taught it
(i.e. with graphs or with or without calculus). Top research universities, such
as Yale, require their TAs to attend all lectures including 100-level courses
and courses the TA has done before. While we do not have an unlimited
number of TA hours, for large enrolment courses we do have enough to require
lecture attendance of at least some of the TAs assigned to the course.
For a half-year course, class attendance uses 24 hours total and for a
full-year course, 48 hours total.
·
If your course has a tutorial hour (third hour per
week), you can use this for weekly TA sessions. This is convenient as
students should not have conflicts during this time and you will automatically
be assigned a classroom. If you have a course that meets only two hours
per week, you can request a third hour for next year by contacting the Economics
Department Associate Chair, Undergraduate Affairs.
·
For weekly TA review sessions (outside of the third
hour of the course) and special review sessions before tests and exams you
should request a room. To request a room, ask the Economics Department Undergraduate Assistant to
book a room for you (make sure to specify the course, sections, number of
students expected to attend, date, and time).
·
For TA office hours or office hours for part-time
instructors that do not have an assigned office, you can e-mail roombookings@economics.utoronto.ca to
reserve time in the Economics Department. You can check the availability of our rooms.
TA's need to pick up the key from Nada during regular business hours.
Alternatively, TA's can meet with students in common areas (such as the
Undergraduate Student Lounge on the main floor of the Economics Department,
which is also open during regular business hours).
·
For accurate marking, it is strongly recommended that
you provide TAs with complete solutions (including alternate solutions that
would also be correct). In addition, you may provide them with marking guidelines
and a marking rubric.
Sidney
Smith Commons (help and support for A&S students, including recognized
study groups)
Policy
on Official Correspondence with Students
First-Year Learning Communities
("Flicks")
Economics
Students' Association (ESA)
Rotman
Commerce Students' Association (RCSA)
ASSU
(St. George Campus)
See the June 2020 NBER Working
Paper My Professor Cares:
Experimental Evidence on the Role of Faculty Engagement.
·
Students are required to have and check a University
of Toronto e-mail address.
·
While it is not required that you communicate with
undergraduate students via e-mail, you are required to clearly state an e-mail
policy and to be otherwise accessible to your students.
university and A&s teaching INitiatives:
A&S Digital Teaching and
Learning (new 2022 Quercus site)
A&S
Teaching & Learning Community of Practice
University
of Toronto Provostial Guidelines on Online Assessment Invigilation
(Exam
sign-in app coming in December 2018 but is yet to arrive for April 2023)
lecturing using PowerPoint (or other software):
When used properly, PowerPoint (or
other slide producing software) can be an effective tool in some branches of
economics. PowerPoint allows you to include more graphs, tables, game trees,
flow charts, data, computer output, etc. Each instructor should choose a
lecture style that works best and PowerPoint is not for every instructor or
every course. Below is some useful information if you decide to use
PowerPoint (or other software such as LaTex) to give lectures at U of T.
Logistics of creating PowerPoint slides and publishing slides:
·
You obtain Microsoft PowerPoint
for FREE.
·
If you publish your slides (i.e. post your lecture
notes online for your students), you also need Adobe to convert the PowerPoint
slides to pdf documents. You can purchase Adobe
Acrobat Professional.
·
When converting PowerPoint slides to pdf documents
using Adobe, it is important that you embed the fonts. Failure to embed
the fonts will mean that some students will not be able to properly view and
print out your lecture notes. This is also useful if you are converting
other documents you want others to be able to reliably read and print. For
Windows users, you can set embedding the fonts (recommended) to the default:
(1) Install Adobe (reboot), (2) Click on start (if have option, choose
Settings), (3) Select "Printers and Faxes", (4) Right click on
"Adobe PDF" , (5) Select "Properties", (6) Click button
"Printing Preferences...", (7) Select the "Layout" tab, (8)
Click button "Advanced...", (9) Under "Graphic", "True
Type Font:" click on what is shown (probably "Substitute with Device
Font") select "Download as Softfont" from the drop down menu
that appears, (10) Click button "OK", (11) Select the "Adobe PDF
Settings" tab, (12) Under "Adobe PDF Conversion Settings",
"Default Settings:" choose "High Quality" from the drop
down menu, (13) UN-check the box "Rely on system fonts only; do not use
document fonts" ("Do not send fonts to "Adobe PDF""),
(14) Click button "Apply", (15) Click button "OK". If
any of this is confusing, you can see pages 4 - 6, How
to PDF document that illustrates most of these steps (in some cases the
instructions are slightly different, follow those given on this web site). You
only need to do this once and not every time you convert a document.
·
To convert a PowerPoint file to pdf: (1) Open the
PowerPoint file you wish to convert, (2) Click on "File", (3) Select
"Print" from the drop down menu, (4) Under "Name:" select
"Adobe PDF" from the drop down menu, (5) If you want to double-check
your settings you can click button "Properties" and you should see
the default settings you set above (i.e. Layout, Advanced, Graphic, Download as
Softfont; Adobe PDF Settings, Default Settings, High Quality; Adobe PDF
Settings, Do not send fonts to "Adobe PDF" is unchecked), (6) Do NOT
check "Print to file" box (even though you will be creating a file
and not a hardcopy), (7) Under "Print what:" choose desired format
(i.e. one slide per page, six slides per page, slides with lines for notes,
etc.), (8) Click button "OK", (9) In Adobe, review the pdf file you have
created and if you need to rotate the image: select Document, Pages, Rotate,
OK, and then if it looks right, File, Save, (10) In Adobe, you can check
whether you have successfully embedded the fonts by selecting File, Document
Properties..., Fonts, and making sure in parentheses you see (Embedded) or
(Embedded Subset) next to each font: for illustration see page
9, How to PDF document.
Logistics of delivering lectures
using a data projector (at St. George campus):
·
For most classrooms you need to bring your own laptop
to class with you. However, if your class meets in a large lecture hall it
might have a U of T
teaching station. If there is a teaching station, you can just plug a
memory stick into one of the USB ports. To access the teaching station you need
to be U of T staff (faculty, sessional or TA) with a valid UTORid and password.
·
To give PowerPoint presentations you will need to
request the use of a data projector in your classroom. The data projector
will either be permanently installed in the classroom (an "electronic
classroom") or it will be wheeled in on a cart prior to each of your
lectures ("portable data projector"). If you will be using
PowerPoint for only some of your lectures, a regular classroom with a portable
data projector will suffice. However, it you plan to use PowerPoint for most or
all of your lectures I strongly recommend that you request an electronic
classroom. You should make this request as early as possible to the Undergraduate
Administrator. Planning for classroom assignments begins in December
for the upcoming academic year. Further, if you are teaching more than
one section back-to-back you should request that you be assigned the same
classroom for those consecutive sections as you need time to set up your laptop
and the data projector. The Department can then try to get your requests
filled with the Office of Space Management (OSM). There is sometimes a
shortage of electronic classrooms, so as a courtesy to your colleagues you
should not request one unless you plan to fully utilize it.
·
Visit your classroom before the first class. Room
assignments are posted at Current A&S Timetable
(regular academic year). Visiting in-person is ideal. You can also check
out the room virtually. (Additionally, rooms will be posted
before the first class on our
website.)
·
Make sure you completely shut down all data projection
equipment when you are done lecturing (unless AV has specifically instructed
you to do otherwise). Leaving this equipment on not only uses up valuable
bulb life, which increases the chance of bulb burn out during lecture, but it
also makes this equipment vulnerable to misuse.
"Art" of effective
PowerPoint use: Some tips, questions, answers:
·
Q: Do we have reliable technical support or will I be
wasting a lot of valuable lecture time trying to get projection equipment
working?
A: In fourteen years and in about 2,500 hours of lecture time (all using
the data projector) I have lost a total of less than 3 hours due to technical
problems. The two worst case scenarios: (1) I had to give the first hour
of lecture on the board because of a complete breakdown of the teaching station
and delayed delivery of a backup and (2) the bulb for the data projector
expired in the middle of lecture and it took 15 minutes before a portable data
projector was delivered and operational. This is not too bad at all: I
have found that the AV support at ACE is very responsive and quick. You
should be proactive though: arrive 10 minutes prior to lecture, have a
demonstration with a technician before first day of class so you know how to
operate everything, and immediately report any problems with the equipment
(especially if the image is not a bright as usual as this is a sign the bulb is
about to go).
·
Q: How about hybrid lectures: part PowerPoint and part
blackboard?
A: Good idea in theory but hard to implement given that most classrooms
have the projection screen directly on top of the black board (white board). A
great alternative is to use a tablet PC with a pen so you can write in
real-time directly on your slides.
·
Q: Isn't it time consuming to create
PowerPoint lectures?
A: Yes, it is fairly time consuming. How time consuming depends a lot on
you and the course material. You may want to try creating a couple of
PowerPoint lectures to see how long it takes.
·
Q: Should I post all of my lecture
notes on my course web site?
A: Understandably, students really, really want the lecture notes
posted. However, to make sure that students attend lectures and keep up with the
course in-between assessments, lectures have to have "value
added." Each instructor can use some creativity to figure out a way
to ensure lectures have "value added" even if all the lecture notes
are posted before lecture such that students can print them out and bring them
to lecture. In my case, have found that peppering the lecture notes with
questions that are only answered during lecture and using iClickers helps
ensure high attendance. There are many possibilities. For an econometrics course
you may present a table of estimation results and discuss the interpretation in
class (rather than write out the interpretation in the slides). For a course
more focused on public policy, you could present the facts and assumptions in
the slides but draw out the policy implications in class. For a theory
course, you could present a model and derive the results in the slides but work
through the intuition and the importance of various assumptions in class. Of
course there are more direct means of ensuring class attendance such as having
a TA pass around a sign-in sheet to take attendance, including "class
participation" in the marking scheme, pop quizzes or other short in-class
exercises. Make it very clear to your students that you expect them to
take notes and attend class.
·
Q: Does using PowerPoint mean that I just read from
slides in class?
A: Try hard not to read your slides aloud in lecture: this is boring and
students get frustrated because they can read faster than you can talk. Focus
on providing context, interpretation, explanations, elaborations, insight, and
encouraging some class room discussion (if appropriate).
·
Q: Does using PowerPoint mean I should change the
content of my course?
A: The means of delivery should not dictate the content. If you feel
that the content of your course is not well-suited to using PowerPoint then
don't use PowerPoint.
Overview of iClickers
at U of T
Ed Discussions (very similar to Piazza,
but allows moderation, whereas Piazza does not: see this chart)
Which
tool should I use for asynchronous discussions? (Support from FASE's
Education Technology Office)
How
to use Piazza in your Quercus Course (Students)
Quizzes
section of the Canvas
Instructor Guide (detailed instructions with screenshots)
New Quizzes
(Quizzes.Next) (Launched in September 2019: STILL NOT RECOMMENDED TO USE NEW QUIZZES AS OF APRIL 2023: STICK WITH
CLASSIC QUIZZES)
See Quizzes
in Quercus, which is a more up-to-date version of the below
You can run online quizzes in Quercus. If you are familiar with Aplia or
MyEconLab, which are for-profit online testing options sometimes packaged with
textbooks, Quercus quizzes are the same idea. However, Quercus quizzes are free
for our students but costly in terms of your time/effort and/or TA hours. In
other words, you or your TAs prepare and input the questions. Starting in
Summer 2015 but ending after 2020/21, I have used weekly online quizzes. I
prepared questions and answers in a plain text document. I e-mailed these to my
Head TA who inputs and sets up the parameters (points, time limit, due date,
etc.) How much work the input part is depends on the number of questions, style
of questions, and whether you want randomization.
Next is Q&A to address likely
questions. However, some answers are TBA because from Summer 2015 through
Summer 2018, we used portal (Blackboard) and we will use Quercus for the first
time in Fall 2018. If you choose to use quizzes, there will invariably be more
you and your TA will need to figure out as you go. Hence, while I believe there
are strong pedagogical benefits, I am certain there are considerable costs.
Quercus quizzes may be the best idea in larger enrolment courses where the
fixed costs are spread over many students and where you have considerable TA
hours and can delegate much of this.
·
Q: What
is the potential value of online quizzes?
·
A: It
can be an efficient way to administer frequent quizzes in your class without
managing paper (i.e. no photocopies, no sorting, no returning papers). I would
NOT recommend it as a substitute for regular tests because online tests are not
invigilated. However, if you are looking for a way to help students regularly
engage in your course – e.g. every week – in addition to regular tests, this is
a potentially good option.
·
Q: What
kind of questions can I ask in Quercus quizzes?
·
A:
Pretty much any kind (although not as many as Blackboard). The Canvas Guide
titled “How
do I create a quiz with individual questions?” gives the options. It can be
“Numerical Answer,” which means that students type a numeric answer. It can be
“Multiple Choice,” which means students select a letter answer (e.g. (A) – (E),
but any number of alternatives are possible). At the other extreme, it can be
“Essay” or “File Upload.”
·
Q: How
do I handle students registered with Accessibility Services who get extra time?
·
A: It is
like for term tests except that usually the course TA (or instructor) adds the
extra time in Quercus (rather than giving external people full editing
privileges in your Quercus course). Almost always these are a multiplicative of
the class’s time limit (e.g. 1.5 times the usual time limit) or a flat addition
of extra minutes (e.g. plus 30 minutes no matter the class’s time limit). For
any special cases (which have been rare in my experience), you should consult
with Accessibility Services and be prepared to be reasonable and flexible. To
grant extra time within a Quercus Quiz see the Canvas Guide titled “Once I publish
a timed quiz, how can I give my students extra time?”
·
Q: I
have many questions about how to create and deploy quizzes in Canvas and I want
detailed instructions with screenshots. Where do I start?
·
A: I
would strongly recommend the Quizzes
section of the Canvas
Instructor Guide. Note: Canvas is currently (as of August 2018) working on
a major upgrade to Quizzes called Quizzes.Next. However, we (as of August 2018)
only have access to Quizzes so you should make sure you look at the Quizzes
section and not the Quizzes.Next section of the guide.
·
Q: Can
I put tables, equations, graphs, etc. in the question?
·
A: TBD
(it was nearly impossible in Blackboard, which had extreme formatting issues,
but maybe it will be possible in Canvas).
·
Q: How are
Quercus quizzes marked?
·
A: It
depends on the type of questions you set. For short answer questions (like
“Numerical Answer” or “Multiple Choice”) you input the correct answer (and
margin of error) with the question to enable automatic marking.
·
Q: What
if I realize there is an error in the quiz after students have already started
taking it?
·
A: You
have a mess to deal with. The current regrade features are extremely limited
and there is nothing to be done (other than manually going through every
submission) if you make many kinds of errors: see “What options
can I use to regrade a quiz in a course?” under Canvas Guides. It looks
like Quizzes.Next (not yet available as of August 2018) will be better on this
front. (Also, there were serious bugs with this functionality with Blackboard
portal that were never fixed after literally years of repeated reporting.)
Unfortunately, you cannot even just delete a problematic question or make its
point value zero (things that you could do in Blackboard portal). If you make
changes to point values or questions after some (or all) students have
submitted this will just create inconsistencies: most things are not updated
(like students grades) but some things are updated (like total possible
points). Hence, at least for now, you need to have an extremely detail-oriented
Head TA working on your online quizzes. Any mistake discovered after some (or
all) students have already submitted will be a real headache.
·
Q: Can
I turn off the big screen suggesting that students submit a comment on their
mark (each of which will trigger an e-mail notification to the instructor)?
·
A: I
think it is not possible disallow comments (or to remove the big invitation to
submit them for each mark). You can change your notification settings: you can
still see comments from with Grades. You can have a regrade policy in your
syllabus that requires a more formal mechanism (e.g. e-mail to Head TA) to
dispute a mark.
·
Q: How
easy is it to use Quercus quizzes?
·
A:
Mostly it is easier than Blackboard portal, but still not easy.
·
Q: Why
should I do all this extra work? Why not just use Aplia or the online question
bank that comes with my textbook?
·
A:
Maybe you should use a paid service instead. It depends. If there is something
already set up and ready-to-go and the questions work well for your course (and
it costs students less than $65
for an H course or less than $130 for a Y course so that you comply with the
Student Ancillary Fees rules), then that may be the best option. Also, some of
these paid services have nice options (like embedded graphs that students can
edit), which would be hard (or in some cases impossible) for you to do
yourself. However, the pre-packaged questions do not align well with your
course, then Quercus quizzes are a good option because you write the questions.
·
Q: What
about simply doing paper-based tests (not online)?
·
A:
First, I would not view Quercus quizzes as a substitute for regular tests
(invigilated and paper-based). Focusing on frequent quizzes and weekly
participation, Crowdmark is also an attractive (paper
based) option. The advantage of Quercus quizzes is that the marking costs for
certain types of questions approach zero.
·
Q: How
much should Quercus quizzes be worth?
·
A:
Obviously it depends. However, it seems like 5% - 10% of the course grade is
often reasonable if students will have to do many quizzes (e.g. 10 to 20).
That’s enough to motive students but not so much as to give an extra big
incentive for cheating.
·
Q: What
about cheating?
·
A: This
is a big concern. Online tests (Quercus quizzes or paid services) are not
invigilated. There are strategies to address potential cheating. There is test
design: e.g. give students random subsets of questions or have randomly
generated numbers in questions so that tests and answers differ across
students. There are collaboration policies: e.g. allowing students to work in
teams. However, after 2020/21 I realized the cheating issues were too large to
allow weekly quizzes to continue.
·
Q: How
long do students have to complete tests?
·
A: You
pick this (and a long list of other options). I like to give students a timer:
e.g. 60 to 90 minutes once you start. This is another method to discourage
cheating (but that is not the only rationale for having a timer).
·
Q: What
if students are late? Miss a Quercus quiz?
·
A: Usual
dilemmas. I like to keep it simple and not allow late submissions and to have a
make-up quiz (open to everyone) that replaces the X lowest marks. Section 11.2 of the 2017/18 ECO220Y syllabus
shows a possible policy.
·
Q: How
do students review their work after it is marked?
·
A: It’s
obvious for students. They can either go to the Quizzes area or the Grades
area. Also, they get notifications.
·
Q: What
about partial credit for trying (participation points)?
A: Not
possible within Quizzes tool in Canvas.
·
Q: I’m
thinking about trying Quercus quizzes in my class. How do I set them up?
·
A: For
full instructions (with screenshots), see official support sites linked above.
Below I give the specific steps I’ll ask my Head TA to follow (which include a
bunch of specific decisions based on quiz parameters I like to set). Also, I
would strongly recommend delegating the task of inputting and checking
questions to a Head TA and maintaining your own master file of questions in a format/organization
separate from any specific technology.
Example
(from ECO220Y1Y) of steps to run a quiz in Quercus:
1. Enter
the Quercus site for our course.
2. Click
Quizzes on the main menu.
3. Click
the + Quiz button.
4. Add
the official name of the quiz (e.g. “Quercus Quiz 1” or “DACM Module A Online
Test”) and copy-and-paste the quiz instructions.
5. Leave
defaults for Quiz Type (Graded Quiz) and Assignment Group (Assignments).
6. Leave
Shuffle Answers unchecked. (However, if the quiz includes any multiple choice
questions then please DO check this box.)
7. Check
the box for Time Limit and enter the minutes (e.g. 60 or 90).
8. Leave
the box for Allow Multiple Attempts unchecked.
9. Uncheck
the box for Let Students See Their Quiz Responses (Incorrect Questions Will Be
Marked in Student Feedback). (Note: After
the quiz window closes we will change this so students can see the questions,
their replies, and the correct answers.)
10. Leave
the box for Show one question at a time unchecked.
11. Leave
Quiz Restrictions unchecked.
12. Leave
defaults for Assign to (Everyone).
13. Enter
the exact due date and due time. Double-check your entry (making sure that am
or pm is correctly specified).
14. Enter
the exact date and time the quiz becomes available. Enter the due date and time
(again) for the end date of availability. (This means that we do not permit
late submissions.) This step is very important: double-check that everything is
exactly correct before proceeding.
15. Click
Save & Publish. (I know it seems crazy to “publish” the quiz before we have
even entered the questions, but we have to in order to add accommodations for
individual students. However, students will not be able to see this quiz until
the date set under Available from. It is VERY IMPORTANT that you correctly set
the Available from and Until fields before taking this step.)
16. Make
sure the grades are hidden from students until you are ready to release them.
In mid-summer 2019 A&S changed how this is done. It is confusing. I’d
recommend that instructors go into Gradebook and click the settings gear icon
(top right). Under the Grade Posting Policy tab select Manually Post Grades.
This way you can control when to release them.
17. Go
back to Quizzes and click on the quiz you are inputting. In top right, you will
see Moderate This Quiz. Click that. Click the pencil for the individual
students getting extra time and enter the number of EXTRA minutes each gets
over and above the class’s time limit (not the total). For example, for a 60
minute quiz, if the accommodation is 1.25 time, enter 15. Leave the field for
Extra Attempts blank and leave unchecked the box for Manually unlock…
18. Go
back to Quizzes and click on the quiz you are inputting. Click the Edit button
and then click the Questions tab.
19. At
the bottom of the screen, click the + New Question Group button. Enter the
question group title in the .txt file I sent (without the ALL CAPS part). (This
is for us: students cannot see the name of the group.) Unless otherwise
instructed, leave the default to Pick 1 question and enter the pts per question
specified in the.txt file. Click the Create Group button.
20. To
add a question to the group, click the + in the top right of the frame for that
question group. Leave the name of the question version as the default
(Question) and choose the question type from the drop-down menu. Copy and paste
the question from the .txt file (keyboard shortcuts work). Some things you need
to know about the three types of questions we use most often:
a. Numerical Answer. For answers, we
use Exact Answer (the default). Copy and paste the correct answer and the error
margin: keyboard shortcuts for paste do not work for the answer and margin of
error fields: right-click and use the drop down menu. NOTE: Canvas can accept
numeric answers with at most four
decimal places: if you (or a student taking a quiz) try to input a more
precise answer, it automatically rounds it to four decimal places and shows
four decimal places (e.g. if a student types 2.05 it will auto-convert to
2.0500). Hence, for questions requiring precision of four decimal places the
margin of error is zero (which will appear to students as “0.0”). (Also, when
Canvas display the correct answer for students it removes trailing zeros.) Using
the garbage can, delete the other exact answer fields. (It is hard to imagine a
question where there would be more than one exact numeric answer, but by
default Canvas allows 4 different exact answers.) Do this right away for the
first question in the group because when you go to add the next question
version for that group it will only have one exact answer field by default. IMPORTANT: You must delete those extra
answers. By default it will list all of them for students when they get results
and it says that 0.0 is an accepted answer: this it is very confusing for
students AND a student that entered 0 for every question would get a 100%!
Click the Update Question button at the bottom.
b. Fill in Multiple Blanks. For Fill in
Multiple Blanks, after you paste from the .txt file, which already is correctly
formatted, you enter the correct answer for each blank using a drop-down menu.
(It is weird and inconvenient that it is a drop-down menu, but it is.)
Fortunately, these are not case sensitive (e.g. if the answer is “A” and a
student enters “a” it is marked correct). Also, it does not mark things
incorrect because of leading or training blanks (which is also good). Also,
unlike the numerical answer case (above), you do not have to delete the
extraneous alternative answers. These are blank by default, but Canvas DOES
mark blank answers incorrect (so even if you do not delete the blank
alternates, students will not be marked correct for entering blanks).
c. Multiple Choice. For multiple
choice, you do NOT have to put the correct answer first: enter them in the same
order as the .txt file and then click the correct answer (by default it clicks
the first choice as the correct answer so you must click the actual correct
answer). Note: To randomize the order of the answers for all multiple choice
questions in your quiz, after creating the quiz, under Details, scroll down to
Options and check the box "Shuffle Answers."
d. Any type of question where you wish to give students
STATA results (e.g. regression, summary) with proper alignment. It
is easier to include output in Canvas questions than in Blackboard, but it is
still not obvious. For example, suppose you wish to input this question (in plain
text format, .txt file) as a quiz question. This link shows
how to do it (with screenshots).
e. Any type of question where you wish to give students a
table with proper formatting cut-and-paste from a .csv or .xlsx file. For
example, suppose you wish to input this
table (in .csv format) or this table
(in .xlsx format) as part of a quiz question. This link
shows how to do it (with screenshots). (You could also use the HTML editor if
you’re into that: see “Making
table width automatically resize on a Canvas content page”.)
21. After
carefully entering all question groups and questions and checking over your
work, click the Save button.
22. Click
into Quizzes and click the title of this quiz. Make sure the total points and
other parameters are showing correctly. Scroll down and click the Preview
button. Make sure the questions appear as intended.
23. Once
you have double-checked everything, e-mail me to let me know you have the quiz
set up (at least two days before it opens for students).
24. After
the submission window, deal with any outstanding student quiz submissions. See
“How do I
manually submit outstanding student quiz submissions?” in the Canvas
Guides.
25. We
will have to figure out how to run an item analysis. Canvas does not currently
have the ability to run standard analyses except in very limited circumstances
(multiple choice and true/false questions only). After the quiz, under Quiz
Statistics we can run Student Analysis and download the .csv data and write
software to compute our own statistics.
26. What
if there are any errors in the quiz that we discover after some students have
already submitted (and/or after the quiz is over)? It is really hard to deal
with this because Canvas has almost no options to regrade (we cannot just
delete a problematic question or update its answer). First, we should try never
to let this happen. If it does, it seems there are some costly fixes. We can
use SpeedGrader, which despite its name means manually scrolling through every
submission and changing the points earned by hand for a problematic question
(e.g. to give everyone marks or to accept the actual correct answer). Further,
this does not update the fact that students will see that they have been marked
incorrect on that question even though they earned full marks (hence it will be
confusing for students). Under Settings, Options there is an option to Grade by
question (beta), which may make it a little easier (not have to scroll down
each submission to get to the problematic question). However, I cannot pretest
this functionality without actual multiple submissions.
27. After
we have checked over the results and we are ready to release results to
students, go to the Quizzes page and click on the title of the quiz. Click the
Edit button in the top right. Scroll down and check the box for Let Students
See Their Quiz Responses (Incorrect Questions Will Be Marked in Student
Feedback). Check the box (that now appears) for Let Students See The Correct
Answers. Leave the dates fields blank. Scroll to the bottom and click the Save
button. Go to Grades, mouse over the title of the quiz, click on Grade Posting
Policy from the dropdown menu. Click the Automatically button.
How do I connect my Quercus course to Crowdmark?
(NICE! Posted March 23, 2023)
See Crowdmark,
which is a more up-to-date version of the below
Some collected wisdom in
Economics of using Crowdmark by Jennifer Murdock (refresh for the Sept 5,
2019 version)
Using
Crowdmark with Quercus by Jennifer Murdock (refresh for the Sept 4, 2019
version)
Multiple
choice questions in Crowdmark
Using
Crowdmark in Arts & Science
What is Crowdmark?
Starting in 2014/15, select courses
in Economics have utilized Crowdmark to manage term work electronically (returning
term work to students electronically). Here is an Introduction to
Crowdmark. Also, for first time users, a good place to start is on
Crowdmark’s “What can Crowdmark
do?” page. For how to use Crowdmark for a Quercus course, see Using Crowdmark with Quercus. It may help to see
a sample Crowdmark test from an Economics course with 500 students. Each
student gets a unique test paper (i.e. #1 - #500): here is paper #1 and the separate supplement (not through Crowdmark).
Why use Crowdmark?
There are many distinct advantages of Crowdmark that add up to a big
advantage over the traditional paper-based approach. Some most valued by those
in the Economics Department:
·
Improved student access to their own term work (no
lost papers; convenient electronic copies): surveys of our students show that
students really value Crowdmark
·
Greatly improves the efficiency of having TAs
specialize in subsets of questions (which is great for accuracy, consistency
and speed): for example, you can have 8 TAs each specializing in one question
or parts of a question without wasting time passing around papers and flipping
through them to get to Question (3)(d) (for example)
·
Instructors can see marking progress of TA team in
real time and see how many hours each TA spent on marking
·
Marking is portable: TAs can mark even if they are off
campus or abroad
·
Substantial savings of TA hours
o
Marking is more efficient and especially for tests
marked by a team of TAs
§ TAs
do not need to pass papers around to each other
o
Papers do not need to be sorted
o
Marks entry is faster (and more accurate)
·
More accurate marks entry (and no need to write subtotals
on front of test papers, which are very prone to clerical errors)
·
Greater ability of students to check for and
efficiently report clerical errors
·
Saving valuable class/tutorial time because do not
have to return papers to students
·
Faster return of students’ work: can send results to
students as soon as marking is complete and do not need to wait until the next
class/tutorial
·
Allows instructors on-going and convenient access to
all submitted term work (including for reviewing TA marking, preparing letters
of recommendation, etc.)
·
Elimination of the possibility of ex post cheating
(falsified re-mark requests)
·
Allows students to write term work entirely in pencil
and use an eraser as needed, which is better for test-takers and markers
·
More efficient handling of re-mark requests
·
Potential for more comments on student work: online
marking makes it sensible to invest in preparing good comments for common
errors that can be easily reused
Also, you may request a copy of
“Crowdmark Report: Select ECO Courses” by Jennifer Murdock, Economics
Department, University of Toronto, August 12, 2015 by e-mail (jennifer.murdock@utoronto.ca).
How can I use Crowdmark in my
course?
You will receive an e-mail near the
start of the term with procedures.
How do I actually start using
Crowdmark?
While using a new technology can be
daunting and especially when producing your tests on a deadline, using
Crowdmark is quick and easy. The steps you will need to take to enable Crowdmark
within your Quercus site are explained in Using
Crowdmark with Quercus. You can visit Crowdmark’s
help page and they are anxious to help you personally: just e-mail them at support@crowdmark.com. They respond
very quickly and will walk you through step-by-step over the phone if need be.
Also, here is some collected
wisdom in Economics of using Crowdmark after several years of use, which
will help you with some common practical questions about using Crowdmark. (Note
that you cannot have students write any answers to be marked on the first page:
the first page is used to collect student identifying information).
How many TA hours do I need to
allocate for scanning and matching in Crowdmark?
For scanning the papers, a rule of thumb is 5 minutes per 100 pieces
of paper (our scanner scans both sides simultaneously) or, equivalently, 1,200
pieces of paper per hour. Hence for a test paper with 8 pages (4 sheets of
paper) and 500 students, scanning would take about 100 minutes. This includes
time to correct scanning errors (stuck pages). It helps to neatly stack papers
before scanning to minimize scanning errors. Without scanning errors it takes
about 3 minutes per 100 sheets of paper.
For matching the student names, including double-checking the matches,
a rule of thumb is 10 – 12 minutes per 100 students. Hence, for a class
of 500 students, matching would take a bit less than an hour. However, if
students do not correctly fill in their identifying information, it can take up
to 25 minutes per 100 students to ensure 100% accurate matching and a class of
500 students could take closer to 2 hours. As an alternative to using TA hours,
please see you may consider using the Instructional Support Staff in Economics
(new for 2022, the St George campus only) for scanning, uploading, and matching
for Crowdmark.
Generative
Artificial Intelligence Policies and Resources by A&S Digital Teaching
and Learning (Don Boyes)
ChatGPT
and Generative AI in the Classroom by Office of the Vice-Provost,
Innovations in Undergraduate Education (Susan McCahan)