Teaching

· You can view the course outlines of your colleagues on the undergraduate course time tables at St. George (and sometimes via a link from the faculty member's own homepage).

· ***Your course outline must be uploaded to our departmental website: http://www.economics.utoronto.ca/index.php/index/teaching/coursesForUser***

· The syllabus and the first day of class are your chance to communicate your expectations to your students. Make your expectations clear and precise for a smoother course and improved student satisfaction.

· Usually you may require students who miss an assessment due to illness to provide a U of T Verification of Student Illness or Injury Form, but notice that that page has special instructions about ACORN declarations for 2020/21, 2021/22, and 2022/23. (As of July 13, 2023 there is no information for 2023/24.)

· Visit your classroom before the first class. Room assignments are posted at Timetable Builder. Visiting in-person is ideal. You can also check out the room virtually. (Additionally, rooms will be posted before the first class on our website.)

Developing as an instructor:

Teaching and Learning Community of Practice (CoP), Department of Economics

Conference on Teaching and Research in Economics Education (CTREE) (in person and since 2011)

TeachECONference (online and since 2020)

EdTech Office for the Faculty of Engineering (all U of T faculty welcome & often relevant for Economics)
Center for Teaching Support & Innovation (CTSI) Programming and Teaching & Learning Symposium (annually)

· You may periodically check The Journal of Economic Education. From a U of T computer, you get full electronic access directly at http://www.tandfonline.com/toc/vece20/current.

· CTSI gives seminars and courses to help you in your professional development as a university teacher. Visit their web site to sign up. Subscribe to the CTSI listserv to ensure you receive timely information about upcoming events, workshops, lunch-time roundtable discussions, etc. To subscribe, you may send an e-mail to CTSI asking them to add you.

· You may choose to include a mid-course review of your own design. Students typically given more substantive and constructive comments on these than end-of-term evaluations. CTSI has a booklet entitled "Gathering Formative Feedback with Mid-Course Evaluations". MS Forms is a useful, easy, quick, and institutionally supported tool that lets you create a professional looking online survey and easy-to-review results.

· You may consider joining (subscribing) to the Society for Teaching and Learning in Higher Education (STLHE), which is a Canadian association dedicated to the advancement of teaching and learning, or subscribing to a teaching newsletter such as The Teaching Professor, which will help you keep up-to-date with developments in teaching and learning and keep your courses on the cutting edge without taking up much time.

Teaching Evaluation:

Student evaluations:

Course evaluation framework for all of U of T

Online searchable database of undergraduate course evaluation results since 2012/13 on Quercus: click “View Past Evaluation Results”

“U of T Hub: A course selection tool for new and returning students” a September 26, 2021 article in The Varsity discussing https://uofthub.ca/

ASSU Anti-Calendar (St. George Campus) (published summaries of faculty members' student survey results; the last publication is Summer 2012)

Copy of blank student survey form (used until 2012/13) (St. George Campus)

Policies on assessing teaching:

A&S Faculty Resources (links for PTR, Continuing Status Review, Tenure Review)

A&S Guidelines and Procedures for the Assessment of Teaching Stream Faculty (for Probationary, Continuing Status and Promotion Reviews) (Also, Teaching Evaluation Guidelines across U of T)

Teaching Evaluation for Promotion and Tenure (Provost's Guidelines; general for U of T)

A&S Guidelines for the Assessment of Effectiveness of Teaching in Tenure and Promotion Decisions (Research stream; St. George Campus)

academic integrity:

Arts & Science Academic Integrity System

Note: To log in to the SAI to report cases, you must use VPN even if you are on campus using your U of T computer. Using VPN, connect to general.vpn.utoronto.ca and then do your UTORid authentication.

A&S Student Academic Integrity (SAI) (link you can give your students in you’re downtown)

Academic Integrity at the University of Toronto

Plagiarism Detection Tool

tests/examS, ETC.:

Important rules regarding term work (Academic Handbook Sections 2.8 – 2.19)

University of Toronto Libraries Exams and Course Collections A complete repository of old final examinations across all three campuses

See Using multiple choice questions on this website

Academic accommodations for students with disabilities

University of Toronto Provostial Guidelines on Online Assessment Invigilation

· At U of T there is a major distinction between term work (which is anything you administer during the term such as term tests, essays, etc.) and final examinations (which you must assist in but you are not in charge of administering it: i.e. you do not control the date, illness policies, or the reporting of results for final examinations to students)

· Scheduling your term tests and obtaining appropriate testing rooms can be challenging.

o To avoid student conflicts with other courses schedule tests during lecture or tutorial time. Note: Section 6.3 of A&S Academic Handbook: “If a student has a conflict between a course holding a test outside its normal class hours and a test or required obligation for a class regularly scheduled into that hour, the regularly-scheduled academic obligation has precedence. The course with the irregularly-scheduled test must accommodate the student in some appropriate way.”

o Avoid dates with religious observances.

o If you are teaching a 200-level course, coordinate with other instructors on term test dates as many students are simultaneously enrolled in intermediate micro (ECO200/204/206), quantitative methods (ECO220/227) and intermediate macro (ECO202/208/209). All streams of intermediate micro should strive to overlap on test dates (as these courses are exclusions for each other so no student could have a conflict). Similarly, all streams of intermediate macro should strive to overlap on test dates. We should strive for minimal overlap in test dates between micro, quantitative methods, and macro.

o Typically, your regular lecture room is not appropriate for testing. To request a room, ask the Economics Department Undergraduate Assistant to book a room for you (make sure to specify the course, sections, number of students taking the test, date, and time). Do this early (before classes start) to ensure that it is possible for you to get an appropriate room at the requested time: sometimes the University will not accommodate you. You will receive a confirmation e-mail: bring a copy to the room as proof that you have reserved the room for that time if there is a conflict. If you have 50-minute-long lectures holding assessments during lecture time can be limiting. For year-long courses (ECO###Y) you can request time for a midterm test (2 or 3 hours) during the December examination period by contacting the Undergraduate Assistant early in the term and the University will schedule your exam and assign a room (you lose control of the time and date).

· A&S sets the times and locations for all final examinations with the schedule published a couple of months in advance.

· Final examinations from previous years are publicly available from the University of Toronto Libraries Exams and Course Collections. If you wish not to have your examination made publicly available see “Restricted Exams” in Section 9.4 in the A&S Academic Handbook for Instructors.

· For term tests, Accommodated Testing Services (ATS) will contact you (via e-mail) if there are students in your course that require special accommodations. (In most cases you will have at least one student requiring accommodation.) For non-online assessments, they coordinate everything, but you need to upload the details and a copy of the test to the CIS system.

WRITING:

Writing at the University of Toronto (an excellent and comprehensive website for faculty and students)

WIT Program (Writing Integrated Learning) (no website as of September 6, 2019)

Writing Economics: A Guide for Harvard Economics Concentrators (August 2021 version)

Grading:

Percentage marks, letter marks, and GPA at U of T

Important rules regarding term work (Academic Handbook Sections 2.8 – 2.19)

Marks Distribution Guidelines (Section 10.3 in the Academic Handbook)

· See the A&S Academic Handbook for important grading information including the U of T guidelines on mark distributions.

· Given that marks can be curved up but not curved down, there appears an incentive to write assessments that are difficult or time-pressured to ensure that marks are not too high and then to retroactively curve the marks up as need be. This is highly unpopular with students. Even if after the curve the marks are typical, students seem to never forget (or forgive you for) the initially low un-curved marks. While there are several possible explanations, one is that students believe that the assessment reflects your expectations of them and if they do not do well on the assessment they feel they have fallen short of expectations possibly despite extensive studying and preparation. The ex-post curve does not change this feeling. Many students will blame you for this failure because you have not effectively communicated your expectations to them or are perceived as having expectations that are impossible for them to meet.

· Term marks (not inclusive of the final examination) should give students a good idea of how they are doing in your course. This means that you should strive to follow the guidelines during the term. It is not fair to students to give easy term work that is generously graded and then hit them with a much more difficult final examination that differs from their expectations based on the term work.

· Students' official transcripts indicate not only the student's grade but also the overall course average.

Multiple Choice Questions and Software:

In 2018/19 Crowdmark launched a functional way to include multiple choice questions within a Crowdmark test/exam and have them machine marked, which can replace our use of the separate A&S bubble forms. This section of this website discusses our old way using a separate A&S bubble form and having the papers machine marked.

With scanned answers, your TA can run the marking software below. I would recommend having your TA visit this web site (instructions for your TA (txt)). You will need to provide your TA with a solution key and a spreadsheet with the list of students registered in your class. Your TA can set up and execute the code. Your TA needs to have access to Stata version 10 or higher to do this, which is available at computing labs around campus. Your TA can give you the automatically produced report summarizing student performance, a spreadsheet with the marks, and a file that you can post to your course web site so students can see their answers and their marks.

Marking Software: version (April 2014)

I have written software to grade and analyze student performance on multiple choice questions. This description explains what it can do.

Instructions for your TA that explains how to do marking

Software to mark multiple choice question data (STATA do-file) version (April 2014)

Template for online posting of results for students (Excel 2010 .xlsx)

Illustrative example from Fall 2011, Term Test #1 in ECO220Y Term Test #1 test paper (form A); Term Test #1 test paper (form B)

Inputs

Outputs

Term Test #1 solutions (Form A) (pdf file)

Term Test #1 solutions (Form B) (pdf file)

Raw Data from scanning machine (spreadsheet) (note: names and student numbers are fake)

Course List from ACORN (spreadsheet) (note: names and student numbers are fake)

Term Test #1 marking software (Stata do-file)

Report on student performance on Term Test #1 (text file)
Marks on Term Test #1 (spreadsheet)
Histogram of marks on Term Test #1 (emf) (pdf)
Marks on Term Test #1 to post (unformatted spreadsheet)
Marks on Term Test #1 to post (formatted spreadsheet) (pdf)

This illustrative example uses real data (but with fake student names and student numbers). For each question, the report on student performance (sample report) indicates the percent of students that chose the correct answer, the percent that picked each of the wrong answers, the percent that chose the correct answer broken down by quartiles (first quartile, second quartile, third quartile and fourth quartile based on overall performance on the multiple choice questions), the Discrimination Index (DI), R-squared (r2) and the slope. The DI, r2 and slope are all statistics that can be used to assess the effectiveness of each of your questions.

· The DI is the difference in the percent correct for the top and bottom quartiles of the class. One measure of the quality of a particular multiple choice question is the DI: the higher the DI the better the question is at separating proficient students from incompetent students. If a question is too easy the DI will be small: proficient students and incompetent students get it right. If the answer is obvious even if you don't understand the concept the DI will be small: proficient students and incompetent students get it right. If the question is way too hard the DI will be small: proficient students and incompetent students get it wrong. If students who understand an important concept get the question right but others get it wrong then the DI will tend to be large: proficient students get it right and incompetent students get it wrong. If you write a really poor or misleading question it is possible that you get a negative DI index: proficient students are misled while incompetent students guess and some get it right. In general, the larger the DI the better.

· The R-squared (r2) uses the information in the data more efficiently than the DI index, which makes it a better measure. The R-squared for a question tells you what percent of the variation in the students' overall percentage correct on all other questions is explained by their performance of this question. A relatively high R-squared indicates that the question is a good predictor of the students’ performance on the other questions. This is one sign of a good question. A bad question that confuses students who know the material would be expected to have a low R-squared. A hard question where most students guess would again have a low R-squared.

· The slope for a question measures the how much higher on average the percentage correct on all other questions is for students that got this question correct. A high slope is an indicator of a good question, whereas a low slope is an indicator of a bad question.

· If you see a relatively low R-squared and a relatively large slope that is a sign that the question is relatively easy. Hence most students got it, but those who missed it really didn't know the material and also did very poorly on other questions. It is a good idea to try to include some questions like these to separate the incompetent students (F's) from the borderline competent students (D's and low C's).

· If you see a question with a low R-squared and a low slope, this is a sign that the question might have problems. It means that this question is a poor predictor of students' success on other questions and students who got this question right didn't tend to fare better than average on other questions. This could be consistent with: an overly hard question where all students (even top students) are guessing, confusing wording that leads proficient students astray, or an overly obvious question where all students (even incompetent students) can easily spot the right answer. You should review the specific question you wrote and also look at which distracters students selected.

By carefully reading the report, you can study exactly what happened on each question and potentially identify questions that did not work well. For well-written questions you can identify common misconceptions amongst the students if you see numerous students selecting the same wrong answer.

Basic tips for writing effective multiple choice questions (pdf)

Pros and cons of using multiple choice questions in your assessments

Pros

Cons

· You control marking

· Ex post you can decide to accept alternate answers

· Marking is objective

· Marking is completely consistent across students

· Marking is done by machine so that even for hundreds of students it takes only a few hours total

· Students can get quick feedback: each student can have their marked paper "returned electronically" within hours of the assessment (responses and marks posted on website)

· Allows for more frequent testing: for example, with quizzes

· Potentially frees some TA hours for more student contact

· Gives very detailed feedback on students’ performance on each question

· You "set the agenda" for the assessment so students cannot avoid the questions (i.e. give long and irrelevant answers)

· No opportunity for ex post cheating (i.e. not possible to modify the test paper and then submit it for a remark)

· It is time consuming to write multiple choice questions

· Likely not a useful tool if you have few enough students such that you can mark most assessments yourself

· Poorly written questions lead to extremely frustrated students: see basic tips for writing multiple choice questions

· Poorly written questions can "give away" answer: see basic tips for writing multiple choice questions

· Multiple choice questions are not suitable for all of the subject matter taught in our department

· Multiple choice questions do not give students a chance to improve their writing skills

· Guessing can impact students marks and create spurious variation across students

· Some students complain that there is no partial credit (even though that's not true in expectation)

· Because forms are machine read, students that fail to follow instructions could face catastrophic consequences (failure)

Course Web Sites:

Quercus (login with UTORid)

Quickstart Quercus (a 12 minute introductory video to Quercus)

Quercus Support Resources

Arts & Science Quercus Support – Faculty

Canvas Video Guides (these are helpful)

IT IS POSSIBLE TO HAVE A PUBLIC WEBSITE WITHIN QUERCUS: This video is quite informative about how to make your site public. Also, if you like to program in HTML, you can do that from within Quercus. Generally, it is easy to create clean webpages in Quercus: see the Canvas Guide on the Rich Content Editor. Hence, you can incorporate a stand-alone site into Quercus (and still have it public).

MAKING YOUR OWN COURSE WEBSITE:

Instructors that choose to may continue to use self-maintained web pages (i.e. through the Economics Department server) and possibly in conjunction with Quercus.

Teaching Assistants (TA's):

Teaching Assistants' Training Program (TATP)

Graduate Student Directory (for e-mail addresses/telephone numbers)

· TA assignments usually happen around the time that classes begin and sometimes a bit after they have started.

· At the beginning of the course you are required to tell your TAs the breakdown by task of the hours they will spend on the course. Martin Osborne has created a web-based form you need to fill out for each TA. Please log-in to our Economics Department Intranet.

· The Economics Department expects you to evaluate each of your TA's at the conclusion of your course: you will receive an e-mail with links from the Economics Department Associate Chair, Graduate Affairs near the end of an academic session.

· Duties typically assigned to TAs include: marking, conducting review sessions, holding office hours, and invigilation. It is often advisable to have TAs attend your lectures and especially for those TAs that will have student contact through review sessions or office hours. Even for 100 and 200 level courses, the assumption that the TA is sufficiently familiar with the course material to teach it is often a poor one. Further, TAs that are not in lecture cannot be expected to know what you covered each week and what was emphasized or possibly how you taught it (i.e. with graphs or with or without calculus). Top research universities, such as Yale, require their TAs to attend all lectures including 100-level courses and courses the TA has done before. While we do not have an unlimited number of TA hours, for large enrolment courses we do have enough to require lecture attendance of at least some of the TAs assigned to the course. For a half-year course, class attendance uses 24 hours total and for a full-year course, 48 hours total.

· If your course has a tutorial hour (third hour per week), you can use this for weekly TA sessions. This is convenient as students should not have conflicts during this time and you will automatically be assigned a classroom. If you have a course that meets only two hours per week, you can request a third hour for next year by contacting the Economics Department Associate Chair, Undergraduate Affairs.

· For weekly TA review sessions (outside of the third hour of the course) and special review sessions before tests and exams you should request a room. To request a room, ask the Economics Department Undergraduate Assistant to book a room for you (make sure to specify the course, sections, number of students expected to attend, date, and time).

· For TA office hours or office hours for part-time instructors that do not have an assigned office, you can e-mail roombookings@economics.utoronto.ca to reserve time in the Economics Department. You can check the availability of our rooms. TA's need to pick up the key from Nada during regular business hours. Alternatively, TA's can meet with students in common areas (such as the Undergraduate Student Lounge on the main floor of the Economics Department, which is also open during regular business hours).

· For accurate marking, it is strongly recommended that you provide TAs with complete solutions (including alternate solutions that would also be correct). In addition, you may provide them with marking guidelines and a marking rubric.

students & communication:

Sidney Smith Commons (help and support for A&S students, including recognized study groups)

Policy on Official Correspondence with Students

Rotman Commerce Program

First-Year Learning Communities ("Flicks")

Economics Students' Association (ESA)

Rotman C ommerce Students' Association (RCSA)

ASSU (St. George Campus)

See the June 2020 NBER Working Paper My Professor Cares: Experimental Evidence on the Role of Faculty Engagement.

· Students are required to have and check a University of Toronto e-mail address.

· While it is not required that you communicate with undergraduate students via e-mail, you are required to clearly state an e-mail policy and to be otherwise accessible to your students.

university and A&s teaching INitiatives:

A&S Digital Teaching and Learning (new 2022 Quercus site)

A&S Teaching & Learning Community of Practice

University of Toronto Provostial Guidelines on Online Assessment Invigilation

(Exam sign-in app coming in December 2018 but is yet to arrive for April 2023)

lecturing using PowerPoint (or other software):

When used properly, PowerPoint (or other slide producing software) can be an effective tool in some branches of economics. PowerPoint allows you to include more graphs, tables, game trees, flow charts, data, computer output, etc. Each instructor should choose a lecture style that works best and PowerPoint is not for every instructor or every course. Below is some useful information if you decide to use PowerPoint (or other software such as LaTex) to give lectures at U of T.

Logistics of creating PowerPoint slides and publishing slides:

· You obtain Microsoft PowerPoint for FREE.

· If you publish your slides (i.e. post your lecture notes online for your students), you also need Adobe to convert the PowerPoint slides to pdf documents. You can purchase Adobe Acrobat Professional.

· When converting PowerPoint slides to pdf documents using Adobe, it is important that you embed the fonts. Failure to embed the fonts will mean that some students will not be able to properly view and print out your lecture notes. This is also useful if you are converting other documents you want others to be able to reliably read and print. For Windows users, you can set embedding the fonts (recommended) to the default: (1) Install Adobe (reboot), (2) Click on start (if have option, choose Settings), (3) Select "Printers and Faxes", (4) Right click on "Adobe PDF" , (5) Select "Properties", (6) Click button "Printing Preferences...", (7) Select the "Layout" tab, (8) Click button "Advanced...", (9) Under "Graphic", "True Type Font:" click on what is shown (probably "Substitute with Device Font") select "Download as Softfont" from the drop down menu that appears, (10) Click button "OK", (11) Select the "Adobe PDF Settings" tab, (12) Under "Adobe PDF Conversion Settings", "Default Settings:" choose "High Quality" from the drop down menu, (13) UN-check the box "Rely on system fonts only; do not use document fonts" ("Do not send fonts to "Adobe PDF""), (14) Click button "Apply", (15) Click button "OK". If any of this is confusing, you can see pages 4 - 6, How to PDF document that illustrates most of these steps (in some cases the instructions are slightly different, follow those given on this web site). You only need to do this once and not every time you convert a document.

· To convert a PowerPoint file to pdf: (1) Open the PowerPoint file you wish to convert, (2) Click on "File", (3) Select "Print" from the drop down menu, (4) Under "Name:" select "Adobe PDF" from the drop down menu, (5) If you want to double-check your settings you can click button "Properties" and you should see the default settings you set above (i.e. Layout, Advanced, Graphic, Download as Softfont; Adobe PDF Settings, Default Settings, High Quality; Adobe PDF Settings, Do not send fonts to "Adobe PDF" is unchecked), (6) Do NOT check "Print to file" box (even though you will be creating a file and not a hardcopy), (7) Under "Print what:" choose desired format (i.e. one slide per page, six slides per page, slides with lines for notes, etc.), (8) Click button "OK", (9) In Adobe, review the pdf file you have created and if you need to rotate the image: select Document, Pages, Rotate, OK, and then if it looks right, File, Save, (10) In Adobe, you can check whether you have successfully embedded the fonts by selecting File, Document Properties..., Fonts, and making sure in parentheses you see (Embedded) or (Embedded Subset) next to each font: for illustration see page 9, How to PDF document.

Logistics of delivering lectures using a data projector (at St. George campus):

· For most classrooms you need to bring your own laptop to class with you. However, if your class meets in a large lecture hall it might have a U of T teaching station. If there is a teaching station, you can just plug a memory stick into one of the USB ports. To access the teaching station you need to be U of T staff (faculty, sessional or TA) with a valid UTORid and password.

· To give PowerPoint presentations you will need to request the use of a data projector in your classroom. The data projector will either be permanently installed in the classroom (an "electronic classroom") or it will be wheeled in on a cart prior to each of your lectures ("portable data projector"). If you will be using PowerPoint for only some of your lectures, a regular classroom with a portable data projector will suffice. However, it you plan to use PowerPoint for most or all of your lectures I strongly recommend that you request an electronic classroom. You should make this request as early as possible to the Undergraduate Administrator. Planning for classroom assignments begins in December for the upcoming academic year. Further, if you are teaching more than one section back-to-back you should request that you be assigned the same classroom for those consecutive sections as you need time to set up your laptop and the data projector. The Department can then try to get your requests filled with the Office of Space Management (OSM). There is sometimes a shortage of electronic classrooms, so as a courtesy to your colleagues you should not request one unless you plan to fully utilize it.

· Visit your classroom before the first class. Room assignments are posted at Current A&S Timetable (regular academic year). Visiting in-person is ideal. You can also check out the room virtually. (Additionally, rooms will be posted before the first class on our website.)

· Make sure you completely shut down all data projection equipment when you are done lecturing (unless AV has specifically instructed you to do otherwise). Leaving this equipment on not only uses up valuable bulb life, which increases the chance of bulb burn out during lecture, but it also makes this equipment vulnerable to misuse.

"Art" of effective PowerPoint use: Some tips, questions, answers:

· Q: Do we have reliable technical support or will I be wasting a lot of valuable lecture time trying to get projection equipment working?
A: In fourteen years and in about 2,500 hours of lecture time (all using the data projector) I have lost a total of less than 3 hours due to technical problems. The two worst case scenarios: (1) I had to give the first hour of lecture on the board because of a complete breakdown of the teaching station and delayed delivery of a backup and (2) the bulb for the data projector expired in the middle of lecture and it took 15 minutes before a portable data projector was delivered and operational. This is not too bad at all: I have found that the AV support at ACE is very responsive and quick. You should be proactive though: arrive 10 minutes prior to lecture, have a demonstration with a technician before first day of class so you know how to operate everything, and immediately report any problems with the equipment (especially if the image is not a bright as usual as this is a sign the bulb is about to go).

· Q: How about hybrid lectures: part PowerPoint and part blackboard?
A: Good idea in theory but hard to implement given that most classrooms have the projection screen directly on top of the black board (white board). A great alternative is to use a tablet PC with a pen so you can write in real-time directly on your slides.

· Q: Isn't it time consuming to create PowerPoint lectures?
A: Yes, it is fairly time consuming. How time consuming depends a lot on you and the course material. You may want to try creating a couple of PowerPoint lectures to see how long it takes.

· Q: Should I post all of my lecture notes on my course web site?
A: Understandably, students really, really want the lecture notes posted. However, to make sure that students attend lectures and keep up with the course in-between assessments, lectures have to have "value added." Each instructor can use some creativity to figure out a way to ensure lectures have "value added" even if all the lecture notes are posted before lecture such that students can print them out and bring them to lecture. In my case, have found that peppering the lecture notes with questions that are only answered during lecture and using iClickers helps ensure high attendance. There are many possibilities. For an econometrics course you may present a table of estimation results and discuss the interpretation in class (rather than write out the interpretation in the slides). For a course more focused on public policy, you could present the facts and assumptions in the slides but draw out the policy implications in class. For a theory course, you could present a model and derive the results in the slides but work through the intuition and the importance of various assumptions in class. Of course there are more direct means of ensuring class attendance such as having a TA pass around a sign-in sheet to take attendance, including "class participation" in the marking scheme, pop quizzes or other short in-class exercises. Make it very clear to your students that you expect them to take notes and attend class.

· Q: Does using PowerPoint mean that I just read from slides in class?
A: Try hard not to read your slides aloud in lecture: this is boring and students get frustrated because they can read faster than you can talk. Focus on providing context, interpretation, explanations, elaborations, insight, and encouraging some class room discussion (if appropriate).

· Q: Does using PowerPoint mean I should change the content of my course?
A: The means of delivery should not dictate the content. If you feel that the content of your course is not well-suited to using PowerPoint then don't use PowerPoint.

iclickers:

Overview of iClickers at U of T

piazza & Ed Discussions:

Piazza

Ed Discussions (very similar to Piazza, but allows moderation, whereas Piazza does not: see this chart)

Which tool should I use for asynchronous discussions? (Support from FASE's Education Technology Office)

How to use Piazza in your Quercus Course (Students)

Quizzes IN QUERCUS:

Quizzes section of the Canvas Instructor Guide (detailed instructions with screenshots)

New Quizzes (Quizzes.Next) (Launched in September 2019: STILL NOT RECOMMENDED TO USE NEW QUIZZES AS OF APRIL 2023: STICK WITH CLASSIC QUIZZES)

See Quizzes in Quercus, which is a more up-to-date version of the below

You can run online quizzes in Quercus. If you are familiar with Aplia or MyEconLab, which are for-profit online testing options sometimes packaged with textbooks, Quercus quizzes are the same idea. However, Quercus quizzes are free for our students but costly in terms of your time/effort and/or TA hours. In other words, you or your TAs prepare and input the questions. Starting in Summer 2015 but ending after 2020/21, I have used weekly online quizzes. I prepared questions and answers in a plain text document. I e-mailed these to my Head TA who inputs and sets up the parameters (points, time limit, due date, etc.) How much work the input part is depends on the number of questions, style of questions, and whether you want randomization.

Next is Q&A to address likely questions. However, some answers are TBA because from Summer 2015 through Summer 2018, we used portal (Blackboard) and we will use Quercus for the first time in Fall 2018. If you choose to use quizzes, there will invariably be more you and your TA will need to figure out as you go. Hence, while I believe there are strong pedagogical benefits, I am certain there are considerable costs. Quercus quizzes may be the best idea in larger enrolment courses where the fixed costs are spread over many students and where you have considerable TA hours and can delegate much of this.

· Q: What is the potential value of online quizzes?

· A: It can be an efficient way to administer frequent quizzes in your class without managing paper (i.e. no photocopies, no sorting, no returning papers). I would NOT recommend it as a substitute for regular tests because online tests are not invigilated. However, if you are looking for a way to help students regularly engage in your course – e.g. every week – in addition to regular tests, this is a potentially good option.

· Q: What kind of questions can I ask in Quercus quizzes?

· A: Pretty much any kind (although not as many as Blackboard). The Canvas Guide titled “How do I create a quiz with individual questions?” gives the options. It can be “Numerical Answer,” which means that students type a numeric answer. It can be “Multiple Choice,” which means students select a letter answer (e.g. (A) – (E), but any number of alternatives are possible). At the other extreme, it can be “Essay” or “File Upload.”

· Q: How do I handle students registered with Accessibility Services who get extra time?

· A: It is like for term tests except that usually the course TA (or instructor) adds the extra time in Quercus (rather than giving external people full editing privileges in your Quercus course). Almost always these are a multiplicative of the class’s time limit (e.g. 1.5 times the usual time limit) or a flat addition of extra minutes (e.g. plus 30 minutes no matter the class’s time limit). For any special cases (which have been rare in my experience), you should consult with Accessibility Services and be prepared to be reasonable and flexible. To grant extra time within a Quercus Quiz see the Canvas Guide titled “Once I publish a timed quiz, how can I give my students extra time?”

· Q: I have many questions about how to create and deploy quizzes in Canvas and I want detailed instructions with screenshots. Where do I start?

· A: I would strongly recommend the Quizzes section of the Canvas Instructor Guide. Note: Canvas is currently (as of August 2018) working on a major upgrade to Quizzes called Quizzes.Next. However, we (as of August 2018) only have access to Quizzes so you should make sure you look at the Quizzes section and not the Quizzes.Next section of the guide.

· Q: Can I put tables, equations, graphs, etc. in the question?

· A: TBD (it was nearly impossible in Blackboard, which had extreme formatting issues, but maybe it will be possible in Canvas).

· Q: How are Quercus quizzes marked?

· A: It depends on the type of questions you set. For short answer questions (like “Numerical Answer” or “Multiple Choice”) you input the correct answer (and margin of error) with the question to enable automatic marking.

· Q: What if I realize there is an error in the quiz after students have already started taking it?

· A: You have a mess to deal with. The current regrade features are extremely limited and there is nothing to be done (other than manually going through every submission) if you make many kinds of errors: see “What options can I use to regrade a quiz in a course?” under Canvas Guides. It looks like Quizzes.Next (not yet available as of August 2018) will be better on this front. (Also, there were serious bugs with this functionality with Blackboard portal that were never fixed after literally years of repeated reporting.) Unfortunately, you cannot even just delete a problematic question or make its point value zero (things that you could do in Blackboard portal). If you make changes to point values or questions after some (or all) students have submitted this will just create inconsistencies: most things are not updated (like students grades) but some things are updated (like total possible points). Hence, at least for now, you need to have an extremely detail-oriented Head TA working on your online quizzes. Any mistake discovered after some (or all) students have already submitted will be a real headache.

· Q: Can I turn off the big screen suggesting that students submit a comment on their mark (each of which will trigger an e-mail notification to the instructor)?

· A: I think it is not possible disallow comments (or to remove the big invitation to submit them for each mark). You can change your notification settings: you can still see comments from with Grades. You can have a regrade policy in your syllabus that requires a more formal mechanism (e.g. e-mail to Head TA) to dispute a mark.

· Q: How easy is it to use Quercus quizzes?

· A: Mostly it is easier than Blackboard portal, but still not easy.

· Q: Why should I do all this extra work? Why not just use Aplia or the online question bank that comes with my textbook?

· A: Maybe you should use a paid service instead. It depends. If there is something already set up and ready-to-go and the questions work well for your course (and it costs students less than $65 for an H course or less than $130 for a Y course so that you comply with the Student Ancillary Fees rules), then that may be the best option. Also, some of these paid services have nice options (like embedded graphs that students can edit), which would be hard (or in some cases impossible) for you to do yourself. However, the pre-packaged questions do not align well with your course, then Quercus quizzes are a good option because you write the questions.

· Q: What about simply doing paper-based tests (not online)?

· A: First, I would not view Quercus quizzes as a substitute for regular tests (invigilated and paper-based). Focusing on frequent quizzes and weekly participation, Crowdmark is also an attractive (paper based) option. The advantage of Quercus quizzes is that the marking costs for certain types of questions approach zero.

· Q: How much should Quercus quizzes be worth?

· A: Obviously it depends. However, it seems like 5% - 10% of the course grade is often reasonable if students will have to do many quizzes (e.g. 10 to 20). That’s enough to motive students but not so much as to give an extra big incentive for cheating.

· Q: What about cheating?

· A: This is a big concern. Online tests (Quercus quizzes or paid services) are not invigilated. There are strategies to address potential cheating. There is test design: e.g. give students random subsets of questions or have randomly generated numbers in questions so that tests and answers differ across students. There are collaboration policies: e.g. allowing students to work in teams. However, after 2020/21 I realized the cheating issues were too large to allow weekly quizzes to continue.

· Q: How long do students have to complete tests?

· A: You pick this (and a long list of other options). I like to give students a timer: e.g. 60 to 90 minutes once you start. This is another method to discourage cheating (but that is not the only rationale for having a timer).

· Q: What if students are late? Miss a Quercus quiz?

· A: Usual dilemmas. I like to keep it simple and not allow late submissions and to have a make-up quiz (open to everyone) that replaces the X lowest marks. Section 11.2 of the 2017/18 ECO220Y syllabus shows a possible policy.

· Q: How do students review their work after it is marked?

· A: It’s obvious for students. They can either go to the Quizzes area or the Grades area. Also, they get notifications.

· Q: What about partial credit for trying (participation points)?

A: Not possible within Quizzes tool in Canvas.

· Q: I’m thinking about trying Quercus quizzes in my class. How do I set them up?

· A: For full instructions (with screenshots), see official support sites linked above. Below I give the specific steps I’ll ask my Head TA to follow (which include a bunch of specific decisions based on quiz parameters I like to set). Also, I would strongly recommend delegating the task of inputting and checking questions to a Head TA and maintaining your own master file of questions in a format/organization separate from any specific technology.

Example (from ECO220Y1Y) of steps to run a quiz in Quercus:

1. Enter the Quercus site for our course.

2. Click Quizzes on the main menu.

3. Click the + Quiz button.

4. Add the official name of the quiz (e.g. “Quercus Quiz 1” or “DACM Module A Online Test”) and copy-and-paste the quiz instructions.

5. Leave defaults for Quiz Type (Graded Quiz) and Assignment Group (Assignments).

6. Leave Shuffle Answers unchecked. (However, if the quiz includes any multiple choice questions then please DO check this box.)

7. Check the box for Time Limit and enter the minutes (e.g. 60 or 90).

8. Leave the box for Allow Multiple Attempts unchecked.

9. Uncheck the box for Let Students See Their Quiz Responses (Incorrect Questions Will Be Marked in Student Feedback). (Note: After the quiz window closes we will change this so students can see the questions, their replies, and the correct answers.)

10. Leave the box for Show one question at a time unchecked.

11. Leave Quiz Restrictions unchecked.

12. Leave defaults for Assign to (Everyone).

13. Enter the exact due date and due time. Double-check your entry (making sure that am or pm is correctly specified).

14. Enter the exact date and time the quiz becomes available. Enter the due date and time (again) for the end date of availability. (This means that we do not permit late submissions.) This step is very important: double-check that everything is exactly correct before proceeding.

15. Click Save & Publish. (I know it seems crazy to “publish” the quiz before we have even entered the questions, but we have to in order to add accommodations for individual students. However, students will not be able to see this quiz until the date set under Available from. It is VERY IMPORTANT that you correctly set the Available from and Until fields before taking this step.)

16. Make sure the grades are hidden from students until you are ready to release them. In mid-summer 2019 A&S changed how this is done. It is confusing. I’d recommend that instructors go into Gradebook and click the settings gear icon (top right). Under the Grade Posting Policy tab select Manually Post Grades. This way you can control when to release them.

17. Go back to Quizzes and click on the quiz you are inputting. In top right, you will see Moderate This Quiz. Click that. Click the pencil for the individual students getting extra time and enter the number of EXTRA minutes each gets over and above the class’s time limit (not the total). For example, for a 60 minute quiz, if the accommodation is 1.25 time, enter 15. Leave the field for Extra Attempts blank and leave unchecked the box for Manually unlock…

18. Go back to Quizzes and click on the quiz you are inputting. Click the Edit button and then click the Questions tab.

19. At the bottom of the screen, click the + New Question Group button. Enter the question group title in the .txt file I sent (without the ALL CAPS part). (This is for us: students cannot see the name of the group.) Unless otherwise instructed, leave the default to Pick 1 question and enter the pts per question specified in the.txt file. Click the Create Group button.

20. To add a question to the group, click the + in the top right of the frame for that question group. Leave the name of the question version as the default (Question) and choose the question type from the drop-down menu. Copy and paste the question from the .txt file (keyboard shortcuts work). Some things you need to know about the three types of questions we use most often:

a. Numerical Answer. For answers, we use Exact Answer (the default). Copy and paste the correct answer and the error margin: keyboard shortcuts for paste do not work for the answer and margin of error fields: right-click and use the drop down menu. NOTE: Canvas can accept numeric answers with at most four decimal places: if you (or a student taking a quiz) try to input a more precise answer, it automatically rounds it to four decimal places and shows four decimal places (e.g. if a student types 2.05 it will auto-convert to 2.0500). Hence, for questions requiring precision of four decimal places the margin of error is zero (which will appear to students as “0.0”). (Also, when Canvas display the correct answer for students it removes trailing zeros.) Using the garbage can, delete the other exact answer fields. (It is hard to imagine a question where there would be more than one exact numeric answer, but by default Canvas allows 4 different exact answers.) Do this right away for the first question in the group because when you go to add the next question version for that group it will only have one exact answer field by default. IMPORTANT: You must delete those extra answers. By default it will list all of them for students when they get results and it says that 0.0 is an accepted answer: this it is very confusing for students AND a student that entered 0 for every question would get a 100%! Click the Update Question button at the bottom.

b. Fill in Multiple Blanks. For Fill in Multiple Blanks, after you paste from the .txt file, which already is correctly formatted, you enter the correct answer for each blank using a drop-down menu. (It is weird and inconvenient that it is a drop-down menu, but it is.) Fortunately, these are not case sensitive (e.g. if the answer is “A” and a student enters “a” it is marked correct). Also, it does not mark things incorrect because of leading or training blanks (which is also good). Also, unlike the numerical answer case (above), you do not have to delete the extraneous alternative answers. These are blank by default, but Canvas DOES mark blank answers incorrect (so even if you do not delete the blank alternates, students will not be marked correct for entering blanks).

c. Multiple Choice. For multiple choice, you do NOT have to put the correct answer first: enter them in the same order as the .txt file and then click the correct answer (by default it clicks the first choice as the correct answer so you must click the actual correct answer). Note: To randomize the order of the answers for all multiple choice questions in your quiz, after creating the quiz, under Details, scroll down to Options and check the box "Shuffle Answers."

d. Any type of question where you wish to give students STATA results (e.g. regression, summary) with proper alignment. It is easier to include output in Canvas questions than in Blackboard, but it is still not obvious. For example, suppose you wish to input this question (in plain text format, .txt file) as a quiz question. This link shows how to do it (with screenshots).

e. Any type of question where you wish to give students a table with proper formatting cut-and-paste from a .csv or .xlsx file. For example, suppose you wish to input this table (in .csv format) or this table (in .xlsx format) as part of a quiz question. This link shows how to do it (with screenshots). (You could also use the HTML editor if you’re into that: see “Making table width automatically resize on a Canvas content page”.)

21. After carefully entering all question groups and questions and checking over your work, click the Save button.

22. Click into Quizzes and click the title of this quiz. Make sure the total points and other parameters are showing correctly. Scroll down and click the Preview button. Make sure the questions appear as intended.

23. Once you have double-checked everything, e-mail me to let me know you have the quiz set up (at least two days before it opens for students).

24. After the submission window, deal with any outstanding student quiz submissions. See “How do I manually submit outstanding student quiz submissions?” in the Canvas Guides.

25. We will have to figure out how to run an item analysis. Canvas does not currently have the ability to run standard analyses except in very limited circumstances (multiple choice and true/false questions only). After the quiz, under Quiz Statistics we can run Student Analysis and download the .csv data and write software to compute our own statistics.

26. What if there are any errors in the quiz that we discover after some students have already submitted (and/or after the quiz is over)? It is really hard to deal with this because Canvas has almost no options to regrade (we cannot just delete a problematic question or update its answer). First, we should try never to let this happen. If it does, it seems there are some costly fixes. We can use SpeedGrader, which despite its name means manually scrolling through every submission and changing the points earned by hand for a problematic question (e.g. to give everyone marks or to accept the actual correct answer). Further, this does not update the fact that students will see that they have been marked incorrect on that question even though they earned full marks (hence it will be confusing for students). Under Settings, Options there is an option to Grade by question (beta), which may make it a little easier (not have to scroll down each submission to get to the problematic question). However, I cannot pretest this functionality without actual multiple submissions.

27. After we have checked over the results and we are ready to release results to students, go to the Quizzes page and click on the title of the quiz. Click the Edit button in the top right. Scroll down and check the box for Let Students See Their Quiz Responses (Incorrect Questions Will Be Marked in Student Feedback). Check the box (that now appears) for Let Students See The Correct Answers. Leave the dates fields blank. Scroll to the bottom and click the Save button. Go to Grades, mouse over the title of the quiz, click on Grade Posting Policy from the dropdown menu. Click the Automatically button.

CROWDMARK:

How do I connect my Quercus course to Crowdmark? (NICE! Posted March 23, 2023)

See Crowdmark, which is a more up-to-date version of the below

Some collected wisdom in Economics of using Crowdmark by Jennifer Murdock (refresh for the Sept 5, 2019 version)

Using Crowdmark with Quercus by Jennifer Murdock (refresh for the Sept 4, 2019 version)

Multiple choice questions in Crowdmark

Using Crowdmark in Arts & Science

What is Crowdmark?

Starting in 2014/15, select courses in Economics have utilized Crowdmark to manage term work electronically (returning term work to students electronically). Here is an Introduction to Crowdmark. Also, for first time users, a good place to start is on Crowdmark’s “What can Crowdmark do?” page. For how to use Crowdmark for a Quercus course, see Using Crowdmark with Quercus. It may help to see a sample Crowdmark test from an Economics course with 500 students. Each student gets a unique test paper (i.e. #1 - #500): here is paper #1 and the separate supplement (not through Crowdmark).

Why use Crowdmark?

There are many distinct advantages of Crowdmark that add up to a big advantage over the traditional paper-based approach. Some most valued by those in the Economics Department:

· Improved student access to their own term work (no lost papers; convenient electronic copies): surveys of our students show that students really value Crowdmark

· Greatly improves the efficiency of having TAs specialize in subsets of questions (which is great for accuracy, consistency and speed): for example, you can have 8 TAs each specializing in one question or parts of a question without wasting time passing around papers and flipping through them to get to Question (3)(d) (for example)

· Instructors can see marking progress of TA team in real time and see how many hours each TA spent on marking

· Marking is portable: TAs can mark even if they are off campus or abroad

· Substantial savings of TA hours

o Marking is more efficient and especially for tests marked by a team of TAs

§ TAs do not need to pass papers around to each other

o Papers do not need to be sorted

o Marks entry is faster (and more accurate)

· More accurate marks entry (and no need to write subtotals on front of test papers, which are very prone to clerical errors)

· Greater ability of students to check for and efficiently report clerical errors

· Saving valuable class/tutorial time because do not have to return papers to students

· Faster return of students’ work: can send results to students as soon as marking is complete and do not need to wait until the next class/tutorial

· Allows instructors on-going and convenient access to all submitted term work (including for reviewing TA marking, preparing letters of recommendation, etc.)

· Elimination of the possibility of ex post cheating (falsified re-mark requests)

· Allows students to write term work entirely in pencil and use an eraser as needed, which is better for test-takers and markers

· More efficient handling of re-mark requests

· Potential for more comments on student work: online marking makes it sensible to invest in preparing good comments for common errors that can be easily reused

Also, you may request a copy of “Crowdmark Report: Select ECO Courses” by Jennifer Murdock, Economics Department, University of Toronto, August 12, 2015 by e-mail (jennifer.murdock@utoronto.ca).

How can I use Crowdmark in my course?

You will receive an e-mail near the start of the term with procedures.

How do I actually start using Crowdmark?

While using a new technology can be daunting and especially when producing your tests on a deadline, using Crowdmark is quick and easy. The steps you will need to take to enable Crowdmark within your Quercus site are explained in Using Crowdmark with Quercus. You can visit Crowdmark’s help page and they are anxious to help you personally: just e-mail them at support@crowdmark.com. They respond very quickly and will walk you through step-by-step over the phone if need be. Also, here is some collected wisdom in Economics of using Crowdmark after several years of use, which will help you with some common practical questions about using Crowdmark. (Note that you cannot have students write any answers to be marked on the first page: the first page is used to collect student identifying information).

How many TA hours do I need to allocate for scanning and matching in Crowdmark?

For scanning the papers, a rule of thumb is 5 minutes per 100 pieces of paper (our scanner scans both sides simultaneously) or, equivalently, 1,200 pieces of paper per hour. Hence for a test paper with 8 pages (4 sheets of paper) and 500 students, scanning would take about 100 minutes. This includes time to correct scanning errors (stuck pages). It helps to neatly stack papers before scanning to minimize scanning errors. Without scanning errors it takes about 3 minutes per 100 sheets of paper.

For matching the student names, including double-checking the matches, a rule of thumb is 10 – 12 minutes per 100 students. Hence, for a class of 500 students, matching would take a bit less than an hour. However, if students do not correctly fill in their identifying information, it can take up to 25 minutes per 100 students to ensure 100% accurate matching and a class of 500 students could take closer to 2 hours. As an alternative to using TA hours, please see you may consider using the Instructional Support Staff in Economics (new for 2022, the St George campus only) for scanning, uploading, and matching for Crowdmark.

GENERATIVE AI:

Generative Artificial Intelligence Policies and Resources by A&S Digital Teaching and Learning (Don Boyes)

ChatGPT and Generative AI in the Classroom by Office of the Vice-Provost, Innovations in Undergraduate Education (Susan McCahan)

Important links	Important dates	Getting started
Developing as an instructor	Teaching evaluation	Academic Integrity
Tests/Exams/etc.	Writing	Grading
Multiple choice (software)	Course sites (including Quercus)	TAs
Students & Communication	University and A&S teaching initiatives	PowerPoint or Other Software
iClickers	Piazza & Ed Discussions	Quizzes in Quercus
Crowdmark	Teaching and Learning Community of Practice (CoP), Department of Economics, 2016-2023	Generative AI