Rethinking Your Problem Sets in the World of Generative AI

Rethinking Your Problem Sets in the World of Generative AI

An Open AI DALL-E generated cartoon drawing made with the following prompt: A color illustration of a friendly robot lifting a person up a mountain by the hand.

Introduction

In most STEM subjects, problem sets (psets) are both a central learning tool and a key assessment method.

When students grapple with the solution of challenging, well-posed problems, they learn not only to retrieve and apply concepts related to the particular topic of study (e.g., conservation of momentum, integration methods, balancing equations, etc.) but also about the problem-solving process itself.

How does the current state of generative AI (genAI) impact the traditional affordances of psets?

Much has been written about generative AI’s ability to successfully solve standard pset questions from many disciplines. This, combined with an inability to readily and accurately detect genAI output, will require us to think more critically about goals for student learning vis-a-vis problem sets in this new world. Keep in mind that you will not be able to completely prevent students from using generative AI for their psets, assuming you still require that they complete them outside of class time (see “A fresh look at blended classrooms” below). Therefore, regardless of your view of genAI and its role in higher ed – it is worth knowing its capabilities with respect to assignments in your subject.

At present, OpenAI’s ChatGPT-3.5 is open-access and free but is more prone to producing incorrect (“hallucinated”) responses. ChatGPT-4 is much “better” than ChatGPT-3.5 but costs $20/month. Bing, Microsoft’s version, has comparable capability to ChatGPT-4 and is “free” but requires a Microsoft account and must be used in the Microsoft Edge browser.

In the face of these challenges, consider incorporating the use of genAI into traditional psets. Doing so will support student learning of both your course content, in particular, and problem solving, in general. In addition, it can help students become more savvy generative AI users. All are likely to be important for success at MIT and beyond.

As mentioned in our post, Teaching & Learning with ChatGPT: Opportunity or Quagmire? Part III – whether or not you explicitly allow or ask students to use generative AI in your courses, you’ll want to consider the issues of equity and accessibility; and student data privacy. Without some planning on your part – some students in your class will have access to ChatGPT-4, while others may only have ChatGPT-3.5 or Bing.

Plan for the semester

Step I – Gather information. Understanding what generative AI can do with the assignments in your course is the first step in developing a plan for the semester.

  • Sign up for a ChatGPT(-3.5 or -4)1 (OpenAI) or Bing (Microsoft) account.
  • Input some pset questions from your course. For each question, consider the following:
    • What is your rationale for asking this question on a problem set? What are you hoping students learn? Be as specific as possible.
    • How did the genAI do?
      • What types of questions is generative AI “good” at? Where does it fall short? Can you make any generalizations about the kinds of problems that it can/cannot handle? How might these observations inform how you structure problem sets and other assignments?
  • Prompt AI to iterate on its incorrect and/or incomplete answers. Ask a range of questions (from the specific to the general). For example, “Are you sure there should be an inverse-squared-term in Part A? What does your response say about the system’s behavior at long times?” “Why did you take a partial derivative with respect to volume at constant pressure?” See how it responds. (Warning: these rabbit holes can be long and tortuous!)
    • Ask yourself if students would know whether the generated response was incorrect or incomplete.
    • How might you turn these incomplete & incorrect responses into learning experiences?
    • How you can enable and encourage students to critically evaluate AI-generated responses in your field.

Try to keep an open mind during this process. Are there things that you have traditionally asked students to do that may not now be necessary given generative AI’s capabilities?2

A look at ChatGPT-4 responses to questions from Thermodynamics

I asked ChatGPT-4 some questions from a graduate-level Materials Thermodynamics Course that I taught for many years.3 It completely nailed some of the questions. Other questions yielded a range of incorrect responses: from sign errors to flawed logic to a general lack of nuance. You can view my exchanges with ChatGPT-4, along with prompts, iterations, and my annotations, here.

Step 2 – Consider your options. Now that you have a better sense of what generative AI can do with some of your course material, how will you handle its use in your subject?

What you do next will depend on the specifics of your teaching context, in particular:

  • The discipline
  • The subject level
  • The number of students likely to enroll
  • Your teaching team (do you have a co-instructor? TAs? Other support?)
  • The nature of your current psets & assignments (quantity, size & format)
  • Generative AI’s responses to your questions

The following sections offer some suggestions for incorporating generative AI (I) and for minimizing its role in your course (II) & (III).

Incorporate generative AI into your assignments

Consider how you can create opportunities for students to engage, critically evaluate, and perhaps leverage genAI responses in order to learn the desired content and become better problem solvers.

  1. Require that students complete (without the use of genAI) the following steps in class or as part of a “pre-problem set” assignment that they must submit before beginning the actual pset. For each question (or a subset of questions), ask them to:
    • Identify potential issues, points of confusion, etc.
    • Consider key aspects of the problem’s solution, e.g., limiting cases, orders of magnitude, general trends, etc. These are all skills of expert problem solvers, but novices often skip these steps and jump directly into calculations. By requiring that students engage with this pre-solution step, you can help develop students’ problem-solving skills.4
    • Ask students to complete the problem set. You will need to decide how proscriptive you will be about their use of genAI. For equity and privacy reasons, if you explicitly allow students to use generative AI in their solutions, make sure to provide all students with access to the same “level” of AI-generated solutions. This can be accomplished by providing students with access to an account – or by providing students with an “as-is” genAI solution).

Provide students with explicit, written guidelines for the citation and use of AI-generated solutions.5 Share your rationale for the policy and be clear about the consequences of its violation.

  1. For each problem (or a selected subset of problems), ask students to submit responses to a set of reflective questions, for example:
    • Is the solution completely correct? Are there typos, misstatements, incorrect equations, or conclusions?
      • How does the solution align with your pre-problem set predictions?
        Does the magnitude of the answer make sense?
      • What happens for limiting cases?
    • How does this relate to the example on XXX that we did in class?
    • If you or they used generative AI for the solution, what additional questions/prompts/caveats/values would they like to input to genAI to better refine the answer?
  2. Allow students to see how genAI responds to these additional prompts (this could mean providing students with access to an account – or creating a means by which students can submit follow-up questions – and you or a TA can input the info).
    • How does the answer change?
      • Is it more complete? Is it accurate? Why or why not?
    • If students are not satisfied with the answer – ask them to consider what additional questions they would ask.
  3. After students are satisfied with the answer, ask:
    What’s the significance of the answer?
    Why should we care about this problem?

Note that any of these steps can be done – as part of a traditional homework/pset or as part of an active learning activity during class time.

Whether or not you explicitly incorporate generative AI into all your assignments (or a subset of your assignments), make sure to stress that it doesn’t always produce the correct answer – provide examples. Underscore that genAI output requires reflection and input from humans. As noted in a previous post, for at least one or two questions per problem set, you can ask students to explain their thought processes as they solve/engage with a particular problem. A few (of many possible) helpful prompts may include asking them to describe:

  • Why they chose a particular method;
  • Why they made certain assumptions and/or simplifications;
  • Where they ran into dead ends, how they found their way forward; and
  • What broader takeaways they learned from solving the problem.6

Add weekly, low-stakes quizzes to supplement problem sets

If you feel that students may rely too heavily on generative AI to complete their problem sets, consider implementing weekly quizzes wherein you ask them to solve a close variant of one or two of the pset questions from the week and/or a previous week.

Frequent, low-stakes quizzing allows students to practice retrieval – a key component of the learning process, and if the quizzes pull questions from previous weeks, it can further develop students’ ability to retrieve and apply recently learned concepts and skills. See TLL’s How to Teach pages for additional information on retrieval and spaced & interleaved practices.

If you use this strategy, you’ll want to:

  • Be explicit with students about what you are doing and why.
  • Include the follow-up questions from the previous section (above) in your pset – to encourage students to reflect on the problem-solving process.
  • During the first few weeks of the semester, model the type of reflective and engaged problem solving that you are looking for on the quizzes (and possibly the psets)
  • Decrease or eliminate the point value of the psets. This will help to maintain a reasonable grading load for course staff.

Take a fresh look at Blended Learning (aka Flipped Classrooms)

If you are committed to limiting students’ use of genAI in your subject – you may want to take a fresh look at Blended Learning (BL). In BL, students generally view recordings of lectures and/or engage with pre-class readings to gain a basic understanding of relevant topics (this is the “information-delivery” component of the class). They then engage in active problem solving (information retrieval and application and knowledge creation) during class time. Depending on your subject (topic, level, etc.), you may be able to restrict students’ use of computers during class – and therefore ensure that they are engaging in traditional problem solving as they grapple with key problems from the week’s material.7 Additionally, even if students are required to use particular software or applications – you can monitor their use during class time.

Staff from the Teaching + Learning Lab are available to help you implement blended learning in your courses. Additional Resources are provided below.

Summary

Generative AI (genAI) is here, and students are going to use it. In order to continue to support the development of deep learning and enduring understanding, instructors need to acknowledge genAI’s existence, (conditional) widespread accessibility, and its impact on student learning. Some relatively straightforward changes to the way we think about assignments in light of genAI can have a big impact. Regardless of how you plan to use (or not use) generative AI in your subject, be explicit with your students about your strategies and your reasons.

Please reach out to TLL with any questions about incorporating genAI into your subjects.

Resources

For ideas on how to use generative AI in a variety of other assignments, see:

Brett Becker. Programming Is Hard – Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation
Derek Bruff. Agile Learning blog. See in particular, his “Assignment Makeover” posts.
Ryan Cordell. Building a (better) book) – Course taught at UIUC: Lab 1: Amanuenses to AI
Ethan Mollick. One Useful Thing blog.
Daniel Stanford. Incorporating AI in Teaching: Practical Examples for Busy Instructors.
Inara Scott. The “Less content, more application” challenge.

Notes

1 If you elect to open an OpenAI account, you may find it useful to compare the responses produced by the 2 versions.

2 Over the years our ideas have changed with respect to what students really need to know how to do “from scratch” and what they can outsource to technology. For example, many courses permit (encourage) the use of software packages for mathematical operations and/or the analysis of scientific and engineering processes (e.g., Wolfram Alpha, Matlab, etc.); and the use of existing pieces of code (e.g., GitHub, etc.) in programming. And, of course – there is the calculator.

3 The textbook for the course was Callen, Herbert. Thermodynamics and an Introduction to Thermostatistics, 2nd Edition. Wiley (1991) for the first part of the course, and DeHoff, Robert. Introduction to Materials Thermodynamics, 2nd Edition. CRC Press (2006) for the second.

4 For additional information on teaching problem solving, see: Polya (1957), and Wankat & Oreovicz (2005).

5 For additional information on the development, revision and communication of your academic integrity statement, see TLL’s post on Teaching & Learning with ChatGPT: Opportunity or Quagmire? Part III

6 For more information on the use of self-explanations and reflection to support learning, see TLL’s page on Helping Students to Retain, Organize and Integrate Knowledge.

7 If you use this approach – make sure that you can accommodate students who require the use of computers, and/or specific software, etc. Reach out to Disability and Access Services (DAS) for guidance.

AI