Instructor Notes
Some thoughts from the final discussion session in the course:
- data analysis narrative vs. more “normal”
program-to-do-some-stuff narrative
- data analysis -> validation as high-level activity that cannot be possibly outsourced: statistics offer lots of possiblities to “numerically successfully” calculate numbers, which are nevertheless not the correct answer to the data analysis question. (Note: Claudia has examples showing this depending on subtle details of the data analysis task; related pairs of tasks that lead to different calculations)
- “normal coding tasks” tend to have errors that lead to exceptions; though errors in program logic are possible here as well.
- “normal coding task” errors with AI-agents: program logic mistake may lead to wrong file getting deleted etc. => think over again what is a good narrative for the course
- balance:
- ability to read code to detect bugs (in the extreme requiring super-human concentration)
- ability to formulate (here: ask for) good unit tests = high level ability
Introducing LLMs as a Tool to Learn R
Writing Effective AI Prompts
Analysing a Real Program
How To Run This Challenge
We recommend doing the first part of the challenge below together with the group. Have the learners follow along with you as you write and submit the prompt, then spend a few minutes discussing the output you received. Ask learners whether anyone received a substantially different response, and encourage them to share it in the collaborative notes if so.
Ask learners if the response(s) include any jargon that they are not familiar with, and help them understand the response if so.
Learning About Programming Concepts
Modifying Code
Gather these suggestions
You may find it helpful to gather and de-duplicate the suggestions generated from learners’ discussions in this exercise. You can return to this list in the next section, comparing it to the reponses generated by chatbots.
Leading this discussion
It is unlikely that an audience of novice programmers will be able to answer the discussion prompts completely, and learners may lack confidence to contribute to this discussion when they do have ideas. You should lead this discussion, drawing on your own experience to review and comment on the suggestions generated by your chatbot and any additional points raised in the responses that learners received. Be explicit about your thought process: insight into how an expert thinks about and approaches the evaluation of chatbot output will be highly valuable to learners. Be judicious about which suggestions you recommend that learners pursue and which they put aside to potentially follow up on later: adjusting variable names and file paths is more likely to be understandable to a novice than pinning package versions.
Some suggestions for how you could facilitate this discussion:
- Share the response you received from the same prompt. Ask learners if their responses included any suggestions that do not appear in yours. If so, gather the additional suggestions in the shared notes so that you ahve a single list to work through.
- Work through the complete list of suggestions, evaluating them one at a time. Invite learners to share their thoughts, but expect to do most of the talking since you are the expert.
- Encourage learners to reflect on this mismatch between your ability to evaluate the suggestions and theirs, and emphasise that they will continue to develop their skills as they gain more experience through practice.
- After you have completed this group discussion, you could share your screen while asking the chatbot, in a new chat/session, to review the suggestions and provide feedback on them/prioritise them. Then, discuss how the chatbots response compares to the conclusions drawn from your evaluation.
Possible extensions
Suggestions might include:
- read multiple data files and combine them before plotting
- plot more continents
- specify the continent to plot as a configuration parameter at the top of the script, alongside the path to the data file
- give the script a command line interface, e.g. read data file path and other parameters as command line arguments, save the figures to a file
Construct the prompt together
Lead the group in a discussion of how to prompt the chatbot to make one of these changes. Write the prompt and execute it on your system. Encourage the learners to think about what they expect this to look like and to reflect on what you get back.