Validating What We Have Made

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • FIXME

Objectives

After following this episode, learners will be able to…

  • Recognise that evaluation of generated code and validation of results is essential.
  • Identify ways to test the code generated by a chatbot.
  • Evaluate the importance of testing different parts of the code they generate.

But Is It True?

  • Regardless of whether you wrote the code from scratch or generated it with AI, you will held responsible for the results it produces
  • LLM models make mistakes. Perhaps not often, and especially not when working with common libraries, functions, and tasks similar to those that many people have done in the past.
    • (Toby: not really mistakes, since “mistake” implies intent on the part of the mistake maker: “oops, I was wanted to do X but instead Y happened”)
  • But, whereas a person might give you some indication when they are not confident that they are giving the right answer/good advice (“I’m not really sure but if I had to guess I would say…” or “You should really ask Jessica that question: she knows a lot more about this stuff than I do…”), the output of your chatbot will project the same confidence and positivity regardless of the relevance or factual accuracy of its response.
  • It is up to you to determine whether or not the output generated is accurate and/or does what you need it to do.
  • The code tracing/review skills you have picked up in this lesson will be helpful. But looking at the code is often not enough on its own:
    • Once the script/code base gets large, it becomes difficult for one person to keep a complete mental model of how it works, and how any given change will affect the functioning of the whole. (A similar problem exists for the LLM once the code base gets very large: if the size of all the relevant information exceeds the context window of the model, performance can deteriorate as the model “forgets” critical information.)
    • It can also be difficult to consider the code that isn’t there. Changes generated by a model might all seem sensible by themselves, but be missing something that later turns out to be important.
    • Models perform worse on unusual and niche tasks, those that are not well-represented in their training data. (Toby: find the paper that Greg W shared recently re:model performance vs a benchmark of scientific challenges.) Since research computing tasks are often concerned with questions/data at the limits of current human understanding, you should be prepared to encounter more problems than somebody generating code for more routine tasks in well-trodden territory.

What are some good validation strategies?

  • Test your code!
    • Minimally, run it on a small, test dataset. Check results carefully. Think first about what you expect to see, and compare the outcome with that expectation.
    • Allocate your rigour: consider how crucial each part of the code is, and treat those parts accordingly. For example, routine plotting functions and other standard “boilerplate” is more likely to be generated correctly – and, in the case of the plotting code, errors might be detected quite easily. You might choose to spend less effort reviewing that code. But complex and/or crucial processing and calculations that are particular important for the results you will be reporting from your work merit a much closer look.
    • Consider adding some unit tests to your code: that is, code that tests the correctness of the other code. Programming languages typically have at least one framework to do this, and automated code generation has made it much easier and faster to do. Of course, you will also need to check the tests so that you can be confident that they are correct and/or testing the right things.
      • Toby: I guess unit testing is beyond the scope of this lesson? In which case, I think we should at least point learners towards a resource where they can learn more. Or we could include at least a basic example?
      • Bear in mind that scientific code may be more difficult to test: a lot of unit tests are concerned with things that are quite easily and logically testable, e.g. whether the output of a function is a positive integer, whether the function fails gracefully when provided with unexpected values, etc. But results produced by scientific code may need to be tested for their plausibility, e.g. whether they respect the laws of thermodynamics, whether a number fits within a biologically feasible range, etc. You will need to use your own domain expertise to inform this kind of evaluation, and it is worth noting that LLMs lack the kind of mental model of the world and how it works that you apply in your daily work, making them less likely to provide useful output in these cases.
  • Ask the chatbot to mark its own homework. Some users have reported good results from asking the chatbot/model e.g. “did you miss anything?”. You could also try starting a new chat session or even opening up a different chatbot/model, providing some context for the project, then asking it to review the code and provide constructive feedback on how it could be improved. (Humans can also be good at this!)