Report

Installation

1. Clone the repository

git clone git@gitlab.developers.cam.ac.uk:phy/data-intensive-science-mphil/assessments/m2_coursework/es833.git

2. Set up the environment

Using Conda:

conda env create -f environment.yml
conda activate M2

Using pip:

pip install -r requirements.txt

Repository overview

The main code is in the src/ directory:

• the core classes:
  • data_module.py contains the SequenceDataModule class which handles the preparation of the tokenized dataset and the splitting into training and validation data loaders
  • lit_module.py contains the LMModule class which defines the model architecture, loss, optimizer, and training, validation and prediction steps
• model extension and evaluation:
  • lora.py contains the LoRALinear class which applies LoRA to a given linear layer, and runs the script for the training and evaluation of such LoRA'ed models in part 3
  • flops.py contains the functions for our FLOPs estimates
  • test_generation contains code for testing model predictions in numerical space
    • compares with the baseline models in baselines.py
• utilities:
  • preprocessor.py contains preprocessing (numerics -> text) functions, including one to undo the preprocessing, and one to show the action of the functions on an example
  • processor.py contains the processing (text -> token IDs) functions
  • qwen.py contains functions to load the Qwen model and inspect its architecture

Outside src/ we have:

• the report main.pdf and walkthrough notebook report.ipynb in report/ (along with the relevant coursework pdfs)
• figures in figures/
• the best model checkpoint checkpoints/best.ckpt (we remove other .ckpt's to save gitlab storage space)
• untrained predictions in untrained_predictions/
• data in lotka_volterra_data.h5
• configuration files: pyproject.toml, environment.yml etc.

LLM and Web use

• Coding.
  • I used LLMs primarily for syntax-heavy, concept-light tasks -- this includes tasks like plotting, drafts of docstrings/comments (which I then check and correct) -- and debugging
• Information.
  • I use the web and LLMs for advice on packages, coding styles, coding practices etc.
  • I relied on a few books for the coursework and course more generally:
    • Machine Learning with PyTorch and Scikit-learn (Raschka 2022)
    • Understanding Deep Learning (Prince 2024)
    • Information theory, inference, and learning algorithms (Mackay 2003)
• Writing.
  • I sometimes use LLMs for rewording suggestions and checking grammar, though I rarely actually follow them. Mostly this functions as a time-wasting exercise where I ask if my grammar and wording is okay in a bid for machine approval, and it responds with a rewrite that I like less and ignore.