Unsloth Fine-Tuning — Colab vs Kaggle Guide

Overview

This guide covers how to fine-tune a small LLM (Llama 3.2 3B, Phi-3, Gemma 2) with Unsloth + QLoRA on Google Colab and Kaggle. Both platforms give you free GPU access — no local hardware required.

Your local GPU has only 8 GB, which limits you to 1B-3B models at most. These cloud platforms give you 16-32 GB for free, unlocking 7B models.

[!TIP] If your dataset is small (< 1000 examples), even 3B models can learn formatting tasks well. Start small, then scale up.

Platform Comparison

Feature	Google Colab (Free)	Kaggle
GPU	T4 (16 GB)	T4 x 2 (32 GB) or P100 (16 GB)
GPU hours	~2-4h / session (disconnects)	30h / week (no disconnect)
Cost	Free (or $10/mo Pro)	Free
Speed (3B model)	~20-30 min	~15-25 min
Speed (7B model)	~45-60 min	~30-45 min
Persistence	Google Drive	`/kaggle/working/` + download zip
Data upload	File browser	+Add Data button or Kaggle dataset
Multi-GPU	No (single T4)	Yes (2×T4 via DDP)
Best for	Quick tests, learning	Long training, bigger models

[!WARNING] Colab Free disconnects after a period of inactivity (~2h). Save intermediate checkpoints to Google Drive. Kaggle has no such limit.

Notebooks

Two notebooks are provided, one per platform:

Colab Notebook

docs/AI-ML/Fine Tuning/unsloth-finetune-colab.ipynb

How to use: 1. Open Google Colab 2. File → Upload Notebook → select unsloth-finetune-colab.ipynb 3. Runtime → Change runtime type → T4 GPU 4. Run cells in order

Key cells:

Cell	What it does
1	Mount Google Drive (model save location)
2	Install Unsloth
3	Load Llama 3.2 3B in 4-bit
4	Attach LoRA adapters
5	Prepare dataset (Alpaca format)
6	Train with SFTTrainer
7	Save LoRA adapters to Drive
8	Run inference test
9	Reload saved model snippet
10	Tips & next steps

Kaggle Notebook

docs/AI-ML/Fine Tuning/unsloth-finetune-kaggle.ipynb

How to use: 1. Go to Kaggle Notebooks 2. Click New Notebook 3. Click File → Import Notebook → select unsloth-finetune-kaggle.ipynb 4. In Settings → Accelerator → select GPU T4 x 2 (or P100) 5. Run cells in order

Key differences from Colab: - No Drive mount — saves to /kaggle/working/ - Downloads as a .zip file - Supports dual GPU (DDP) for 2× faster training - No session timeout — can run for hours

Dataset Format

Both notebooks expect data in Alpaca instruction format:

[
  {
    "instruction": "Your task description here.",
    "input": "Optional context or input (can be empty string \"\")",
    "output": "The expected response the model should learn."
  }
]

To use your own data: 1. Prepare a JSON file with the structure above 2. Upload it: - Colab: File browser sidebar → upload - Kaggle: +Add Data button → upload 3. Update the dataset loading cell to point to your file

Model Options

You can swap the base model in cell 3 by changing MODEL_NAME:

Model	Params	VRAM needed	Notes
`unsloth/Llama-3.2-3B-Instruct`	3.2B	~5 GB	Default — best balance
`unsloth/Phi-3-mini-4k-instruct`	3.8B	~5 GB	Good for code/structured output
`unsloth/gemma-2-2b-it`	2.6B	~4 GB	Fastest option
`unsloth/Mistral-7B-Instruct-v0.3`	7.3B	~8 GB	Needs 16 GB GPU (P100 or T4)
`unsloth/Llama-3.1-8B-Instruct`	8B	~9 GB	Requires T4 x 2 or A100

[!NOTE] For a 7B model on 16 GB (single T4 or P100), set per_device_train_batch_size=1 and gradient_accumulation_steps=8.

Hyperparameter Cheat Sheet

Parameter	3B model (default)	7B model (16 GB)	7B model (32 GB)
`per_device_train_batch_size`	2	1	2
`gradient_accumulation_steps`	4	8	4
`effective batch size`	8	8	8
`max_seq_length`	2048	2048	2048
`learning_rate`	2e-4	2e-4	2e-4
`num_train_epochs`	3-5	3-5	3-5
`lora r`	16	16	32

Saving & Loading

Save (LoRA adapters only, ~10 MB)

model.save_pretrained("path/to/lora-dir")
tokenizer.save_pretrained("path/to/lora-dir")

Save (merged 16-bit weights, ~2 GB for 3B)

model.save_pretrained_merged("path/to/merged", tokenizer, save_method="merged_16bit")

Load in a new session

from unsloth import FastLanguageModel
from peft import PeftModel

# 1. Load base model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=2048, dtype=None, load_in_4bit=True,
)
# 2. Apply LoRA adapters
model = PeftModel.from_pretrained(model, "path/to/lora-dir")
FastLanguageModel.for_inference(model)

[!TIP] LoRA adapters are tiny (~10 MB) — save only the adapters, not the full weights. The base model is downloaded from HuggingFace each time (cached after first download).

Next Steps

1. Word Document Formatting (your use case)

Instead of training a model to format Word docs, combine: - LLM (fine-tuned) → generates structured content (JSON, Markdown) - python-docx → applies a deterministic template with your styles

def llm_to_docx(content_json: dict, template_path: str) -> str:
    """Generate a formatted .docx from LLM output."""
    from docx import Document
    doc = Document(template_path)
    # Apply styles programmatically
    doc.save("output.docx")
    return "output.docx"

If you want the LLM itself to output .docx files, see the Fine-tuning for tool use approach below.

2. Fine-tune for tool calling

If you want the model to call python-docx functions directly, your dataset should look like:

{
  "instruction": "Create a report with title 'Q1 Sales' in blue, bold, 16pt.",
  "input": "",
  "output": "{\"function\": \"create_docx\", \"args\": {\"title\": \"Q1 Sales\", \"style\": {\"font_size\": 16, \"bold\": true, \"color\": \"blue\"}}}"
}

3. Push to HuggingFace Hub

model.push_to_hub("your-username/model-name")
tokenizer.push_to_hub("your-username/model-name")

Troubleshooting

Problem	Solution
`CUDA out of memory`	Reduce `per_device_train_batch_size` to 1, reduce `max_seq_length` to 1024
Colab disconnected mid-training	Enable Save checkpoints to Google Drive every N steps
`pip install unsloth` hangs	Restart runtime (Runtime → Restart runtime)
Model doesn't follow instructions	Increase `num_train_epochs` to 5-10, check dataset quality
Kaggle: "No module named 'unsloth'"	Run Cell 2 (install) again after restarting
Slow training on 2×T4	Make sure `ddp_find_unused_parameters=False` is set

File Paths

docs/AI-ML/Fine Tuning/
├── unsloth-finetune-colab.ipynb    # Google Colab notebook
├── unsloth-finetune-kaggle.ipynb   # Kaggle notebook
└── Unsloth Fine-Tuning — Colab vs Kaggle Guide.md   # This document