RunPod — Private Unsloth Fine-Tuning Guide

Overview

RunPod provides dedicated GPU instances in the cloud. You get root SSH access to a machine with a high-end GPU (RTX 3090, RTX 4090, A100, etc.) and pay by the second. Your data never touches shared infrastructure — it is as private as your own machine.

This guide covers the full workflow: create a pod, install Unsloth, upload your private dataset, fine-tune, and retrieve the results.

[!IMPORTANT] RunPod is for private fine-tuning. Unlike Colab and Kaggle, no one else can see your data. No automated scanning. No ToS restrictions on your dataset.

---

Pricing (Community Cloud)

GPU	VRAM	Price/h	Time for 3B LoRA	Total cost
RTX 3090	24 GB	~$0.20	~15-20 min	~$0.07
RTX 4090	24 GB	~$0.34	~10-15 min	~$0.08
A5000	24 GB	~$0.28	~15-20 min	~$0.09
A100 40 GB	40 GB	~$0.59	~8-12 min	~$0.10

[!NOTE] Secure Cloud costs ~2x more but guarantees dedicated hardware with no co-tenants. Community Cloud shares the physical node but isolates you via Docker — adequate for most privacy needs.

---

Step 1: Create a RunPod Account

1. Go to runpod.io
2. Sign up (email or Google)
3. Add credits: minimum $10 typically lasts for 10-50 training runs
4. Go to Pods → Deploy

---

Step 2: Deploy a Pod

Configuration

Setting	Value
GPU Type	RTX 3090 (best value) or RTX 4090
Template	RunPod PyTorch (Ubuntu 22.04 + CUDA 12.1 + Python 3.10)
Storage	50 GB (enough for models + datasets) or 100 GB to be safe
Container Disk	5 GB
Exposed Ports	None needed (SSH only)

SSH Key Setup (required)

1. Generate an SSH key pair on your local machine if you don't have one:

   ssh-keygen -t ed25519 -f ~/.ssh/runpod
   

2. Copy the public key:

   cat ~/.ssh/runpod.pub
   

3. In RunPod's deploy screen, paste it into the SSH Public Key field.

Deploy

Click Deploy On-Demand. Wait ~2 minutes for the pod to start.

---

Step 3: Connect to the Pod

Once the pod status shows Running, find the connection details:

1. Click your pod → copy the SSH Command (looks like ssh -p 12345 root@123.45.67.89)
2. Connect from your terminal:

   ssh -i ~/.ssh/runpod -p <port> root@<ip>
   

3. Verify the GPU:

   nvidia-smi
   

   You should see an RTX 3090 / 4090 with ~24 GB.

---

Step 4: Install Requirements

# Inside the pod
cd /workspace

# Update system
apt-get update && apt-get install -y git-lfs

# Install Unsloth
git clone https://github.com/unslothai/unsloth.git
cd unsloth
pip install -e .

# Verify
python -c "from unsloth import FastLanguageModel; print('✓ Unsloth ready')"

---

Step 5: Upload Your Private Dataset

Option A: scp (easiest for small datasets)

From your local machine:

scp -i ~/.ssh/runpod -P <port> my_dataset.json root@<ip>:/workspace/

Option B: Direct download from private storage

# S3
aws s3 cp s3://my-private-bucket/dataset.json /workspace/

# HuggingFace private dataset
huggingface-cli login
huggingface-cli download my-private-org/my-dataset --local-dir /workspace/data

Option C: Git clone private repo

git clone https://github.com/your-org/private-dataset.git /workspace/data

---

Step 6: Run Training

Create a training script train.py in /workspace/:

from unsloth import FastLanguageModel
from datasets import Dataset
from trl import SFTTrainer
from transformers import TrainingArguments
import json, torch

MODEL_NAME = "unsloth/Llama-3.2-3B-Instruct"
MAX_SEQ_LENGTH = 2048

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME, max_seq_length=MAX_SEQ_LENGTH,
    dtype=None, load_in_4bit=True,
)

# Attach LoRA
model = FastLanguageModel.get_peft_model(
    model, r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16, lora_dropout=0, bias="none",
    use_gradient_checkpointing="unsloth", random_state=42,
)

# Load your private dataset
with open("/workspace/my_dataset.json") as f:
    raw_data = json.load(f)

def format_example(ex):
    text = f"""Below is an instruction. Write a response.

### Instruction:\n{ex['instruction']}\n\n### Response:\n{ex['output']}"""
    return {"text": text}

dataset = Dataset.from_list(raw_data).map(format_example)

# Train
trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=dataset, dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=5,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1, optim="adamw_8bit",
        weight_decay=0.01, seed=42,
        output_dir="/workspace/output", report_to="none",
    ),
)

trainer.train()

# Save LoRA adapters (tiny ~10 MB)
model.save_pretrained("/workspace/output/lora-adapter")
tokenizer.save_pretrained("/workspace/output/lora-adapter")
print(f"✓ Model saved to /workspace/output/lora-adapter")

Run it:

cd /workspace
python train.py

---

Step 7: Download Your Fine-Tuned Model

Option A: scp (for small LoRA adapters)

# From your local machine
scp -i ~/.ssh/runpod -P <port> -r root@<ip>:/workspace/output/lora-adapter ./my-private-model/

Option B: Zip and download (faster for multiple files)

# On the pod
cd /workspace/output
zip -r /workspace/finetuned-lora.zip lora-adapter/

# From your local machine
scp -i ~/.ssh/runpod -P <port> root@<ip>:/workspace/finetuned-lora.zip ./

Option C: Sync to your own storage

aws s3 sync /workspace/output s3://my-private-bucket/finetuned-model/

---

Step 8: Terminate the Pod

1. Go to RunPod dashboard → Pods
2. Click the pod → Terminate
3. Confirm

Everything is wiped. Your only copy is what you downloaded.

[!WARNING] Always download your results before terminating. Pod storage is ephemeral and cannot be recovered after termination.

---

Template: Complete One-Shot Script

Save this as run_training.sh and execute it on a fresh pod:

#!/bin/bash
set -e

echo "=== Installing Unsloth ==="
cd /workspace
git clone https://github.com/unslothai/unsloth.git
cd unsloth && pip install -e . -q

echo "=== Creating training script ==="
cat > /workspace/train.py << 'PYEOF'
from unsloth import FastLanguageModel
from datasets import Dataset
from trl import SFTTrainer
from transformers import TrainingArguments
import json, torch

MODEL_NAME = "unsloth/Llama-3.2-3B-Instruct"
MAX_SEQ_LENGTH = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME, max_seq_length=MAX_SEQ_LENGTH,
    dtype=None, load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model, r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16, lora_dropout=0, bias="none",
    use_gradient_checkpointing="unsloth", random_state=42,
)

with open("/workspace/data/train.json") as f:
    raw_data = json.load(f)

def fmt(ex):
    return {"text": f"### Instruction:\\n{ex['instruction']}\\n\\n### Response:\\n{ex['output']}"}

dataset = Dataset.from_list(raw_data).map(fmt)

trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=dataset, dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=5,
        learning_rate=2e-4,
        fp16=True, logging_steps=1,
        optim="adamw_8bit", weight_decay=0.01,
        seed=42, output_dir="/workspace/output",
        report_to="none",
    ),
)

trainer.train()
model.save_pretrained("/workspace/output/lora-adapter")
tokenizer.save_pretrained("/workspace/output/lora-adapter")
print("✓ Done")
PYEOF

echo "=== Training ==="
python /workspace/train.py

echo "=== Packaging ==="
cd /workspace/output && zip -r /workspace/finetuned-lora.zip lora-adapter/
echo "✅ /workspace/finetuned-lora.zip ready for download"

Usage:

# 1. Upload your dataset
scp -i ~/.ssh/runpod -P <port> my_dataset.json root@<ip>:/workspace/data/train.json

# 2. Upload and run the script
scp -i ~/.ssh/runpod -P <port> run_training.sh root@<ip>:/workspace/
ssh -i ~/.ssh/runpod -p <port> root@<ip> "bash /workspace/run_training.sh"

# 3. Download the result
scp -i ~/.ssh/runpod -P <port> root@<ip>:/workspace/finetuned-lora.zip ./

---

Cost Examples

Scenario	GPU	Time	Cost
First test: 3B model, 50 examples, 5 epochs	RTX 3090	12 min	$0.04
Real dataset: 3B model, 500 examples, 5 epochs	RTX 3090	45 min	$0.15
Large: 7B model, 1000 examples, 5 epochs	RTX 3090	2 h	$0.40
Fine-tune + download + terminate	RTX 4090	30 min	$0.17

[!TIP] | A typical experiment cycle costs $0.04-0.40. With an initial $10 credit, you can run 25-250 training runs before needing to top up.

---

Troubleshooting

Problem	Solution
CUDA out of memory	Reduce per_device_train_batch_size to 1 or set max_seq_length to 1024
Pod stuck on "Initializing"	Wait 3-5 min or terminate and re-deploy
ssh: connection refused	Pod is still starting. Wait 2 min and retry.
pip install fails	Make sure you used the RunPod PyTorch template (has CUDA pre-installed)
Need to transfer > 1 GB dataset	Use rsync instead of scp for resume support
Want persistent storage	Attach a Network Volume ($0.07/GB/month) — survives pod termination

---

When to Use RunPod vs Other Options

Situation	Choice
Data is confidential / proprietary	RunPod
Budget is primary concern	Kaggle (free, 30h/wk)
Quick 15-min test	Colab (free, T4)
Need A100 80 GB for big model	RunPod A100 ($0.79/h)
Want to automate many experiments	RunPod + script
Never want data to leave your desk	Local 8 GB + TinyLlama

---

File Path

docs/AI-ML/Fine Tuning/
├── unsloth-finetune-colab.ipynb
├── unsloth-finetune-kaggle.ipynb
├── Unsloth Fine-Tuning — Colab vs Kaggle Guide.md
└── RunPod — Private Unsloth Fine-Tuning Guide.md          # ← This document