487 lines
13 KiB
Markdown
487 lines
13 KiB
Markdown
---
|
||
title: Full Guide - Install Ubuntu Server and Configure Ollama with CodeGemma and Phi-3 Mini
|
||
description: AI Project
|
||
published: true
|
||
date: 2025-06-06T12:27:30.919Z
|
||
tags: ollama, ai, guide, walk-though, ubuntu server, server
|
||
editor: markdown
|
||
dateCreated: 2025-06-06T12:27:28.985Z
|
||
---
|
||
|
||
# Install Ubuntu Server and Configure Ollama with CodeGemma and Phi-3 Mini
|
||
|
||
This guide provides step-by-step instructions to set up a headless **Ubuntu Server 24.04 LTS** on a PC with the following specs, install **Ollama** with **CodeGemma 7B** for user `arti` (Python coding assistance) and **Phi-3 Mini (3.8B)** for user `phixr` (system administration tasks), and restrict each user’s SSH access to their respective interactive AI session:
|
||
|
||
- **GPU**: Radeon RX 6600 (8 GB VRAM)
|
||
- **CPU**: AMD Ryzen 7 2700 (8 cores, ~3.2 GHz)
|
||
- **RAM**: 64 GB (2133 MT/s)
|
||
- **Storage**: 465.8 GB NVMe SSD (`nvme0n1`), 2x 931.5 GB SSDs (`sda`, `sdb`)
|
||
|
||
The setup is command-line only, with no desktop environment or window manager, and assumes you’re replacing any existing OS (e.g., Proxmox). Both models use Q4_K_M quantization to fit within 8 GB VRAM and <20 GB disk space, leveraging ROCm for GPU acceleration.
|
||
|
||
---
|
||
|
||
## Step 1: Prepare for Ubuntu Server Installation
|
||
|
||
Let’s prepare to install Ubuntu Server on your NVMe SSD, replacing any existing OS.
|
||
|
||
1. **Download Ubuntu Server 24.04 LTS**:
|
||
- On another computer, download the ISO:
|
||
```bash
|
||
wget https://releases.ubuntu.com/24.04/ubuntu-24.04-live-server-amd64.iso
|
||
```
|
||
- Or download manually from [ubuntu.com](https://ubuntu.com/download/server).
|
||
- Verify the ISO:
|
||
```bash
|
||
sha256sum ubuntu-24.04-live-server-amd64.iso
|
||
```
|
||
- Check the hash against [Ubuntu’s checksums](https://releases.ubuntu.com/24.04/).
|
||
|
||
2. **Create a Bootable USB Drive**:
|
||
- Use a USB drive (≥4 GB). Identify it with:
|
||
```bash
|
||
lsblk
|
||
```
|
||
- Write the ISO (replace `/dev/sdX` with your USB device):
|
||
```bash
|
||
sudo dd if=ubuntu-24.04-live-server-amd64.iso of=/dev/sdX bs=4M status=progress && sync
|
||
```
|
||
- **Warning**: Double-check `/dev/sdX` to avoid overwriting other drives.
|
||
- Alternatively, use Rufus (Windows) or Etcher (cross-platform).
|
||
|
||
3. **Backup Existing Data**:
|
||
- If replacing Proxmox or another OS, back up data to an external drive or another system:
|
||
```bash
|
||
scp -r /path/to/data user@other-machine:/destination
|
||
```
|
||
|
||
4. **Boot from USB**:
|
||
- Insert the USB, reboot, and enter the BIOS (usually `Del` or `F2`).
|
||
- Set the USB as the first boot device.
|
||
- Save and reboot to start the Ubuntu installer.
|
||
|
||
---
|
||
|
||
## Step 2: Install Ubuntu Server 24.04 LTS
|
||
|
||
Let’s install Ubuntu Server on the NVMe SSD (`nvme0n1`, 465.8 GB).
|
||
|
||
1. **Start the Installer**:
|
||
- Select “Install Ubuntu Server”.
|
||
- Set language (English), keyboard layout, and network (DHCP or static IP).
|
||
|
||
2. **Configure Storage**:
|
||
- Choose “Custom storage layout”.
|
||
- Partition `nvme0n1`:
|
||
- **EFI Partition**: 1 GB, `fat32`, mount at `/boot/efi`.
|
||
- **Root Partition**: 464.8 GB, `ext4`, mount at `/`.
|
||
- Example (in installer):
|
||
- Select `nvme0n1`, create partitions as above.
|
||
- Write changes and confirm.
|
||
- Optional: Use `sda` or `sdb` (931.5 GB SSDs) for additional storage (e.g., mount as `/data`).
|
||
|
||
3. **Set Up Users and SSH**:
|
||
- Set hostname (e.g., `ai-server`).
|
||
- Create an admin user (e.g., `admin`):
|
||
- Username: `admin`
|
||
- Password: Set a secure password.
|
||
- Enable “Install OpenSSH server”.
|
||
- Skip importing SSH keys unless needed.
|
||
|
||
4. **Complete Installation**:
|
||
- Select no additional packages (Ollama and ROCm will be installed later).
|
||
- Finish and reboot.
|
||
|
||
5. **Verify Boot**:
|
||
- Remove the USB, boot into Ubuntu, and log in as `admin` via a local terminal or SSH:
|
||
```bash
|
||
ssh admin@<server-ip>
|
||
```
|
||
|
||
---
|
||
|
||
## Step 3: Install AMD ROCm for Radeon RX 6600
|
||
|
||
Let’s set up ROCm to enable GPU acceleration for Ollama.
|
||
|
||
1. **Update System**:
|
||
```bash
|
||
sudo apt update && sudo apt upgrade -y
|
||
```
|
||
|
||
2. **Add ROCm Repository**:
|
||
- Install dependencies and add ROCm 5.7:
|
||
```bash
|
||
sudo apt install -y wget gnupg
|
||
wget -qO - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -
|
||
echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/5.7 ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list
|
||
```
|
||
|
||
3. **Install ROCm**:
|
||
```bash
|
||
sudo apt update
|
||
sudo apt install -y rocm-libs rocminfo
|
||
```
|
||
|
||
4. **Verify ROCm**:
|
||
- Reboot:
|
||
```bash
|
||
sudo reboot
|
||
```
|
||
- Check GPU:
|
||
```bash
|
||
rocminfo
|
||
```
|
||
- Look for “Navi 23 [Radeon RX 6600]”.
|
||
- Check VRAM:
|
||
```bash
|
||
rocm-smi --showmeminfo vram
|
||
```
|
||
- Expect ~8192 MB.
|
||
|
||
5. **Troubleshooting**:
|
||
- If no GPU is detected, verify:
|
||
```bash
|
||
lspci | grep -i vga
|
||
```
|
||
- Try ROCm 5.6:
|
||
```bash
|
||
echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/5.6 ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list
|
||
sudo apt update && sudo apt install -y rocm-libs
|
||
```
|
||
|
||
---
|
||
|
||
## Step 4: Install Ollama and Models
|
||
|
||
Let’s install Ollama and download both **CodeGemma 7B** and **Phi-3 Mini**.
|
||
|
||
1. **Install Ollama**:
|
||
```bash
|
||
curl -fsSL https://ollama.com/install.sh | sh
|
||
```
|
||
- Verify:
|
||
```bash
|
||
ollama --version
|
||
```
|
||
|
||
2. **Pull CodeGemma 7B**:
|
||
- Download Q4_K_M (~4.2 GB):
|
||
```bash
|
||
ollama pull codegemma:7b
|
||
```
|
||
- Verify:
|
||
```bash
|
||
ollama list
|
||
```
|
||
- Expect `codegemma:7b` (q4_k_m).
|
||
|
||
3. **Test CodeGemma**:
|
||
- Run:
|
||
```bash
|
||
ollama run codegemma:7b
|
||
```
|
||
- Prompt: “Debug: `x = [1, 2]; print(x[2])`.”
|
||
- Expected: “Check the list’s length with `len(x)`.”
|
||
- Exit: `Ctrl+D`.
|
||
|
||
4. **Pull Phi-3 Mini**:
|
||
- Download Q4_K_M (~2.3 GB):
|
||
```bash
|
||
ollama pull phi3:mini
|
||
```
|
||
- Verify:
|
||
```bash
|
||
ollama list
|
||
```
|
||
- Expect `phi3:mini` (q4_k_m).
|
||
|
||
5. **Test Phi-3 Mini**:
|
||
- Run:
|
||
```bash
|
||
ollama run phi3:mini
|
||
```
|
||
- Prompt: “Walk me through configuring a firewall.”
|
||
- Expected: “Install `ufw` with `sudo apt install ufw`. Enable with `sudo ufw enable`.”
|
||
- Exit: `Ctrl+D`.
|
||
|
||
6. **Verify GPU Usage**:
|
||
- During a session, check:
|
||
```bash
|
||
rocm-smi
|
||
```
|
||
- CodeGemma: ~5–6 GB VRAM.
|
||
- Phi-3 Mini: ~3.5–4.5 GB VRAM.
|
||
|
||
7. **Enable Ollama Service**:
|
||
```bash
|
||
sudo systemctl enable ollama
|
||
sudo systemctl start ollama
|
||
```
|
||
- Verify:
|
||
```bash
|
||
systemctl status ollama
|
||
```
|
||
|
||
---
|
||
|
||
## Step 5: Configure User `arti` for CodeGemma 7B
|
||
|
||
Let’s restrict `arti`’s SSH access to an interactive CodeGemma 7B session for Python coding.
|
||
|
||
1. **Create User `arti`**:
|
||
```bash
|
||
sudo adduser arti
|
||
```
|
||
- Set a secure password, optional details (e.g., full name: “Artificial Intelligence”).
|
||
|
||
2. **Restrict Home Directory**:
|
||
```bash
|
||
sudo chown arti:arti /home/arti
|
||
sudo chmod 700 /home/arti
|
||
```
|
||
- Verify:
|
||
```bash
|
||
ls -ld /home/arti
|
||
```
|
||
- Expect: `drwx------ arti arti`
|
||
|
||
3. **Create Shell Script**:
|
||
```bash
|
||
sudo nano /usr/local/bin/ollama-shell
|
||
```
|
||
- Add:
|
||
```bash
|
||
#!/bin/bash
|
||
echo "Starting CodeGemma 7B interactive session..."
|
||
/usr/bin/ollama run codegemma:7b
|
||
```
|
||
- Save and exit.
|
||
- Make executable:
|
||
```bash
|
||
sudo chmod +x /usr/local/bin/ollama-shell
|
||
sudo chown root:root /usr/local/bin/ollama-shell
|
||
sudo chmod 755 /usr/local/bin/ollama-shell
|
||
```
|
||
|
||
4. **Set Shell**:
|
||
```bash
|
||
sudo usermod -s /usr/local/bin/ollama-shell arti
|
||
```
|
||
- Verify:
|
||
```bash
|
||
getent passwd arti
|
||
```
|
||
- Expect: `arti:x:1000:1000:,,,:/home/arti:/usr/local/bin/ollama-shell`
|
||
|
||
5. **Add GPU Access**:
|
||
```bash
|
||
sudo usermod -a -G render arti
|
||
```
|
||
|
||
6. **Restrict SSH**:
|
||
```bash
|
||
sudo nano /etc/ssh/sshd_config
|
||
```
|
||
- Add:
|
||
```bash
|
||
Match User arti
|
||
ForceCommand /usr/local/bin/ollama-shell
|
||
```
|
||
- Restart SSH:
|
||
```bash
|
||
sudo systemctl restart sshd
|
||
```
|
||
|
||
7. **Limit Permissions**:
|
||
```bash
|
||
sudo usermod -G nogroup arti
|
||
```
|
||
|
||
8. **Test SSH**:
|
||
```bash
|
||
ssh arti@<server-ip>
|
||
```
|
||
- Expect: `Starting CodeGemma 7B interactive session...`
|
||
- Prompt: “Debug: `x = '5'; y = 3; print(x + y)`.”
|
||
- Expected: “Check types with `type(x)`.”
|
||
- Exit: `Ctrl+D` (terminates SSH).
|
||
- Try: `ssh arti@<server-ip> bash` (should fail).
|
||
|
||
---
|
||
|
||
## Step 6: Configure User `phixr` for Phi-3 Mini
|
||
|
||
Let’s restrict `phixr`’s SSH access to a Phi-3 Mini session for system administration.
|
||
|
||
1. **Create User `phixr`**:
|
||
```bash
|
||
sudo adduser phixr
|
||
```
|
||
- Set password, optional details (e.g., full name: “Phi-3 System Admin”).
|
||
|
||
2. **Restrict Home Directory**:
|
||
```bash
|
||
sudo chown phixr:phixr /home/phixr
|
||
sudo chmod 700 /home/phixr
|
||
```
|
||
- Verify:
|
||
```bash
|
||
ls -ld /home/phixr
|
||
```
|
||
- Expect: `drwx------ phixr phixr`
|
||
|
||
3. **Create Shell Script**:
|
||
```bash
|
||
sudo nano /usr/local/bin/ollama-phi3-shell
|
||
```
|
||
- Add:
|
||
```bash
|
||
#!/bin/bash
|
||
echo "Starting Phi-3 Mini interactive session..."
|
||
/usr/bin/ollama run phi3:mini
|
||
```
|
||
- Save and exit.
|
||
- Make executable:
|
||
```bash
|
||
sudo chmod +x /usr/local/bin/ollama-phi3-shell
|
||
sudo chown root:root /usr/local/bin/ollama-phi3-shell
|
||
sudo chmod 755 /usr/local/bin/ollama-phi3-shell
|
||
```
|
||
|
||
4. **Set Shell**:
|
||
```bash
|
||
sudo usermod -s /usr/local/bin/ollama-phi3-shell phixr
|
||
```
|
||
- Verify:
|
||
```bash
|
||
getent passwd phixr
|
||
```
|
||
- Expect: `phixr:x:1001:1001:,,,:/home/phixr:/usr/local/bin/ollama-phi3-shell`
|
||
|
||
5. **Add GPU Access**:
|
||
```bash
|
||
sudo usermod -a -G render phixr
|
||
```
|
||
|
||
6. **Restrict SSH**:
|
||
```bash
|
||
sudo nano /etc/ssh/sshd_config
|
||
```
|
||
- Add (below `Match User arti`):
|
||
```bash
|
||
Match User phixr
|
||
ForceCommand /usr/local/bin/ollama-phi3-shell
|
||
```
|
||
- Restart SSH:
|
||
```bash
|
||
sudo systemctl restart sshd
|
||
```
|
||
|
||
7. **Limit Permissions**:
|
||
```bash
|
||
sudo usermod -G nogroup phixr
|
||
```
|
||
|
||
8. **Test SSH**:
|
||
```bash
|
||
ssh phixr@<server-ip>
|
||
```
|
||
- Expect: `Starting Phi-3 Mini interactive session...`
|
||
- Prompt: “Walk me through installing pfSense.”
|
||
- Expected: “Download the ISO from pfsense.org. Create a USB with `dd if=pfSense.iso of=/dev/sdX bs=4M`.”
|
||
- Exit: `Ctrl+D` (terminates SSH).
|
||
- Try: `ssh phixr@<server-ip> bash` (should fail).
|
||
|
||
---
|
||
|
||
## Step 7: Optimize and Troubleshoot
|
||
|
||
Let’s ensure optimal performance and address potential issues.
|
||
|
||
1. **Performance Optimization**:
|
||
- **CodeGemma 7B**: ~5–6 GB VRAM, ~8–12 tokens/second. Good for Python debugging.
|
||
- **Phi-3 Mini**: ~3.5–4.5 GB VRAM, ~10–15 tokens/second. Ideal for system administration guidance.
|
||
- **Prompting**:
|
||
- `arti`: “Debug this Python code: [snippet].”
|
||
- `phixr`: “Walk me through [task] step-by-step.”
|
||
- **Temperature**: For precise responses, set temperature to 0.2:
|
||
- For CodeGemma:
|
||
```bash
|
||
nano ~/.ollama/models/codegemma-modelfile
|
||
```
|
||
Add:
|
||
```
|
||
FROM codegemma:7b
|
||
PARAMETER temperature 0.2
|
||
```
|
||
Create:
|
||
```bash
|
||
ollama create codegemma-lowtemp -f ~/.ollama/models/codegemma-modelfile
|
||
```
|
||
Update `/usr/local/bin/ollama-shell` to use `ollama run codegemma-lowtemp`.
|
||
- For Phi-3 Mini:
|
||
```bash
|
||
nano ~/.ollama/models/phi3-modelfile
|
||
```
|
||
Add:
|
||
```
|
||
FROM phi3:mini
|
||
PARAMETER temperature 0.2
|
||
```
|
||
Create:
|
||
```bash
|
||
ollama create phi3-lowtemp -f ~/.ollama/models/phi3-modelfile
|
||
```
|
||
Update `/usr/local/bin/ollama-phi3-shell` to use `ollama run phi3-lowtemp`.
|
||
|
||
2. **Troubleshooting**:
|
||
- **No Session**:
|
||
- Check scripts:
|
||
```bash
|
||
ls -l /usr/local/bin/ollama-shell /usr/local/bin/ollama-phi3-shell
|
||
cat /usr/local/bin/ollama-shell
|
||
cat /usr/local/bin/ollama-phi3-shell
|
||
```
|
||
- **GPU Issues**: If slow (~1–5 tokens/second), verify ROCm:
|
||
```bash
|
||
rocminfo
|
||
rocm-smi --showmeminfo vram
|
||
```
|
||
- Reinstall ROCm 5.6/5.7 if needed.
|
||
- **Shell Access**: If `arti` or `phixr` access Bash:
|
||
```bash
|
||
getent passwd arti
|
||
getent passwd phixr
|
||
```
|
||
- Confirm shells. Re-run `usermod -s`.
|
||
- **SSH Errors**:
|
||
```bash
|
||
sudo systemctl status sshd
|
||
```
|
||
- Restart: `sudo systemctl restart sshd`.
|
||
|
||
---
|
||
|
||
## Expected Performance
|
||
- **Hardware Fit**: CodeGemma (~5–6 GB VRAM, ~4.2 GB disk) and Phi-3 Mini (~3.5–4.5 GB VRAM, ~2.3 GB disk) fit your Radeon RX 6600, Ryzen 7 2700, 64 GB RAM, and 465.8 GB NVMe SSD.
|
||
- **Use Case**:
|
||
- `arti`: Guides Python coding/debugging (e.g., “Check your list index with `len()`”).
|
||
- `phixr`: Provides detailed system administration instructions (e.g., “Download pfSense ISO, then use `dd`”).
|
||
- **Speed**: CodeGemma (~8–12 tokens/second), Phi-3 Mini (~10–15 tokens/second). Responses in ~1–2 seconds.
|
||
- **Restriction**: `arti` locked to CodeGemma; `phixr` to Phi-3 Mini. No Bash access.
|
||
|
||
## Example Usage
|
||
- **For `arti`**:
|
||
```bash
|
||
ssh arti@<server-ip>
|
||
>>> Debug: x = [1, 2]; print(x[2]).
|
||
The error suggests an invalid index. Check the list’s length with `len(x)`.
|
||
```
|
||
- **For `phixr`**:
|
||
```bash
|
||
ssh phixr@<server-ip>
|
||
>>> Walk me through installing pfSense.
|
||
Download the ISO from pfsense.org. Create a USB with `dd if=pfSense.iso of=/dev/sdX bs=4M`. Check with `lsblk`.
|
||
```
|