Measuring the WAL vs DB Performance Gap on Ceph USB OSDs
Nine of my fifteen Ceph OSDs use WAL-only acceleration while six use DB. I set out to measure the performance gap and discovered the real story isn't WAL vs DB — it's the USB 3.0 hardware ceiling that dominates everything. The matched-hardware comparison shows DB is 5-15% faster on reads, not the 32% that naive cross-node testing suggested.
SSH Key-Based Access to a Proxmox Cluster
Typing passwords every time you SSH into a Proxmox node gets old fast — especially with a six-node cluster. Here's how to set up SSH key authentication and an SSH config so you can just type 'ssh harlan' and be in.
Hybrid Ceph Storage: SSD WAL/DB Acceleration with USB Drive Data
Running Ceph on USB drives sounds crazy until you put the WAL and DB on an SSD. Here's how separating metadata onto a Crucial MX500 transformed my 15-OSD homelab cluster from sluggish to surprisingly capable — at a fraction of all-SSD costs.
When ZFS and Ceph Problems Collide: Diagnosing Overlapping Failures on Proxmox
A routine ZFS scrub alert on harlan turned into a multi-hour debugging session when a hostid mismatch fix collided with a pre-existing Ceph OSD failure from a dead USB drive. Here's how overlapping storage problems can mask each other and how to untangle them.
Ceph OSD Recovery After Power Failure: SAN Switch Was Dead the Whole Time
A power outage knocked my Ceph cluster from 15 healthy OSDs down to 4. The recovery took days of debugging — heartbeat cascades, a ceph.conf misconfiguration, and a dead SAN switch hiding behind NO-CARRIER flags on every node.
USB Drive SMART Updates: Fast-Track to the GRUB Solution
New Seagate USB drives arrived for the Ceph cluster and predictably won't report SMART data. After months of production use, GRUB boot parameters are the only method that reliably survives kernel updates and cluster reboots.
Homelab Storage Economics: Ceph vs Single Drive Costs
Real-world cost-per-GB analysis of distributed Ceph storage versus single-drive solutions in a homelab — the same infrastructure investment framework applies at any scale.
Debian Linux Oh-Crap Moment in the Homelab
I renamed /usr to /usr-root as root. On a live system. Here's the recovery — and what the mistake teaches about operational discipline.
Essential CLI Tools for Linux System Management
The CLI tools I actually reach for when Linux systems misbehave: htop, iotop, ncdu, iftop, mtr, and a handful of specialists for hardware and performance diagnosis.
Enabling SMART Monitoring on Seagate USB Drives
Seagate USB drives hide SMART data behind UAS quirks. Disabling UAS via GRUB boot parameters unlocks full health monitoring without sacrificing stability.
Dell Wyse 3040 eMMC Storage Health Monitoring
eMMC storage doesn't support SMART, but the JEDEC health reporting built into mmc-utils gives you life time estimation and pre-EOL status for Dell Wyse 3040 boot drives.
Optimizing Ceph Performance in Proxmox Homelab
Performance tuning Ceph on USB storage and constrained hardware — mClock configuration, IOPS optimization, and the realities of USB 3.0 as a storage tier.
Managing Ceph Nearfull Warnings in Proxmox Homelab
Ceph's nearfull warnings are capacity planning signals, not emergencies — if you understand the thresholds and respond before the cluster goes read-only.
Proxmox 8 Lessons Learned in the Homelab
Hard-won lessons from running Proxmox since version 7.4 through 8.2.2 — upgrade gotchas, Ceph integration tips, and the patterns that apply at any scale.
Upcoming Articles Roadmap: September - December 2025
A publishing schedule for the remaining 16 articles planned through end of 2025, covering Proxmox, Ceph, hardware monitoring, CLI tools, and Jekyll optimization.
Debian 12 SystemD nightly reboots on Dell Wyse 3040s
Adding a nightly reboot schedule to RAM-constrained Dell Wyse 3040 Proxmox nodes using SystemD timers instead of crontab.
Backlog of Posts from 2024
A status update on the growing draft backlog heading into 2025 — what got written, what's still waiting, and what keeps getting added.
Power Supply upgrade for GPUs in the Homelab
Evaluating a 750W Dell OptiPlex PSU upgrade to support a discrete GPU in an aging Proxmox homelab node, with a breakdown of PCIe and connector power limits.
Ceph Cluster Complete Removal on Proxmox for the Homelabs
How to completely remove a broken Ceph cluster from Proxmox — every service, package, config file, and orphaned mount point — so you can start fresh.
Linux Disk I/O Performance in the Homelab
Quick disk I/O benchmarks comparing eMMC and USB3 storage on Dell Wyse 3040s — understanding the performance characteristics of your storage tier before making capacity decisions.
Powerline Networking for the Homelabs
Powerline networking repeatedly tripped the AFCI breakers in a modern house — turns out arc fault interrupters and powerline adapters are fundamentally incompatible.
Proxmox VE 8.1 to 8.2 upgrade issues in the Homelabs
An extended power outage plus a remote kernel upgrade left half the cluster unbootable, video outputs dead, and Ceph unstable — a crash cart recovery story with four overlapping root causes.
ProxMox 8.2.4 Upgrade on Dell Wyse 3040s
Ceph Monitor refuses to start after a Proxmox 8.2.4 upgrade because root partitions hit 95% — fixed by clearing apt cache, removing atop logs, and purging old PVE and Debian kernels.
Tailscale on Dell Wyse 3040 with Debian 12
Deploying Tailscale on Dell Wyse 3040 thin clients as always-on subnet router nodes connecting three geographically separate homelab networks.
Proxmox Ceph settings for the Homelab
Tuning Ceph scrub and deep-scrub intervals to reduce wear on spinning rust drives in a homelab cluster — spreading daily scrubs over 7 days and weekly deep scrubs over 28 days.
ProxMox 8.2.2 Cluster on Dell Wyse 3040s
Building a three-node Proxmox 8 + Ceph test cluster on Dell Wyse 3040 thin clients to safely evaluate SDN and Ceph configuration changes without risking the semi-production cluster.
Debian 12 on Dell Wyse 3040s
Installing Debian 12 on Dell Wyse 3040 thin clients: BIOS setup, Ventoy boot, the GRUB EFI removable media fix for the bad BIOS, and the dw_dmac blacklist that prevents lockups on shutdown.
Dell Wyse 3040 CMOS CR2032 Battery Replacement
Replacing the proprietary CMOS battery connector on Dell Wyse 3040 thin clients using heat shrink tubing and standard CR2032 cells — $23 for nine units versus $90 for pre-made replacements.
Hard Drives for the Homelabs
Storage has hit a penny per GB. A 20TB renewed drive for $200 and a 20TB USB-C enclosure for $220 make the math work for a Ceph offsite backup strategy.
Backlog of Posts
A mid-2024 status dump of unpublished drafts and stalled projects: site-to-site VPN, LAG/LACP, PiKVM, Ceph performance, Kubernetes, and a growing media server backlog.
ProxMox 8.2 for the Homelabs
Building a Proxmox 8 + Ceph HA cluster on decade-old Dell OptiPlex 990 hardware with Seagate USB drives — the background, the hardware inventory, and why Ceph beat a Synology NAS.
Ceph Cluster rebalance issue
Fixing a severely imbalanced Ceph cluster where OSDs added in batches left the first three drives at 75-85% while newer ones sat nearly empty — by tuning backfill and recovery parallelism.
Buying a 10Gbps or higher network on a homelab budget
Evaluating the options for 10Gbps homelab networking on a budget: DAC point-to-point rings, affordable MikroTik SFP+ switches, and the HP ProCurve 5406zl chassis — with no decision reached yet.