<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://mcgarrah.org/feed.xml" rel="self" type="application/atom+xml" /><link href="https://mcgarrah.org/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-07-14T11:25:39+00:00</updated><id>https://mcgarrah.org/feed.xml</id><title type="html">McGarrah Technical Blog</title><subtitle>Engineering deep dives on infrastructure, distributed systems, and AI/ML — from homelab experiments to enterprise architecture.</subtitle><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><entry><title type="html">Jekyll Run Plugin: Fixing the Multi-Root Workspace Crash</title><link href="https://mcgarrah.org/jekyll-run-plugin-multiroot-workspace-bug/" rel="alternate" type="text/html" title="Jekyll Run Plugin: Fixing the Multi-Root Workspace Crash" /><published>2026-07-14T00:00:00+00:00</published><updated>2026-07-14T00:00:00+00:00</updated><id>https://mcgarrah.org/jekyll-run-plugin-multiroot-workspace-bug</id><content type="html" xml:base="https://mcgarrah.org/jekyll-run-plugin-multiroot-workspace-bug/"><![CDATA[<p>In my <a href="/jekyll-run-vscode-plugin-local-development/">previous post on the Jekyll Run plugin</a>, I covered how to configure the extension for local development — the flags that matter, settings precedence, and the <code class="language-plaintext highlighter-rouge">_config.yml</code> trap. That post assumed the plugin actually starts. This one covers what happens when it doesn’t.</p>

<p>If you use a multi-root workspace in VS Code on macOS, the Jekyll Run extension crashes before it even tries to run Jekyll. The error is unhelpful:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TypeError: Cannot read properties of null (reading 'toString')
</code></pre></div></div>

<p>No stack trace in the UI. No hint about what’s null. Just a dead button. I spent hours chasing this through VS Code settings, plugin source code, and compiled JavaScript patches before finding the real cause — and it had nothing to do with any of that. Misleading error messages that point to the wrong layer are the most expensive bugs to diagnose — true in VS Code extensions and equally true in distributed systems.</p>

<!-- excerpt-end -->

<h2 id="the-symptom">The Symptom</h2>

<p>You open a multi-root workspace with your Jekyll blog and other projects. You click the Jekyll Run button in the status bar. Instead of building your site, you get the TypeError. The extension does nothing.</p>

<p>The same workspace, same plugin version, same settings — works fine on WSL2. Only macOS is broken.</p>

<h2 id="blind-alley-1-vs-code-settings-precedence">Blind Alley #1: VS Code Settings Precedence</h2>

<p>My first theory was settings precedence. The plugin reads its configuration via <code class="language-plaintext highlighter-rouge">getConfiguration().get('jekyll-run')</code>, and I knew VS Code settings come from multiple sources with complex override rules. In a multi-root workspace, an empty <code class="language-plaintext highlighter-rouge">"settings": {}</code> in the <code class="language-plaintext highlighter-rouge">.code-workspace</code> file could shadow per-folder settings.</p>

<p>I added <code class="language-plaintext highlighter-rouge">jekyll-run</code> settings to every possible location:</p>

<table>
  <thead>
    <tr>
      <th>Location</th>
      <th>Result</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">.vscode/settings.json</code> (workspace folder)</td>
      <td>Crash</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">.code-workspace</code> settings block</td>
      <td>Crash</td>
    </tr>
    <tr>
      <td>VS Code User settings</td>
      <td>Crash</td>
    </tr>
    <tr>
      <td>All three simultaneously</td>
      <td>Crash</td>
    </tr>
  </tbody>
</table>

<p>None of them fixed it. The settings were correct everywhere, but the plugin still crashed.</p>

<h2 id="blind-alley-2-plugin-source-code">Blind Alley #2: Plugin Source Code</h2>

<p>I dug into the plugin’s compiled JavaScript. The <code class="language-plaintext highlighter-rouge">Config.get()</code> method uses an unscoped API call:</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// out/config/config.js</span>
<span class="k">return</span> <span class="nx">vscode</span><span class="p">.</span><span class="nx">workspace</span><span class="p">.</span><span class="nf">getConfiguration</span><span class="p">().</span><span class="nf">get</span><span class="p">(</span><span class="nx">extension</span><span class="p">);</span>
</code></pre></div></div>

<p>The VS Code API documentation recommends <code class="language-plaintext highlighter-rouge">getConfiguration('jekyll-run')</code> (passing the section name directly) instead of <code class="language-plaintext highlighter-rouge">getConfiguration().get('jekyll-run')</code> (fetching the root and extracting a key). The unscoped form can return <code class="language-plaintext highlighter-rouge">null</code> in multi-root workspaces.</p>

<p>I patched <code class="language-plaintext highlighter-rouge">config.js</code> to use the scoped form. The TypeError persisted. The settings were resolving correctly — the crash was happening somewhere else entirely.</p>

<h2 id="blind-alley-3-zshenv">Blind Alley #3: <code class="language-plaintext highlighter-rouge">.zshenv</code></h2>

<p>I noticed the plugin spawns <code class="language-plaintext highlighter-rouge">bundle exec jekyll serve</code> with <code class="language-plaintext highlighter-rouge">{ shell: true }</code>. On macOS, VS Code launched from the Dock doesn’t source <code class="language-plaintext highlighter-rouge">~/.zshrc</code>. I created <code class="language-plaintext highlighter-rouge">~/.zshenv</code> (which is supposed to be sourced by all zsh instances, including non-interactive ones) with the Homebrew Ruby PATH.</p>

<p>Fully quit VS Code, reopened — still system Ruby 2.6. VS Code’s extension host process isn’t a zsh instance, so <code class="language-plaintext highlighter-rouge">.zshenv</code> is never sourced.</p>

<h2 id="the-breakthrough-developer-tools">The Breakthrough: Developer Tools</h2>

<p>The actual error was hiding in plain sight. Opening VS Code’s Developer Tools (<code class="language-plaintext highlighter-rouge">Cmd+Shift+P</code> → “Toggle Developer Tools” → Console tab) revealed the real error above the TypeError:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>stderr: /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems.rb:283:
in `find_spec_for_exe': Could not find 'bundler' (4.0.8) required by your Gemfile.lock.
(Gem::GemNotFoundException)
</code></pre></div></div>

<p>The plugin was running <strong>macOS system Ruby 2.6</strong> — not my Homebrew Ruby. System Ruby doesn’t have the correct bundler version, so <code class="language-plaintext highlighter-rouge">bundle exec jekyll serve</code> fails immediately.</p>

<p>The TypeError was a secondary crash in the plugin’s error handler. When stderr contains “ruby”, the plugin tries to extract an Errno message:</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">reject</span><span class="p">(</span><span class="nx">error</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="sr">/</span><span class="se">\B(</span><span class="sr">.+</span><span class="se">)</span><span class="sr">Errno</span><span class="se">(</span><span class="sr">.+</span><span class="se">)</span><span class="sr">/m</span><span class="p">));</span>
</code></pre></div></div>

<p>The bundler error doesn’t contain “Errno”, so <code class="language-plaintext highlighter-rouge">match()</code> returns <code class="language-plaintext highlighter-rouge">null</code>. The plugin calls <code class="language-plaintext highlighter-rouge">reject(null)</code>, and the caller does <code class="language-plaintext highlighter-rouge">null.toString()</code> — crash. The real error (wrong Ruby) is swallowed. The user sees only the misleading TypeError.</p>

<h2 id="the-root-cause-macos-gui-path-inheritance">The Root Cause: macOS GUI PATH Inheritance</h2>

<p>On macOS, applications launched from the Dock, Spotlight, or Finder inherit their environment from <code class="language-plaintext highlighter-rouge">launchd</code>, not from your shell. Your <code class="language-plaintext highlighter-rouge">~/.zshrc</code> PATH modifications — including Homebrew Ruby — are invisible to VS Code when launched this way.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Your terminal sees:</span>
which ruby    → /opt/homebrew/opt/ruby/bin/ruby  <span class="o">(</span>Homebrew Ruby 4.0<span class="o">)</span>

<span class="c"># VS Code's spawned processes see:</span>
which ruby    → /usr/bin/ruby                     <span class="o">(</span>macOS system Ruby 2.6<span class="o">)</span>
</code></pre></div></div>

<p>WSL2 works because VS Code Remote Server runs inside a login shell that sources <code class="language-plaintext highlighter-rouge">~/.zshrc</code>. Same plugin, same workspace, same settings — but different environment inheritance model.</p>

<h2 id="the-solution-rbenv--launch-from-terminal">The Solution: rbenv + Launch from Terminal</h2>

<p>The fix has three parts:</p>

<h3 id="1-use-rbenv-for-ruby-version-management">1. Use rbenv for Ruby Version Management</h3>

<p>Homebrew Ruby 4.0 has its own problem — <code class="language-plaintext highlighter-rouge">--livereload</code> causes a thread exception popup with Jekyll 4.4.1. Rather than fight bleeding-edge compatibility issues, use <code class="language-plaintext highlighter-rouge">rbenv</code> to pin the project to Ruby 3.3, which works perfectly with all Jekyll features:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install rbenv</span>
brew <span class="nb">install </span>rbenv ruby-build

<span class="c"># Install Ruby 3.3</span>
rbenv <span class="nb">install </span>3.3.11

<span class="c"># Set project-local Ruby version</span>
<span class="nb">cd </span>mcgarrah.github.io
rbenv <span class="nb">local </span>3.3.11    <span class="c"># creates .ruby-version</span>

<span class="c"># Install gems under Ruby 3.3</span>
gem <span class="nb">install </span>bundler
bundle <span class="nb">install</span>
</code></pre></div></div>

<h3 id="2-initialize-rbenv-in-your-shell">2. Initialize rbenv in Your Shell</h3>

<p>Add to <code class="language-plaintext highlighter-rouge">~/.zshrc</code> (before <code class="language-plaintext highlighter-rouge">source $ZSH/oh-my-zsh.sh</code>):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># rbenv Ruby version manager (macOS only)</span>
<span class="c"># Uses .ruby-version in project directories to select the correct Ruby</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-s</span><span class="si">)</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"Darwin"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">eval</span> <span class="s2">"</span><span class="si">$(</span>rbenv init - zsh<span class="si">)</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>

<p>And create <code class="language-plaintext highlighter-rouge">~/.zshenv</code> so non-interactive shells (like those VS Code spawns) can find rbenv’s shims:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># rbenv shims for non-interactive shells (VS Code extensions)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-s</span><span class="si">)</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"Darwin"</span> <span class="o">&amp;&amp;</span> <span class="nt">-d</span> <span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.rbenv"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.rbenv/shims:</span><span class="nv">$HOME</span><span class="s2">/.rbenv/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>

<h3 id="3-launch-vs-code-from-a-terminal">3. Launch VS Code from a Terminal</h3>

<p>macOS GUI apps don’t source any shell configuration files. The <code class="language-plaintext highlighter-rouge">~/.zshenv</code> file helps with some non-interactive shells, but VS Code’s extension host may still not pick it up reliably. The bulletproof fix is to launch VS Code from a terminal where your PATH is correct:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>code ~/Personal/Github-mcgarrah/articles-workspace.code-workspace
</code></pre></div></div>

<p>When VS Code is launched this way, it inherits the terminal’s environment, including rbenv’s shims. The plugin finds the correct <code class="language-plaintext highlighter-rouge">bundle</code> and <code class="language-plaintext highlighter-rouge">ruby</code> executables, and Jekyll starts with all features working — including <code class="language-plaintext highlighter-rouge">--livereload</code>.</p>

<h2 id="why-ruby-33-instead-of-40">Why Ruby 3.3 Instead of 4.0</h2>

<p>Homebrew Ruby 4.0.2 works for Jekyll builds, but <code class="language-plaintext highlighter-rouge">--livereload</code> triggers a thread exception:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#&lt;Thread:0x00000001266bb850 .../live_reload_reactor.rb:39 run&gt; terminated with exception
</code></pre></div></div>

<p>The site still serves, but the popup is disruptive. Ruby 3.3.11 has no such issue — all Jekyll 4.4.1 features work correctly, including livereload.</p>

<p>Using <code class="language-plaintext highlighter-rouge">rbenv</code> with a <code class="language-plaintext highlighter-rouge">.ruby-version</code> file means:</p>
<ul>
  <li>The Jekyll project uses Ruby 3.3 (stable, fully compatible)</li>
  <li>Other projects can use whatever Ruby they need</li>
  <li>The version is committed to the repository so collaborators get the same Ruby</li>
  <li>GitHub Actions can read <code class="language-plaintext highlighter-rouge">.ruby-version</code> to match the local development environment</li>
</ul>

<h2 id="the-plugin-bug-still-real">The Plugin Bug (Still Real)</h2>

<p>The misleading TypeError is a genuine bug in the plugin’s error handling, separate from the PATH issue. The plugin hasn’t been updated since 2020 (<a href="https://github.com/Kanna727/jekyll-run">GitHub: Kanna727/jekyll-run</a>).</p>

<p>Two issues worth a PR:</p>

<p><strong>1. Null rejection on non-Errno errors</strong> in <code class="language-plaintext highlighter-rouge">src/cmds/run.ts</code>:</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-            reject(error.match(/\B(.+)Errno(.+)/m));
</span><span class="gi">+            reject(error.match(/\B(.+)Errno(.+)/m) || error);
</span></code></pre></div></div>

<p><strong>2. Unscoped getConfiguration call</strong> in <code class="language-plaintext highlighter-rouge">src/config/config.ts</code>:</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-        return vscode.workspace.getConfiguration().get(extension) as any;
</span><span class="gi">+        return vscode.workspace.getConfiguration(extension);
</span></code></pre></div></div>

<p>The first fix ensures users see the actual Ruby error instead of a cryptic TypeError. The second fix properly resolves settings in multi-root workspaces. Neither is required if you follow the rbenv + terminal launch approach above, but both would make the plugin more robust.</p>

<h2 id="complete-setup-reference">Complete Setup Reference</h2>

<h3 id="shell-configuration-zshrc">Shell Configuration (<code class="language-plaintext highlighter-rouge">~/.zshrc</code>)</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># rbenv Ruby version manager (macOS only)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-s</span><span class="si">)</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"Darwin"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">eval</span> <span class="s2">"</span><span class="si">$(</span>rbenv init - zsh<span class="si">)</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>

<h3 id="shell-environment-zshenv">Shell Environment (<code class="language-plaintext highlighter-rouge">~/.zshenv</code>)</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># rbenv shims for non-interactive shells (VS Code extensions)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-s</span><span class="si">)</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"Darwin"</span> <span class="o">&amp;&amp;</span> <span class="nt">-d</span> <span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.rbenv"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.rbenv/shims:</span><span class="nv">$HOME</span><span class="s2">/.rbenv/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>

<h3 id="project-ruby-version-ruby-version">Project Ruby Version (<code class="language-plaintext highlighter-rouge">.ruby-version</code>)</h3>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3.3.11
</code></pre></div></div>

<h3 id="vs-code-settings-vscodesettingsjson">VS Code Settings (<code class="language-plaintext highlighter-rouge">.vscode/settings.json</code>)</h3>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
    </span><span class="nl">"jekyll-run.commandLineArguments"</span><span class="p">:</span><span class="w"> </span><span class="s2">"--trace --drafts --future --unpublished --livereload --incremental"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"jekyll-run.stopServerOnExit"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="multi-root-workspace-code-workspace">Multi-Root Workspace (<code class="language-plaintext highlighter-rouge">.code-workspace</code>)</h3>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
    </span><span class="nl">"settings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"jekyll-run.commandLineArguments"</span><span class="p">:</span><span class="w"> </span><span class="s2">"--trace --drafts --future --unpublished --livereload --incremental"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"jekyll-run.stopServerOnExit"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
    </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="fallback-script-start-jekyllsh">Fallback Script (<code class="language-plaintext highlighter-rouge">start-jekyll.sh</code>)</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
bundle <span class="nb">exec </span>jekyll serve <span class="nt">--trace</span> <span class="nt">--drafts</span> <span class="nt">--future</span> <span class="nt">--unpublished</span> <span class="nt">--livereload</span> <span class="nt">--incremental</span>
</code></pre></div></div>

<h2 id="known-issue-port-conflict-shows-thread-exception">Known Issue: Port Conflict Shows Thread Exception</h2>

<p>If a Jekyll server is already running on port 4000 (from a terminal <code class="language-plaintext highlighter-rouge">./start-jekyll.sh</code> session, for example), clicking the Jekyll Run button won’t show a clean “port in use” error. Instead, you’ll see a thread exception popup:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#&lt;Thread:0x0000000126cc9a58 .../live_reload_reactor.rb:39 run&gt; terminated with exception
</code></pre></div></div>

<p>This happens because the plugin’s port-detection code parses <code class="language-plaintext highlighter-rouge">lsof</code> output by splitting on single spaces, but macOS <code class="language-plaintext highlighter-rouge">lsof</code> uses variable-width columns with multiple spaces between fields. The PID is never extracted, so the plugin thinks port 4000 is free, launches Jekyll, and the livereload thread crashes on the port conflict before the main server reports <code class="language-plaintext highlighter-rouge">EADDRINUSE</code>.</p>

<p>The workaround is to not run both simultaneously — use either <code class="language-plaintext highlighter-rouge">./start-jekyll.sh</code> or the Jekyll Run button, not both. Kill the terminal server first (<code class="language-plaintext highlighter-rouge">Ctrl+C</code>) before using the plugin.</p>

<p>This is a separate plugin bug that will be covered in a future article on patching the source and submitting a PR.</p>

<h2 id="summary">Summary</h2>

<p>Three problems, layered on top of each other, each masking the next:</p>

<table>
  <thead>
    <tr>
      <th>Problem</th>
      <th>Symptom</th>
      <th>Fix</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>macOS GUI apps don’t inherit shell PATH</td>
      <td>System Ruby 2.6 runs instead of project Ruby</td>
      <td>Launch VS Code from terminal</td>
    </tr>
    <tr>
      <td>Plugin rejects <code class="language-plaintext highlighter-rouge">null</code> on non-Errno Ruby errors</td>
      <td>TypeError instead of useful error message</td>
      <td>Plugin bug (PR needed)</td>
    </tr>
    <tr>
      <td>Ruby 4.0 livereload incompatibility</td>
      <td>Thread exception popup</td>
      <td>Use Ruby 3.3 via rbenv</td>
    </tr>
    <tr>
      <td>Plugin <code class="language-plaintext highlighter-rouge">lsof</code> parsing broken on macOS</td>
      <td>Port conflict shows thread exception instead of clean error</td>
      <td>Don’t run terminal server and plugin simultaneously; plugin bug (PR needed)</td>
    </tr>
  </tbody>
</table>

<p>The first one is the blocker. The second makes debugging harder. The third drove the choice of Ruby 3.3 over 4.0. The fourth is a nuisance if you forget to stop a terminal server before using the plugin.</p>

<h2 id="related-posts">Related Posts</h2>

<ul>
  <li><a href="/jekyll-run-vscode-plugin-local-development/">Jekyll Run Plugin: Local Development Settings That Actually Work</a> — Complete configuration guide for the extension</li>
  <li><a href="/github-pages-jekyll-locally/">Running GitHub Pages Jekyll Locally</a> — Initial local development setup</li>
</ul>

<h2 id="references">References</h2>

<ul>
  <li><a href="https://marketplace.visualstudio.com/items?itemName=Dedsec727.jekyll-run">Jekyll Run Extension</a> — VS Code Marketplace</li>
  <li><a href="https://github.com/Kanna727/jekyll-run">Jekyll Run Source Code</a> — GitHub repository (last updated 2020)</li>
  <li><a href="https://github.com/rbenv/rbenv">rbenv</a> — Ruby version manager</li>
  <li><a href="https://code.visualstudio.com/api/references/vscode-api#workspace.getConfiguration">VS Code getConfiguration API</a> — Official API documentation</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="web-development" /><category term="technical" /><category term="jekyll" /><category term="jekyll" /><category term="vscode" /><category term="multi-root-workspace" /><category term="plugin-bug" /><category term="github-pages" /><category term="ruby" /><category term="rbenv" /><category term="macos" /><summary type="html"><![CDATA[Diagnosing and fixing the Jekyll Run VS Code extension crash in multi-root workspaces on macOS. A debugging story covering the misleading null toString TypeError, blind alleys through VS Code settings and plugin source code, the real root cause (macOS GUI PATH inheritance), and the complete solution using rbenv with Ruby 3.3.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/jekyll-run-plugin-multiroot-workspace-bug.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/jekyll-run-plugin-multiroot-workspace-bug.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">The Google Services Tax on a Simple Blog</title><link href="https://mcgarrah.org/google-service-sprawl/" rel="alternate" type="text/html" title="The Google Services Tax on a Simple Blog" /><published>2026-07-07T00:00:00+00:00</published><updated>2026-07-07T00:00:00+00:00</updated><id>https://mcgarrah.org/google-service-sprawl</id><content type="html" xml:base="https://mcgarrah.org/google-service-sprawl/"><![CDATA[<p>Vendor service sprawl is a real operational cost, and Google is a case study in how “free” services accumulate complexity. Running a technical blog now requires integrating Analytics, Search Console, Custom Search, and AdSense — each with its own dashboard, approval process, privacy implications, and integration quirks. What should be a solved problem for a static site has become a multi-month project with opaque feedback loops and interconnected dependencies that don’t always play nicely together.</p>

<p>Lions and Tigers and AdSense, oh my.</p>

<!-- excerpt-end -->

<h2 id="the-google-adsense-nightmare">The Google AdSense Nightmare</h2>

<p>I’ve been trying to get Google AdSense back online with my website for over a year with no success. I had it working on WordPress prior to 2016, but after migrating to Jekyll and consolidating decades of content, Google flagged my site for “duplicate content that appeared to be plagiarized.”</p>

<p>The irony? It was <strong>my own content</strong> from my previous blogs and websites, consolidated into one place. Fast forward to 2025, and I have over half my content generated in the last two years, yet AdSense still rejects my applications with vague, unhelpful feedback.</p>

<h2 id="the-service-integration-treadmill">The Service Integration Treadmill</h2>

<p>Each Google service requires its own setup dance, and they’re all interconnected in ways that aren’t immediately obvious:</p>

<p><strong>Google Analytics</strong>: Set up in August 2024 during a website modernization push. Relatively straightforward, but required privacy policy updates and cookie consent considerations.</p>

<p><strong>Google Search Console</strong>: This one bit me hard. Spent months with a broken sitemap because I had <code class="language-plaintext highlighter-rouge">localhost:4000</code> URLs instead of my actual domain. The error messages were cryptic, and it took forever to realize the URL configuration issue.</p>

<p><strong>Google Custom Search</strong>: A weird beast that sort of works but feels like a compromise. Better than no search, but the styling integration is a nightmare and it doesn’t always index new content promptly.</p>

<p><strong>Google AdSense</strong>: The white whale. Still rejected after multiple attempts, vague feedback, and jumping through hoops like adding privacy policies, fixing “content quality” issues, and ensuring GDPR compliance.</p>

<h2 id="the-google-service-complexity-matrix">The Google Service Complexity Matrix</h2>

<p><strong>Services I’m Actually Using:</strong></p>

<ul>
  <li><strong>Google Analytics</strong>: Works well once configured, but requires privacy policy updates and cookie consent banners for GDPR compliance</li>
  <li><strong>Google Search Console</strong>: Useful when it works, but error messages are cryptic and debugging sitemap issues is painful</li>
  <li><strong>Google Custom Search</strong>: Functional but feels like a hack - styling integration is terrible and indexing is inconsistent</li>
  <li><strong>Google AdSense</strong>: The promised land I can’t reach - endless rejections with vague feedback</li>
</ul>

<p><strong>Services I Think I Should Be Using:</strong></p>

<ul>
  <li><strong>Google Tag Manager</strong>: Supposedly simplifies tag management, but adds another layer of complexity</li>
  <li><strong>Google PageSpeed Insights</strong>: Helpful for performance, but recommendations often conflict with other Google service requirements</li>
  <li><strong>Google Optimize</strong>: A/B testing sounds great, but requires Tag Manager integration (more complexity)</li>
  <li><strong>Google Trends</strong>: Useful for content planning, but another dashboard to monitor</li>
</ul>

<h2 id="the-real-problem">The Real Problem</h2>

<p>Each service exists in its own silo with its own:</p>

<ul>
  <li>Dashboard and interface</li>
  <li>Approval process and requirements</li>
  <li>Integration complexity</li>
  <li>Privacy and compliance implications</li>
  <li>Performance impact on the website</li>
</ul>

<p>What should be a simple “add analytics and search” becomes a multi-month project involving privacy policies, cookie banners, multiple dashboards, and endless troubleshooting of interconnected systems that don’t always play nicely together.</p>

<h2 id="the-frustration-factor">The Frustration Factor</h2>

<p>The most maddening part? Google’s own services don’t integrate seamlessly with each other. You’d think Google Analytics and Google Search Console would share data more effectively, or that Google Custom Search would automatically index content that Google Search Console knows about.</p>

<p>Instead, each service feels like it was built by a different team with different assumptions about how websites work. The result is a fragmented experience that requires significant time investment to get working properly.</p>

<p>Maybe I’m overthinking this, but for a simple technical blog, the overhead of managing all these Google services sometimes feels like it outweighs the benefits. I just want a blog site to publish fun stuff I’m doing outside of work. On the upside, I’m keeping current on the same integration challenges that small and medium businesses face every day — vendor service sprawl, opaque approval processes, and the hidden cost of “free” platforms. Having managed similar vendor integration complexity in enterprise environments, I can say the pattern is identical at any scale — it just costs more when the stakes are higher.</p>

<h2 id="what-ive-actually-done-about-it">What I’ve Actually Done About It</h2>

<p>I’m not just complaining — I’ve been systematically attacking each of these problems. The results are mixed:</p>

<ul>
  <li><strong>GDPR compliance</strong> — Built a full <a href="/implementing-gdpr-compliance-jekyll-adsense/">cookie consent implementation</a> with region detection, consent management, and conditional script loading. This was a prerequisite for everything else.</li>
  <li><strong>AdSense approval</strong> — Documented the <a href="/adsense-approval-failure-remediation/">debugging process</a> after repeated “site isn’t ready” rejections. Fixed the <a href="/adsense-verification-gdpr-script-loading-fix/">GDPR script loading interaction</a> that was blocking the verification crawler. Improved <a href="/improving-eeat-jekyll-adsense/">E-E-A-T signals</a> across the site. <strong>Still not approved.</strong> The rejections continue with the same vague feedback.</li>
  <li><strong>Sitemap issues</strong> — Fixed <a href="/jekyll-sitemap-bloat-tags-categories-pagination/">sitemap bloat</a> from tags, categories, and pagination pages that were diluting the signal for actual content.</li>
  <li><strong>SEO infrastructure</strong> — Built an <a href="/jekyll-github-actions-cicd-pipeline/">automated SEO health check</a> into the CI/CD pipeline that validates canonical URLs, meta tags, structured data, and broken links on every push.</li>
  <li><strong>Search Console crawling</strong> — Still fighting crawler access issues. The Google bot reports pages it can’t reach, but the same pages load fine in a browser. The debugging cycle is: wait for crawl → read vague error → change something → wait weeks for re-crawl → repeat.</li>
</ul>

<p>Months of work. Dozens of commits. Multiple published articles documenting the journey. And AdSense still says “no” without telling me why.</p>

<h2 id="the-ongoing-battle">The Ongoing Battle</h2>

<p>The frustrating truth is that there’s no finish line. Google’s approval processes are opaque, their feedback is generic, and their timelines are measured in weeks. You can do everything right — proper structured data, clean sitemaps, GDPR compliance, quality content, good E-E-A-T signals — and still get rejected with “Your site isn’t ready.”</p>

<p>I’ll keep documenting the fight. At minimum, the infrastructure improvements make the site better regardless of whether Google ever approves the ads. And the knowledge transfers directly to professional work — every small business website faces these same integration challenges.</p>

<p>Cheers from the Homelab.</p>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="web-development" /><category term="technical" /><category term="google" /><category term="web-development" /><category term="analytics" /><category term="search" /><category term="adsense" /><category term="seo" /><category term="frustration" /><summary type="html"><![CDATA[An honest look at the overhead of integrating Google Analytics, Search Console, Custom Search, and AdSense into a Jekyll blog. The complexity tax is real, the approval process is opaque, and the battle continues.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/google-service-sprawl.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/google-service-sprawl.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Building a Draft Preview Site for Jekyll — Part 3: The Implementation</title><link href="https://mcgarrah.org/jekyll-draft-preview-site-part-3/" rel="alternate" type="text/html" title="Building a Draft Preview Site for Jekyll — Part 3: The Implementation" /><published>2026-06-30T00:00:00+00:00</published><updated>2026-06-30T00:00:00+00:00</updated><id>https://mcgarrah.org/jekyll-draft-preview-site-part-3</id><content type="html" xml:base="https://mcgarrah.org/jekyll-draft-preview-site-part-3/"><![CDATA[<p>In <a href="/jekyll-draft-preview-site-part-1/">Part 1</a>, I explored the options. In <a href="/jekyll-draft-preview-site-part-2/">Part 2</a>, I refined the design. Now it’s time to build it.</p>

<!-- excerpt-end -->

<p>This is Part 3 of a three-part series:</p>
<ul>
  <li><strong>Part 1</strong>: <a href="/jekyll-draft-preview-site-part-1/">Exploring every option I considered</a></li>
  <li><strong>Part 2</strong>: <a href="/jekyll-draft-preview-site-part-2/">Refining the design — config, workflow, feedback, and gaps</a></li>
  <li><strong>Part 3</strong> (this post): The complete implementation</li>
</ul>

<blockquote>
  <p><strong>Status:</strong> The system is live at <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> and deploying reliably on every push to <code class="language-plaintext highlighter-rouge">main</code>. What follows captures the real issues and fixes from first-run deployments — not just the design intent, but what actually happened.</p>
</blockquote>

<h2 id="creating-the-drafts-repo">Creating the Drafts Repo</h2>

<p>The deployment target is <code class="language-plaintext highlighter-rouge">mcgarrah/drafts.mcgarrah.org</code> — a public repo that contains only built HTML output. I chose public because:</p>

<ul>
  <li>GitHub Pages on free accounts requires public repos</li>
  <li>The source markdown is already public in <code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code></li>
  <li>The HTML is Staticrypt-encrypted, so the rendered content isn’t casually browsable</li>
  <li>The repo has no source code of value — it’s a deployment target, not a project</li>
</ul>

<p>Setup steps:</p>
<ol>
  <li>Created the repo initialized with a <code class="language-plaintext highlighter-rouge">README.md</code> (critical — see below)</li>
  <li>Enabled GitHub Pages: Settings → Pages → Deploy from branch → <code class="language-plaintext highlighter-rouge">main</code> → <code class="language-plaintext highlighter-rouge">/ (root)</code></li>
  <li>Enabled GitHub Discussions with a dedicated “Draft Reviews” category for Giscus feedback</li>
  <li>Generated a fine-grained GitHub PAT scoped to only the <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> repo with <code class="language-plaintext highlighter-rouge">repo</code> permissions</li>
  <li>Added <code class="language-plaintext highlighter-rouge">DRAFTS_DEPLOY_TOKEN</code> and <code class="language-plaintext highlighter-rouge">DRAFTS_PASSWORD</code> as secrets on the <code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code> repo</li>
</ol>

<p><strong>The empty-repo trap:</strong> GitHub Pages cannot be configured on a truly empty repo because there’s no <code class="language-plaintext highlighter-rouge">main</code> branch to select. The Pages settings page just shows an error. The repo needs at least one commit before <code class="language-plaintext highlighter-rouge">main</code> exists and Pages can be enabled. Initializing with a <code class="language-plaintext highlighter-rouge">README.md</code> avoids this entirely.</p>

<h2 id="dns-configuration">DNS Configuration</h2>

<p>One-time setup in Porkbun:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drafts.mcgarrah.org  CNAME  mcgarrah.github.io.
</code></pre></div></div>

<p>Propagation took under five minutes. After the first successful deployment, I enabled “Enforce HTTPS” in the drafts repo’s Pages settings — GitHub provisions a Let’s Encrypt certificate automatically.</p>

<h2 id="the-config-overlay">The Config Overlay</h2>

<p>The real <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code> that’s deployed:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://drafts.mcgarrah.org"</span>
<span class="na">canonical_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://drafts.mcgarrah.org"</span>
<span class="na">baseurl</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">draft_preview_site</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">main_site_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://mcgarrah.org"</span>

<span class="c1"># Disable production tracking and ads on the drafts preview site.</span>
<span class="na">google_analytics</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">google_adsense</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">google_cse_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>

<span class="c1"># Enable Giscus comments for the drafts preview site.</span>
<span class="na">giscus</span><span class="pi">:</span>
  <span class="na">repo</span><span class="pi">:</span> <span class="s">mcgarrah/drafts.mcgarrah.org</span>
  <span class="na">repo_id</span><span class="pi">:</span> <span class="s">R_kgDOSG6Quw</span>
  <span class="na">category</span><span class="pi">:</span> <span class="s">Draft Reviews</span>
  <span class="na">category_id</span><span class="pi">:</span> <span class="s">DIC_kwDOSG6Qu84C7PMZ</span>
  <span class="na">mapping</span><span class="pi">:</span> <span class="s">pathname</span>
  <span class="na">strict</span><span class="pi">:</span> <span class="m">0</span>
  <span class="na">reactions_enabled</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">emit_metadata</span><span class="pi">:</span> <span class="m">0</span>
  <span class="na">input_position</span><span class="pi">:</span> <span class="s">top</span>
  <span class="na">theme</span><span class="pi">:</span> <span class="s">preferred_color_scheme</span>
  <span class="na">lang</span><span class="pi">:</span> <span class="s">en</span>
  <span class="na">loading</span><span class="pi">:</span> <span class="s">lazy</span>

<span class="c1"># Mark every rendered page as noindex on the drafts site.</span>
<span class="na">defaults</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">scope</span><span class="pi">:</span>
      <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
    <span class="na">values</span><span class="pi">:</span>
      <span class="na">noindex</span><span class="pi">:</span> <span class="kc">true</span>
</code></pre></div></div>

<p>Key decisions baked into this config:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">draft_preview_site: true</code> and <code class="language-plaintext highlighter-rouge">main_site_url</code> power the orange preview banner in the layout</li>
  <li>The <code class="language-plaintext highlighter-rouge">defaults:</code> block sets <code class="language-plaintext highlighter-rouge">noindex: true</code> on every page site-wide. The layout picks this up as <code class="language-plaintext highlighter-rouge">&lt;meta name="robots" content="noindex, follow"&gt;</code> — the primary defense against search engine indexing</li>
  <li>Giscus points to the drafts repo’s Discussions, keeping preview feedback completely separate from production comments</li>
  <li>Analytics, AdSense, and custom search are all blanked — no tracking on the preview site</li>
</ul>

<h2 id="the-final-workflow">The Final Workflow</h2>

<p>The workflow evolved significantly from the sketch in Part 2. The biggest change: <strong>selective encryption</strong>. Instead of encrypting every HTML file on the site, the workflow identifies only draft and future-dated post pages and encrypts those. Already-published content stays unencrypted — it’s public anyway, and encrypting it just adds build time and makes navigation annoying.</p>

<p>Here’s the structure of <code class="language-plaintext highlighter-rouge">.github/workflows/deploy-drafts.yml</code>:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Deploy Draft Preview Site</span>

<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">branches</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">main"</span><span class="pi">]</span>
  <span class="na">workflow_dispatch</span><span class="pi">:</span>

<span class="na">concurrency</span><span class="pi">:</span>
  <span class="na">group</span><span class="pi">:</span> <span class="s">drafts-pages</span>
  <span class="na">cancel-in-progress</span><span class="pi">:</span> <span class="kc">true</span>
</code></pre></div></div>

<p>The concurrency group ensures only one drafts deployment runs at a time — if I push twice in quick succession, the second run cancels the first rather than racing.</p>

<h3 id="build-step">Build Step</h3>

<p>Standard Jekyll build with the config overlay:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build with Jekyll (drafts + future)</span>
  <span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec jekyll build --drafts --future --config _config.yml,_config_drafts.yml</span>
  <span class="na">env</span><span class="pi">:</span>
    <span class="na">JEKYLL_ENV</span><span class="pi">:</span> <span class="s">production</span>
</code></pre></div></div>

<h3 id="selective-encryption">Selective Encryption</h3>

<p>This is the most complex step. The workflow:</p>

<ol>
  <li>Scans <code class="language-plaintext highlighter-rouge">_drafts/</code> for files matching the <code class="language-plaintext highlighter-rouge">YYYY-MM-DD-*.md</code> pattern (skipping convenience files like <code class="language-plaintext highlighter-rouge">DRAFTS.md</code>)</li>
  <li>Scans <code class="language-plaintext highlighter-rouge">_posts/</code> for future-dated files (post date &gt; today)</li>
  <li>Maps each source file to its rendered HTML in <code class="language-plaintext highlighter-rouge">_site/</code> using Jekyll’s permalink slug</li>
  <li>Encrypts each file individually in an isolated temp directory to avoid Staticrypt’s basename collision problem</li>
</ol>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Only encrypt if it starts with YYYY-MM-DD pattern (actual post format).</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$base</span><span class="s2">"</span> <span class="o">=</span>~ ^[0-9]<span class="o">{</span>4<span class="o">}</span>-[0-9]<span class="o">{</span>2<span class="o">}</span>-[0-9]<span class="o">{</span>2<span class="o">}</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span>add_target_for_slug <span class="s2">"</span><span class="si">$(</span>slug_from_source_file <span class="s2">"</span><span class="nv">$source_file</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>

<p>Each file is encrypted one at a time with hash-based verification:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">before_hash</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">sha256sum</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span><span class="s2">"</span>
<span class="nb">cp</span> <span class="s2">"</span><span class="nv">$encrypted_src</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span>
<span class="nv">after_hash</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">sha256sum</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span><span class="s2">"</span>

<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$before_hash</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"</span><span class="nv">$after_hash</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">echo</span> <span class="s2">"::error::Encryption did not modify </span><span class="nv">$file</span><span class="s2"> (hashes match)"</span>
  <span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre></div></div>

<p>Staticrypt flags used:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">--short</code> — compact password prompt page</li>
  <li><code class="language-plaintext highlighter-rouge">--remember 30</code> — stores the decryption key in <code class="language-plaintext highlighter-rouge">localStorage</code> for 30 days so reviewers don’t re-enter the password on every page click</li>
  <li><code class="language-plaintext highlighter-rouge">--template-title "Draft Preview - mcgarrah.org"</code> — custom title on the password prompt</li>
  <li><code class="language-plaintext highlighter-rouge">-c false</code> — disables Staticrypt’s config file creation (not needed in CI)</li>
  <li><code class="language-plaintext highlighter-rouge">-d &lt;temp_dir&gt;</code> — output to a temp directory, then copy back to preserve the original path structure</li>
</ul>

<h3 id="crawler-protections">Crawler Protections</h3>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Apply crawler protections</span>
  <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">printf "User-agent: *\nDisallow: /\n" &gt; _site/robots.txt</span>
    <span class="s">rm -f _site/feed.xml _site/sitemap.xml _site/sitemapindex.xml</span>

    <span class="s"># Remove feed discovery hints from generated pages</span>
    <span class="s">find _site -type f -name '*.html' \</span>
      <span class="s">-exec sed -i.bak '/type="application\/atom+xml"/d' {} +</span>
    <span class="s">find _site -type f -name '*.bak' -delete</span>
</code></pre></div></div>

<p>Three layers of protection:</p>
<ol>
  <li><code class="language-plaintext highlighter-rouge">robots.txt</code> with <code class="language-plaintext highlighter-rouge">Disallow: /</code> — tells well-behaved crawlers to stay away</li>
  <li><code class="language-plaintext highlighter-rouge">noindex</code> meta tags on every page (via the config overlay defaults)</li>
  <li>Feed and sitemap removal — prevents content leakage through structured formats. The <code class="language-plaintext highlighter-rouge">sed</code> step also strips <code class="language-plaintext highlighter-rouge">&lt;link rel="alternate" type="application/atom+xml"&gt;</code> tags from the HTML so there’s no discoverable feed URL even in the page source.</li>
</ol>

<h3 id="binary-artifact-filtering">Binary Artifact Filtering</h3>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Remove oversized binaries from deploy output</span>
  <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">rm -rf _site/assets/exes</span>
    <span class="s">find _site -type f \( -iname '*.exe' -o -iname '*.msi' \) -print -delete</span>
    <span class="s">find _site -type f -size +45M -print -delete</span>
</code></pre></div></div>

<p>The production site has some downloadable executables under <code class="language-plaintext highlighter-rouge">assets/exes/</code>. These triggered GitHub large-file warnings during deployment and aren’t needed for draft review. The 45MB guard catches anything else that shouldn’t be in a static site deployment.</p>

<h3 id="cross-repo-deploy">Cross-Repo Deploy</h3>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy to drafts.mcgarrah.org repo</span>
  <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">deploy_dir="$(mktemp -d)"</span>
    <span class="s">git clone --depth 1 --branch main \</span>
      <span class="s">"https://x-access-token:${DRAFTS_DEPLOY_TOKEN}@github.com/${TARGET_REPO}.git" \</span>
      <span class="s">"$deploy_dir"</span>

    <span class="s">find "$deploy_dir" -mindepth 1 -maxdepth 1 ! -name '.git' -exec rm -rf {} +</span>
    <span class="s">cp -a _site/. "$deploy_dir"/</span>

    <span class="s">echo "drafts.mcgarrah.org" &gt; "$deploy_dir/CNAME"</span>
    <span class="s">touch "$deploy_dir/.nojekyll"</span>

    <span class="s">cd "$deploy_dir"</span>
    <span class="s">git add -A</span>
    <span class="s">if git diff --cached --quiet; then</span>
      <span class="s">echo "No drafts-site changes to deploy"</span>
      <span class="s">exit 0</span>
    <span class="s">fi</span>

    <span class="s">git commit -m "Deploy drafts site from ${GITHUB_SHA}"</span>
    <span class="s">git push origin HEAD:main --force-with-lease</span>
</code></pre></div></div>

<p>A few details worth noting:</p>
<ul>
  <li>The <code class="language-plaintext highlighter-rouge">.nojekyll</code> file prevents GitHub Pages from re-processing the already-built HTML through Jekyll again</li>
  <li><code class="language-plaintext highlighter-rouge">--force-with-lease</code> is safer than <code class="language-plaintext highlighter-rouge">--force</code> — it fails if someone else pushed to the drafts repo since the clone, rather than silently overwriting</li>
  <li>The <code class="language-plaintext highlighter-rouge">git diff --cached --quiet</code> check skips the push entirely if nothing changed, avoiding empty commits</li>
  <li>The shallow clone (<code class="language-plaintext highlighter-rouge">--depth 1</code>) keeps the operation fast since we don’t need history</li>
</ul>

<h2 id="staticrypt-testing-results">Staticrypt Testing Results</h2>

<p>What works:</p>
<ul>
  <li>Draft and future post pages present the Staticrypt password prompt as expected</li>
  <li>The <code class="language-plaintext highlighter-rouge">--remember 30</code> flag stores the decryption key in <code class="language-plaintext highlighter-rouge">localStorage</code> — after entering the password once, subsequent pages decrypt automatically without re-prompting</li>
  <li>The encrypted HTML contains the Staticrypt wrapper, verified by both hash comparison and string detection in CI</li>
  <li>The custom template title (“Draft Preview - mcgarrah.org”) makes it clear which site you’re on even at the password prompt</li>
</ul>

<p>What needs more testing before broad reviewer rollout:</p>
<ul>
  <li>Cross-browser <code class="language-plaintext highlighter-rouge">--remember</code> behavior over longer sessions (does it survive browser updates?)</li>
  <li>Mobile UX around the password prompt and navigation flow</li>
  <li>Private/incognito windows (should always prompt — <code class="language-plaintext highlighter-rouge">localStorage</code> is session-scoped in incognito)</li>
</ul>

<h2 id="giscus-feedback">Giscus Feedback</h2>

<p>Giscus is configured to point at the <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> repo’s Discussions with a dedicated “Draft Reviews” category. This keeps preview feedback completely separate from production comments.</p>

<p>The Giscus <code class="language-plaintext highlighter-rouge">&lt;script&gt;</code> tag lives inside the encrypted HTML. After Staticrypt decrypts the page in-browser, the browser parses the decrypted DOM and the Giscus script loads normally. This works because Staticrypt replaces the entire page content with the decrypted HTML, which the browser then processes as if it were freshly loaded.</p>

<p>Reviewers need a GitHub account to leave comments. For reviewers without GitHub accounts, the preview banner includes a link back to the main site where they can use the contact page.</p>

<h2 id="the-preview-banner">The Preview Banner</h2>

<p>The banner is implemented in <code class="language-plaintext highlighter-rouge">_layouts/default.html</code> behind the <code class="language-plaintext highlighter-rouge">draft_preview_site</code> flag:</p>

<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{% if site.draft_preview_site %}
<span class="nt">&lt;div</span> <span class="na">style=</span><span class="s">"background:#e67e00;color:#fff;text-align:center;padding:0.5em 1em;font-size:0.9em;font-weight:bold;"</span><span class="nt">&gt;</span>
  ⚠ DRAFT PREVIEW SITE — unpublished content, may change.
  <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"{{ site.main_site_url }}"</span> <span class="na">style=</span><span class="s">"color:#fff;text-decoration:underline;margin-left:0.5em;"</span><span class="nt">&gt;</span>Go to the main site →<span class="nt">&lt;/a&gt;</span>
<span class="nt">&lt;/div&gt;</span>
{% endif %}
</code></pre></div></div>

<p>This turned out to be one of the highest-value additions. It’s immediately obvious to reviewers that they’re on the preview site, and the link back to production makes context-switching effortless.</p>

<h2 id="what-worked">What Worked</h2>

<ul>
  <li><strong>Config overlay approach</strong> cleanly separated production and drafts behavior without touching <code class="language-plaintext highlighter-rouge">_config.yml</code></li>
  <li><strong>Cross-repo deploy pipeline</strong> is stable and repeatable — push to <code class="language-plaintext highlighter-rouge">main</code> updates both sites automatically</li>
  <li><strong>Selective encryption</strong> reduced runtime and removed unnecessary friction on already-public pages</li>
  <li><strong>The preview banner</strong> had outsized impact on reviewer clarity for minimal implementation effort</li>
  <li><strong>Giscus on a separate repo</strong> keeps draft feedback isolated — when a post graduates to <code class="language-plaintext highlighter-rouge">_posts/</code>, the draft comments stay behind, having served their purpose</li>
  <li><strong>Archive ordering</strong> is now deterministic after adding front matter to convenience files</li>
</ul>

<h2 id="what-didnt-work">What Didn’t Work</h2>

<p>Early runs surfaced several workflow-level issues:</p>

<ol>
  <li><strong>GitHub Pages setup on empty repo failed</strong>: the repo needed an initial commit before <code class="language-plaintext highlighter-rouge">main</code> could be selected in Pages settings.</li>
  <li><strong>Encryption step appeared hung</strong>: processing hundreds of pages made the <code class="language-plaintext highlighter-rouge">Encrypt HTML with Staticrypt</code> step look stuck even when still running. The logs were hard to interpret.</li>
  <li><strong>Password prompt did not appear after first success</strong>: pages were deployed unencrypted. Root cause: Staticrypt v3.5.4+ doesn’t support the <code class="language-plaintext highlighter-rouge">-o</code> flag and silently ignores it. Switched to <code class="language-plaintext highlighter-rouge">-d &lt;directory&gt;</code>, but that flattens directory structures — all output goes to <code class="language-plaintext highlighter-rouge">&lt;dir&gt;/basename.html</code> regardless of input nesting. Fixed by processing each file individually in isolated temp directories.</li>
  <li><strong>Deployment included large executable files</strong>: binaries in deploy output triggered GitHub large-file warnings and highlighted the need for artifact filtering.</li>
  <li><strong>Full-site encryption created unnecessary overhead</strong>: encrypting already-public content increased runtime and complexity without adding value. Switched to selective encryption of draft and future posts only.</li>
  <li><strong>Initial verification check was too brittle</strong>: string matching on encrypted output caused false failures. Replaced with SHA-256 hash comparison to detect whether files were actually transformed.</li>
  <li><strong>Special-case files broke encryption targeting</strong>: utility documents like <code class="language-plaintext highlighter-rouge">DRAFTS.md</code> and <code class="language-plaintext highlighter-rouge">SUBDOMAIN-DRAFTS.md</code> (without standard front matter) were included in the encryption scope. Fixed by filtering to only files matching the <code class="language-plaintext highlighter-rouge">YYYY-MM-DD-*.md</code> pattern.</li>
  <li><strong>Convenience files broke archive sort order</strong>: The uppercase convenience files (<code class="language-plaintext highlighter-rouge">DRAFTS.md</code>, <code class="language-plaintext highlighter-rouge">SUBDOMAIN-DRAFTS.md</code>, <code class="language-plaintext highlighter-rouge">DRAFTS-TODO.md</code>) had no front matter, so Jekyll used the file’s filesystem mtime as their date. This caused them to appear at random positions in the archive depending on when they were last edited. Fixed by adding minimal front matter (<code class="language-plaintext highlighter-rouge">layout: none</code>, <code class="language-plaintext highlighter-rouge">date: 2038-01-18</code>, <code class="language-plaintext highlighter-rouge">sitemap: false</code>). The pinned date of 2038-01-18 (day before the Unix Y2K38 epoch overflow) sorts them to the very top of the archive. <code class="language-plaintext highlighter-rouge">layout: none</code> preserves raw markdown rendering without site chrome. <code class="language-plaintext highlighter-rouge">sitemap: false</code> keeps them out of the sitemap.</li>
</ol>

<h2 id="what-not-to-do">What NOT to Do</h2>

<p>A few guardrails I wish I’d written down before starting:</p>

<ul>
  <li><strong>Don’t add <code class="language-plaintext highlighter-rouge">--drafts --future</code> to the production build</strong> — defeats the entire purpose</li>
  <li><strong>Don’t use <code class="language-plaintext highlighter-rouge">published: false</code> as a gating mechanism</strong> — the <code class="language-plaintext highlighter-rouge">_drafts/</code> directory is the correct approach</li>
  <li><strong>Don’t rely solely on <code class="language-plaintext highlighter-rouge">robots.txt</code></strong> — it’s advisory, not enforced. Layer it with <code class="language-plaintext highlighter-rouge">noindex</code> meta tags</li>
  <li><strong>Don’t put passwords in the repo</strong> — use GitHub Secrets</li>
  <li><strong>Don’t modify <code class="language-plaintext highlighter-rouge">_config.yml</code> for the drafts site</strong> — use a config overlay file</li>
  <li><strong>Don’t encrypt the entire site</strong> — only draft and future posts need protection. Encrypting published content adds friction without adding value</li>
  <li><strong>Don’t use Staticrypt’s <code class="language-plaintext highlighter-rouge">-o</code> flag</strong> — it doesn’t exist in v3.5.4+. Use <code class="language-plaintext highlighter-rouge">-d &lt;directory&gt;</code> and handle the path flattening yourself</li>
</ul>

<h2 id="quick-reference-what-goes-where">Quick Reference: What Goes Where</h2>

<p>For anyone implementing this pattern, here’s where everything lives:</p>

<table>
  <thead>
    <tr>
      <th>File</th>
      <th>Repo</th>
      <th>Purpose</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">_config_drafts.yml</code></td>
      <td><code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code></td>
      <td>Jekyll config overlay for drafts build</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">.github/workflows/deploy-drafts.yml</code></td>
      <td><code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code></td>
      <td>GitHub Actions workflow</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">CNAME</code></td>
      <td><code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> (auto-created)</td>
      <td>GitHub Pages custom domain routing</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">.nojekyll</code></td>
      <td><code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> (auto-created)</td>
      <td>Prevents GitHub Pages re-processing</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">DRAFTS_PASSWORD</code> secret</td>
      <td><code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code></td>
      <td>Staticrypt shared password</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">DRAFTS_DEPLOY_TOKEN</code> secret</td>
      <td><code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code></td>
      <td>GitHub PAT for cross-repo push</td>
    </tr>
    <tr>
      <td>Draft preview banner</td>
      <td><code class="language-plaintext highlighter-rouge">_layouts/default.html</code></td>
      <td>Visual indicator for reviewers</td>
    </tr>
    <tr>
      <td>Giscus config</td>
      <td><code class="language-plaintext highlighter-rouge">_config_drafts.yml</code></td>
      <td>Points comments to drafts repo Discussions</td>
    </tr>
  </tbody>
</table>

<h2 id="lessons-learned">Lessons Learned</h2>

<ol>
  <li><strong>Static-site security controls are mostly UX signals</strong> unless source content is private. The password prompt says “this is a preview” — it doesn’t prevent a determined reader from viewing the public source on GitHub.</li>
  <li><strong>Build verification should check file transformation</strong>, not just string signatures. SHA-256 hash comparison catches real changes; string matching produces false negatives.</li>
  <li><strong>Tooling behavior under batch mode matters.</strong> Staticrypt’s output flattening was the key hidden trap — it works fine on a single file but silently breaks when processing multiple files with the same basename (like dozens of <code class="language-plaintext highlighter-rouge">index.html</code> files).</li>
  <li><strong>Selective encryption is better than full-site encryption.</strong> It reduces build time, avoids encrypting already-public content, and makes the password prompt meaningful rather than annoying.</li>
  <li><strong>A tiny UI affordance has outsized impact.</strong> The orange preview banner took five minutes to implement and is the single most useful feature for reviewer clarity.</li>
  <li><strong>Convenience files should get explicit front matter</strong> when they participate in Jekyll collections. Without a pinned date, Jekyll uses filesystem mtime, which causes random archive placement every time the file is edited.</li>
  <li><strong>The concurrency group matters.</strong> Without <code class="language-plaintext highlighter-rouge">cancel-in-progress: true</code>, rapid pushes can queue up multiple deployments that step on each other.</li>
</ol>

<hr />

<p><em>This is Part 3 of a three-part series on building a Jekyll draft preview site:</em></p>
<ul>
  <li><strong>Part 1</strong>: <a href="/jekyll-draft-preview-site-part-1/">Exploring every option I considered</a></li>
  <li><strong>Part 2</strong>: <a href="/jekyll-draft-preview-site-part-2/">Refining the design — config, workflow, feedback, and gaps</a></li>
  <li><strong>Part 3</strong> (this post): The complete implementation</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="jekyll" /><category term="github-pages" /><category term="devops" /><category term="jekyll" /><category term="github-pages" /><category term="staticrypt" /><category term="drafts" /><category term="preview" /><category term="ci-cd" /><category term="github-actions" /><category term="giscus" /><summary type="html"><![CDATA[Complete implementation guide for a Jekyll draft preview site using GitHub Pages, Staticrypt, and GitHub Actions. Includes repo setup, DNS configuration, final workflow, testing results, and lessons learned. Part 3 of a three-part series.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-3.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-3.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Why Data Science Follows the Research Model, Not the Software Development Model</title><link href="https://mcgarrah.org/research-model-for-data-science/" rel="alternate" type="text/html" title="Why Data Science Follows the Research Model, Not the Software Development Model" /><published>2026-06-23T00:00:00+00:00</published><updated>2026-06-23T00:00:00+00:00</updated><id>https://mcgarrah.org/research-model-for-data-science</id><content type="html" xml:base="https://mcgarrah.org/research-model-for-data-science/"><![CDATA[<p>In my previous article on <a href="/five-stages-cloud-data-science-platform/">Five Stages of a Cloud Data Science Platform</a>, I addressed the infrastructure question: how do you give data scientists production data access without compromising your security posture? That article solved the platform architecture problem. This one addresses the operational model problem that sits on top of it.</p>

<p>The core issue: most organizations manage data science teams using software development processes. Sprint planning, story points, predictable delivery timelines, definition of done. And it fails — not because the data scientists are bad at their jobs, but because the work itself follows a fundamentally different success model.</p>

<!-- excerpt-end -->

<h2 id="the-success-rate-problem">The Success Rate Problem</h2>

<p>Good software development succeeds more than 4 out of 5 times. You gather requirements, design a solution, build it, test it, ship it. The outcome is predictable. When a sprint fails, something went wrong — a missed requirement, a technical blocker, a scope change.</p>

<p>Good data science succeeds less than 1 out of 5 times. You form a hypothesis about what patterns exist in the data, design an experiment to test it, run the experiment, and discover that your hypothesis was wrong. This is not failure — this is the process working correctly. The 4 out of 5 “failures” are valuable because they eliminate hypotheses and narrow the search space.</p>

<pre><code class="language-mermaid">graph TD
    subgraph "Software Development Model"
        REQ[Requirements] --&gt; DESIGN[Design]
        DESIGN --&gt; BUILD[Build]
        BUILD --&gt; TEST[Test]
        TEST --&gt; SHIP[Ship ✓]
    end
    subgraph "Data Science / Research Model"
        HYP[Hypothesis] --&gt; EXP[Experiment]
        EXP --&gt; RESULT{Result?}
        RESULT --&gt;|Confirms| PUB[Publish / Deploy ✓]
        RESULT --&gt;|Refutes| DOC[Document Failure]
        DOC --&gt; HYP2[New Hypothesis]
        HYP2 --&gt; EXP
    end
    style SHIP fill:#4CAF50,color:#fff
    style PUB fill:#4CAF50,color:#fff
    style DOC fill:#FF9800,color:#fff
</code></pre>

<p>When you apply the software development model to data science, the 80% “failure” rate looks like a team performance problem. Managers ask why the team is not delivering. Stakeholders lose confidence. The team starts gaming metrics — reporting incremental progress on doomed approaches rather than honestly documenting failures and pivoting.</p>

<h2 id="academic-research-as-the-correct-model">Academic Research as the Correct Model</h2>

<p>Academic research has solved this problem for centuries. The model is:</p>

<ol>
  <li><strong>Form a hypothesis</strong> based on existing knowledge and available data</li>
  <li><strong>Design an experiment</strong> that can confirm or refute the hypothesis</li>
  <li><strong>Execute the experiment</strong> with rigorous methodology</li>
  <li><strong>Document the result</strong> — whether it confirms or refutes the hypothesis</li>
  <li><strong>Publish</strong> — both successes and failures contribute to the field’s knowledge</li>
</ol>

<p>The critical insight: <strong>documented failure is a first-class output.</strong> A paper that demonstrates “approach X does not work for problem Y under conditions Z” is publishable, citable, and valuable. It prevents the next researcher from wasting time on the same dead end.</p>

<p>Data science in an enterprise context should work the same way:</p>

<ul>
  <li>A model that does not improve on the baseline is not a failed sprint — it is a documented experiment that narrows the solution space</li>
  <li>Feature engineering that does not improve model performance is not wasted work — it is evidence about what the data does and does not contain</li>
  <li>A hypothesis about customer behavior that the data refutes is not a missed deadline — it is organizational learning</li>
</ul>

<h2 id="what-this-means-for-platform-engineering">What This Means for Platform Engineering</h2>

<p>The <a href="/five-stages-cloud-data-science-platform/">five-stage platform framework</a> I described previously provides the infrastructure. But the operational model determines how that infrastructure is used:</p>

<pre><code class="language-mermaid">graph LR
    subgraph "Prod Discovery (Interactive)"
        H[Hypothesis] --&gt; E[Experiment]
        E --&gt; F[Document Failure]
        F --&gt; H
        E --&gt; S[Success]
    end
    subgraph "Prod Integration (Automation)"
        S --&gt; V[Validate]
        V --&gt; P[Promote]
    end
    subgraph "Final Production"
        P --&gt; D[Deploy Model]
    end
    style F fill:#FF9800,color:#fff
    style S fill:#4CAF50,color:#fff
    style D fill:#4CAF50,color:#fff
</code></pre>

<p><strong>Prod Discovery</strong> is the research lab — where hypotheses are tested, experiments run, and failures documented. The interactive environment exists because research is iterative and exploratory. You cannot plan a sprint around “discover something useful in this dataset.”</p>

<p><strong>Prod Integration</strong> is where confirmed results get automated — the successful experiment becomes a reproducible pipeline. This is where the software development model applies: you have a known-good approach and you are engineering it for production reliability.</p>

<p><strong>Final Production</strong> is deployment — the model serves customers.</p>

<p>The key architectural insight: the research model operates in Discovery, and the software development model operates in Integration and Production. Trying to apply one model across all three stages is the root cause of most DS team dysfunction.</p>

<h2 id="documenting-failure-as-organizational-knowledge">Documenting Failure as Organizational Knowledge</h2>

<p>In academia, you publish your failures. In enterprise data science, you need the equivalent: a knowledge base of attempted approaches, their results, and the conditions under which they were tested.</p>

<p>This matters for three reasons:</p>

<ol>
  <li><strong>Preventing duplicate work</strong> — when a new data scientist joins the team, they should not spend three months rediscovering that approach X does not work for problem Y. The documentation should tell them immediately.</li>
  <li><strong>Revisiting failures when conditions change</strong> — an approach that failed with last year’s data volume may succeed with this year’s. An approach that failed before a new data source was available may succeed now. But only if the failure conditions are documented.</li>
  <li><strong>Justifying investment</strong> — when leadership asks “what has the DS team produced?”, the answer should include the search space that was eliminated, not just the models that shipped. Narrowing from 100 possible approaches to 5 viable ones is measurable progress.</li>
</ol>

<h2 id="the-management-implications">The Management Implications</h2>

<p>If you are leading a data science organization — or evaluating one — the operational model determines your success metrics:</p>

<table>
  <thead>
    <tr>
      <th>Metric</th>
      <th>SD Model (Wrong for DS)</th>
      <th>Research Model (Correct for DS)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Success rate</td>
      <td>“Why are we only shipping 20% of what we start?”</td>
      <td>“We eliminated 80% of the hypothesis space this quarter”</td>
    </tr>
    <tr>
      <td>Timeline</td>
      <td>“This model was supposed to ship in Sprint 4”</td>
      <td>“We have 3 documented experiments; the 4th shows promise”</td>
    </tr>
    <tr>
      <td>Team performance</td>
      <td>“The team is not delivering”</td>
      <td>“The team is systematically narrowing the solution space”</td>
    </tr>
    <tr>
      <td>Documentation</td>
      <td>“Update the Jira ticket”</td>
      <td>“Publish the experiment notebook with results and conditions”</td>
    </tr>
    <tr>
      <td>Failure</td>
      <td>“What went wrong?”</td>
      <td>“What did we learn?”</td>
    </tr>
  </tbody>
</table>

<p>The organizations that get this right — Google Brain, DeepMind, Meta FAIR — all operate on the research model internally. They publish papers about what did not work. They celebrate negative results that save future effort. They measure progress in knowledge gained, not just models shipped.</p>

<h2 id="connecting-the-pieces">Connecting the Pieces</h2>

<p>This article and the <a href="/five-stages-cloud-data-science-platform/">Five Stages platform framework</a> are two halves of the same argument:</p>

<ul>
  <li><strong>Five Stages</strong> answers: “How do you give data scientists the infrastructure they need?” (production data access with security controls)</li>
  <li><strong>Research Model</strong> answers: “How do you manage data scientists once they have that infrastructure?” (hypothesis-driven experimentation with documented failure as a first-class output)</li>
</ul>

<p>Together, they form the foundation for building an AI organization that can sustain long-term investment in ML — not just ship one model, but systematically build organizational capability in machine learning.</p>

<p>The platform without the operational model produces expensive infrastructure that frustrated data scientists underutilize. The operational model without the platform produces brilliant hypotheses that can never be tested against real data. You need both.</p>

<h2 id="implications-for-ai-leadership">Implications for AI Leadership</h2>

<p>If you are building or evaluating an AI organization:</p>

<ul>
  <li><strong>Staff for research, not just engineering.</strong> Data scientists with research backgrounds understand the failure model intuitively. Engineers retrained as data scientists often struggle with the ambiguity.</li>
  <li><strong>Budget for exploration, not just delivery.</strong> A DS team that must justify every experiment with a business case will only pursue safe, incremental work. The breakthrough insights come from exploratory work that might fail.</li>
  <li><strong>Measure knowledge, not just output.</strong> The documented experiment notebooks — including failures — are the team’s intellectual property. They represent the accumulated understanding of what works and what does not for your specific data and business.</li>
  <li><strong>Separate the research phase from the engineering phase.</strong> Discovery is research. Integration is engineering. Do not apply engineering management to research work, or research timelines to engineering work.</li>
</ul>

<p>The EMBA coursework I am completing has a useful framing for this: it is a portfolio management problem. You invest across a portfolio of hypotheses knowing that most will not pay off — but the ones that do will more than compensate for the failures. The same logic that makes venture capital work makes data science work. You just need the organizational patience to let the portfolio mature.</p>

<h2 id="the-numbers-behind-the-argument">The Numbers Behind the Argument</h2>

<p>The industry failure rate for AI and Machine Learning projects validates this thesis empirically. The 80–95% failure rate is not a technology problem — it is a methodology problem.</p>

<table>
  <thead>
    <tr>
      <th>Metric</th>
      <th>Finding</th>
      <th>Source</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Overall ROI failure</td>
      <td>95% of corporate generative AI pilots failed to deliver measurable P&amp;L impact</td>
      <td>MIT Sloan / NANDA (2025)</td>
    </tr>
    <tr>
      <td>Production deployment failure</td>
      <td>80–85% of enterprise AI initiatives never reach full production</td>
      <td>RAND Corporation, Gartner</td>
    </tr>
    <tr>
      <td>Pilot attrition</td>
      <td>~20% progress to pilot; fewer than 5% deploy with sustained value</td>
      <td>Industry composite</td>
    </tr>
    <tr>
      <td>Abandonment costs</td>
      <td>42% of U.S. companies abandoned at least one major AI initiative; average $7.2M sunk cost per project</td>
      <td>Enterprise tracking (2025)</td>
    </tr>
    <tr>
      <td>Infrastructure scaling</td>
      <td>64% of scaling failures attributed to infrastructure; production costs average 380% higher than pilot projections</td>
      <td>Industry post-mortems</td>
    </tr>
  </tbody>
</table>

<p>The RAND Corporation’s technical report identifies the core drivers: data architecture mismatch (models trained on curated data fail on messy production data), infrastructure scaling walls (cost and latency), and strategic misalignment (horizontal AI tools yielding low macro-level ROI). Critically, failure is rarely caused by a flaw in the foundational models — it stems from treating data science like traditional software development.</p>

<p>The classic Google paper “Hidden Technical Debt in Machine Learning Systems” (Sculley et al.) proves the point architecturally: actual ML model code constitutes only a small fraction of a production system. The rest is configuration, data collection, feature extraction, and verification infrastructure — all of which require the research model’s iterative approach rather than the SDLC’s linear delivery model.</p>

<h3 id="references">References</h3>

<ol>
  <li>RAND Corporation — <a href="https://www.rand.org/pubs/research_reports/RRA2680-1.html">Identifying and Mitigating the Risks of AI</a></li>
  <li>MIT Sloan / NANDA — <a href="https://fortune.com/2025/08/21/an-mit-report-that-95-of-ai-pilots-fail-spooked-investors-but-the-reason-why-those-pilots-failed-is-what-should-make-the-c-suite-anxious/">Why 95% of AI Pilots Fail</a></li>
  <li>RAND Corporation Technical Report — <a href="https://www.rand.org/content/dam/rand/pubs/research_reports/RRA2600/RRA2680-1/RAND_RRA2680-1.pdf">Full PDF</a></li>
  <li>Sculley et al. — “Hidden Technical Debt in Machine Learning Systems” (NeurIPS 2015)</li>
  <li>Gartner — Enterprise AI deployment failure rates (2024–2025 reports)</li>
  <li>Forbes — <a href="https://www.forbes.com/sites/garydrenik/2025/10/15/why-95-of-ai-projects-fail-and-how-better-data-can-change-that/">Why 95% of AI Projects Fail</a></li>
</ol>

<h2 id="where-i-have-seen-this-play-out">Where I Have Seen This Play Out</h2>

<p>The research-vs-SDLC conflict is not abstract to me. I have watched it manifest across every organization where data science and software engineering coexist:</p>

<ul>
  <li>
    <p><strong>USPS (2017–2019)</strong> — Data Engineer working directly with the Chief Data Scientist and his team on the Data Science Initiative (DSI) for all of USPS. Administered a 25-node SAS Viya in-memory analytics cluster (26TB RAM) connected to a 50+ node Hadoop data lake approaching 1PB, operating on a closed network under NIST 800-53 high security controls with DEA data hosted. Built custom data acquisition modules that gave the data science team access to production-quality geospatial and operational datasets they could not obtain from existing sources. The environment was the research model in practice — data scientists iterating on hypotheses about mail delivery optimization, package routing, and operational efficiency — while the production systems serving 160 million delivery points daily ran on the SDLC model. The two tracks coexisted because the platform architecture separated them.</p>
  </li>
  <li>
    <p><strong>Measurement Incorporated (2013–2015)</strong> — Where I first witnessed the conflict without a name for it. PhD researchers iterating on NLP models (research model) while the production scoring system served millions of assessments (SDLC model). The chaos of not separating these tracks is what motivated the five-stage framework.</p>
  </li>
  <li>
    <p><strong>BCBSNC (2019–2021)</strong> — Where the separation was designed in from day one. CarePath data scientists explored hypotheses in their EKS-based research environment while the production inference pipeline ran independently with automated promotion gates.</p>
  </li>
  <li>
    <p><strong>Envestnet (2021–present)</strong> — The mature implementation. SageMaker and Bedrock workloads operate in dedicated accounts with clear boundaries between exploration (Discovery) and production (automated pipelines via Airflow). The DataLake’s vEMR treats data promotion as a first-class engineering discipline.</p>
  </li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="technical" /><category term="ai" /><category term="machine-learning" /><category term="data-science" /><category term="research" /><category term="software-engineering" /><category term="platform-engineering" /><category term="leadership" /><summary type="html"><![CDATA[Why data science organizations fail when managed like software development teams, and how the academic research model — with its emphasis on documented failure, reproducibility, and iterative hypothesis testing — provides the correct operational framework for ML initiatives.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/research-model-for-data-science.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/research-model-for-data-science.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Building a Draft Preview Site for Jekyll — Part 2: Refining the Design</title><link href="https://mcgarrah.org/jekyll-draft-preview-site-part-2/" rel="alternate" type="text/html" title="Building a Draft Preview Site for Jekyll — Part 2: Refining the Design" /><published>2026-06-16T00:00:00+00:00</published><updated>2026-06-16T00:00:00+00:00</updated><id>https://mcgarrah.org/jekyll-draft-preview-site-part-2</id><content type="html" xml:base="https://mcgarrah.org/jekyll-draft-preview-site-part-2/"><![CDATA[<p>In <a href="/jekyll-draft-preview-site-part-1/">Part 1</a>, I explored seven options for creating a draft preview site and eliminated most of them. The survivor: a separate GitHub repo with Staticrypt encryption, automated via GitHub Actions, served at <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code>.</p>

<p>Now it’s time to design the actual implementation. This is where the interesting problems show up.</p>

<!-- excerpt-end -->

<p>This is Part 2 of a three-part series:</p>
<ul>
  <li><strong>Part 1</strong>: <a href="/jekyll-draft-preview-site-part-1/">Exploring every option I considered</a></li>
  <li><strong>Part 2</strong> (this post): Refining the design — config, workflow, feedback, and gaps</li>
  <li><strong>Part 3</strong>: <a href="/jekyll-draft-preview-site-part-3/">The complete implementation</a></li>
</ul>

<h2 id="the-architecture">The Architecture</h2>

<p>The production site and drafts site share the same source but build differently:</p>

<pre><code class="language-mermaid">flowchart TD
    A["mcgarrah.github.io repo\n(push to main)"] --&gt; B["jekyll.yml"]
    A --&gt; C["deploy-drafts.yml"]

    B --&gt; D["Build production"]
    D --&gt; E["mcgarrah.org"]

    C --&gt; F["Build with --drafts --future"]
    F --&gt; G["Apply _config_drafts.yml overlay"]
    G --&gt; H["Encrypt draft/future HTML\nwith Staticrypt"]
    H --&gt; I["Remove feeds, sitemaps\nReplace robots.txt"]
    I --&gt; J["Push _site/ to\ndrafts.mcgarrah.org repo"]
    J --&gt; K["drafts.mcgarrah.org\n(GitHub Pages)"]

    style E fill:#2d8659,color:#fff
    style K fill:#e67e00,color:#fff
</code></pre>

<p>One push to <code class="language-plaintext highlighter-rouge">main</code> triggers both workflows. The production site builds normally. The drafts site builds with everything visible, encrypts only draft and future post pages, and pushes to a separate repo.</p>

<h2 id="the-config-overlay">The Config Overlay</h2>

<p>Jekyll supports multiple config files merged left-to-right. A <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code> overlay in the main repo overrides production values without touching <code class="language-plaintext highlighter-rouge">_config.yml</code>:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://drafts.mcgarrah.org"</span>
<span class="na">canonical_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://drafts.mcgarrah.org"</span>
<span class="na">baseurl</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">draft_preview_site</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">main_site_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://mcgarrah.org"</span>

<span class="c1"># Disable production tracking and ads on the drafts preview site.</span>
<span class="na">google_analytics</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">google_adsense</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">google_cse_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>

<span class="c1"># Redirect Giscus comments to the drafts repo</span>
<span class="na">giscus</span><span class="pi">:</span>
  <span class="na">repo</span><span class="pi">:</span> <span class="s">mcgarrah/drafts.mcgarrah.org</span>
  <span class="na">repo_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;drafts-repo-id&gt;"</span>
  <span class="na">category</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Draft</span><span class="nv"> </span><span class="s">Reviews"</span>
  <span class="na">category_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;category-id&gt;"</span>
  <span class="na">mapping</span><span class="pi">:</span> <span class="s">pathname</span>
  <span class="na">strict</span><span class="pi">:</span> <span class="m">0</span>
  <span class="na">reactions_enabled</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">emit_metadata</span><span class="pi">:</span> <span class="m">0</span>
  <span class="na">input_position</span><span class="pi">:</span> <span class="s">top</span>
  <span class="na">theme</span><span class="pi">:</span> <span class="s">preferred_color_scheme</span>
  <span class="na">lang</span><span class="pi">:</span> <span class="s">en</span>
  <span class="na">loading</span><span class="pi">:</span> <span class="s">lazy</span>

<span class="c1"># Mark every rendered page as noindex on the drafts site.</span>
<span class="na">defaults</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">scope</span><span class="pi">:</span>
      <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
    <span class="na">values</span><span class="pi">:</span>
      <span class="na">noindex</span><span class="pi">:</span> <span class="kc">true</span>
</code></pre></div></div>

<p>A few things to note:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">draft_preview_site: true</code> and <code class="language-plaintext highlighter-rouge">main_site_url</code> power the preview banner (covered below)</li>
  <li>The <code class="language-plaintext highlighter-rouge">defaults:</code> block sets <code class="language-plaintext highlighter-rouge">noindex: true</code> on every page site-wide, which the layout picks up as <code class="language-plaintext highlighter-rouge">&lt;meta name="robots" content="noindex, follow"&gt;</code> — this is the primary defense against search engine indexing</li>
  <li><code class="language-plaintext highlighter-rouge">google_cse_id</code> is blanked to disable the custom search engine on the preview site</li>
</ul>

<p>Build command:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle <span class="nb">exec </span>jekyll build <span class="nt">--drafts</span> <span class="nt">--future</span> <span class="nt">--config</span> _config.yml,_config_drafts.yml
</code></pre></div></div>

<p>Jekyll merges configs left-to-right, so <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code> overrides the production values without touching <code class="language-plaintext highlighter-rouge">_config.yml</code>.</p>

<h2 id="the-feedback-problem">The Feedback Problem</h2>

<p>The whole point of a drafts site is getting feedback. But how?</p>

<h3 id="why-production-giscus-wont-work">Why Production Giscus Won’t Work</h3>

<p>The production site uses <a href="https://giscus.app/">Giscus</a> for comments, configured against <code class="language-plaintext highlighter-rouge">mcgarrah/mcgarrah.github.io</code> with <code class="language-plaintext highlighter-rouge">data-mapping="pathname"</code>. If I use the same config on the drafts site:</p>

<ul>
  <li>Comments would land in the <strong>production repo’s</strong> GitHub Discussions</li>
  <li>Draft feedback would mix with real comments on published posts</li>
  <li>When a draft is promoted to <code class="language-plaintext highlighter-rouge">_posts/</code>, the pathname changes — orphaning any feedback left on the draft URL</li>
</ul>

<h3 id="the-solution-giscus-on-the-drafts-repo">The Solution: Giscus on the Drafts Repo</h3>

<p>Enable GitHub Discussions on the <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> repo with a “Draft Reviews” category. Point Giscus at that repo via the config overlay. This gives:</p>

<ul>
  <li>Per-post threaded comments with the same familiar Giscus UI</li>
  <li>Feedback stays completely separate from production comments</li>
  <li>When a post graduates from <code class="language-plaintext highlighter-rouge">_drafts/</code> to <code class="language-plaintext highlighter-rouge">_posts/</code>, the draft comments stay behind — they served their purpose</li>
  <li>Reviewers need a GitHub account (fine for my audience)</li>
</ul>

<p>One concern worth testing: Staticrypt decrypts the page in-browser, and the Giscus <code class="language-plaintext highlighter-rouge">&lt;script&gt;</code> tag lives inside the encrypted HTML. It should load after decryption since the browser parses the decrypted DOM — but I’ll verify this in Part 3.</p>

<h2 id="staticrypt-the-details-that-matter">Staticrypt: The Details That Matter</h2>

<h3 id="navigation-between-pages">Navigation Between Pages</h3>

<p>This is the biggest UX concern. Staticrypt encrypts each HTML file independently. Click a link → new page → new password prompt. Every. Single. Click.</p>

<p>The fix: <code class="language-plaintext highlighter-rouge">--remember 30</code> stores the decryption key in <code class="language-plaintext highlighter-rouge">localStorage</code>. After entering the password once, subsequent pages decrypt automatically for 30 days. But if a reviewer’s browser blocks <code class="language-plaintext highlighter-rouge">localStorage</code> or they clear it, they’re back to typing the password on every page.</p>

<p>This needs thorough testing before sharing with reviewers.</p>

<h3 id="what-staticrypt-doesnt-encrypt">What Staticrypt Doesn’t Encrypt</h3>

<p>Only <code class="language-plaintext highlighter-rouge">.html</code> files get encrypted. Everything else is served in the clear:</p>

<ul>
  <li>Images in <code class="language-plaintext highlighter-rouge">/assets/images/</code></li>
  <li>PDFs in <code class="language-plaintext highlighter-rouge">/assets/pdfs/</code></li>
  <li>CSS, JavaScript, fonts</li>
  <li><code class="language-plaintext highlighter-rouge">feed.xml</code> — <strong>this leaks draft content as plain text</strong></li>
  <li><code class="language-plaintext highlighter-rouge">sitemap.xml</code> — leaks the URL structure</li>
</ul>

<p>The workflow must delete <code class="language-plaintext highlighter-rouge">feed.xml</code>, <code class="language-plaintext highlighter-rouge">sitemap.xml</code>, and <code class="language-plaintext highlighter-rouge">sitemapindex.xml</code> after the Jekyll build. Images being accessible by direct URL is low risk — nobody is guessing image paths for unpublished posts.</p>

<h3 id="is-staticrypt-even-necessary">Is Staticrypt Even Necessary?</h3>

<p>Given that the source markdown is public on GitHub, Staticrypt is a UX signal, not real security. It says “this is a private preview” and prevents casual browsing. A determined person could read the raw markdown on GitHub instead.</p>

<p>I’m keeping it because:</p>
<ul>
  <li>It’s one line in the build pipeline</li>
  <li>The password prompt sets clear expectations for reviewers</li>
  <li>It blocks web scrapers that ignore <code class="language-plaintext highlighter-rouge">robots.txt</code></li>
  <li>It can be removed anytime without changing anything else</li>
</ul>

<p>If it turns out to be more friction than value, I’ll drop it and fall back to the no-auth approach.</p>

<h2 id="the-visual-banner">The Visual Banner</h2>

<p>Reviewers need to know they’re on the preview site, not production. A Liquid conditional in the layout handles this:</p>

<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{% if site.draft_preview_site %}
<span class="nt">&lt;div</span> <span class="na">style=</span><span class="s">"background:#e67e00;color:#fff;text-align:center;padding:0.5em 1em;font-size:0.9em;font-weight:bold;"</span><span class="nt">&gt;</span>
  ⚠ DRAFT PREVIEW SITE — unpublished content, may change.
  <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"{{ site.main_site_url }}"</span> <span class="na">style=</span><span class="s">"color:#fff;text-decoration:underline;margin-left:0.5em;"</span><span class="nt">&gt;</span>Go to the main site →<span class="nt">&lt;/a&gt;</span>
<span class="nt">&lt;/div&gt;</span>
{% endif %}
</code></pre></div></div>

<p>This renders only when <code class="language-plaintext highlighter-rouge">draft_preview_site: true</code> is set in <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code>, so it’s invisible on production. The link back to the main site gives reviewers a quick way to compare draft content against what’s already published.</p>

<h2 id="dns-and-github-pages-routing">DNS and GitHub Pages Routing</h2>

<p>The subdomain needs a CNAME record in Porkbun:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drafts.mcgarrah.org  CNAME  mcgarrah.github.io.
</code></pre></div></div>

<p>Wait — <code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code> already points to the production site. How does GitHub know which repo to serve?</p>

<p>The DNS CNAME points to GitHub’s <em>apex domain</em> (<code class="language-plaintext highlighter-rouge">mcgarrah.github.io.</code>), not directly to the new repo. GitHub then uses the <code class="language-plaintext highlighter-rouge">CNAME</code> <em>file</em> inside each repo to route the request. This is how multiple repos under one account can each have their own custom domains.</p>

<p>GitHub Pages routes by <code class="language-plaintext highlighter-rouge">Host</code> header:</p>
<ol>
  <li>Browser requests <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code></li>
  <li>DNS resolves to GitHub’s IPs (via the CNAME pointing to <code class="language-plaintext highlighter-rouge">mcgarrah.github.io.</code>)</li>
  <li>GitHub receives the request with <code class="language-plaintext highlighter-rouge">Host: drafts.mcgarrah.org</code></li>
  <li>GitHub searches all repos under the account for a <code class="language-plaintext highlighter-rouge">CNAME</code> file containing <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code></li>
  <li>Finds <code class="language-plaintext highlighter-rouge">mcgarrah/drafts.mcgarrah.org</code> with matching <code class="language-plaintext highlighter-rouge">CNAME</code></li>
  <li>Serves that repo’s content</li>
</ol>

<p>The <code class="language-plaintext highlighter-rouge">CNAME</code> file is the routing key. The workflow creates it automatically during deployment.</p>

<p>The subdomain approach also gives us a separate cookie and <code class="language-plaintext highlighter-rouge">localStorage</code> scope from the production site. This matters for Staticrypt’s <code class="language-plaintext highlighter-rouge">--remember</code> feature — the decryption key stored in <code class="language-plaintext highlighter-rouge">localStorage</code> on <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> is invisible to JavaScript on <code class="language-plaintext highlighter-rouge">mcgarrah.org</code>, keeping the two sites cleanly isolated.</p>

<h2 id="the-workflow-sketch">The Workflow Sketch</h2>

<p>This lives in the main repo as <code class="language-plaintext highlighter-rouge">.github/workflows/deploy-drafts.yml</code>. Here’s the high-level structure — the real workflow evolved significantly during implementation (covered in Part 3), but this captures the design intent:</p>

<ol>
  <li><strong>Build</strong> — Jekyll with <code class="language-plaintext highlighter-rouge">--drafts --future</code> and the config overlay</li>
  <li><strong>Encrypt</strong> — Staticrypt on draft and future post pages only (not the full site)</li>
  <li><strong>Clean</strong> — Replace <code class="language-plaintext highlighter-rouge">robots.txt</code>, remove feeds/sitemaps, strip feed discovery links from HTML</li>
  <li><strong>Filter</strong> — Remove oversized binaries from deploy output</li>
  <li><strong>Deploy</strong> — Push to the drafts repo via PAT</li>
</ol>

<p>Two secrets needed:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">DRAFTS_PASSWORD</code> — the shared Staticrypt password</li>
  <li><code class="language-plaintext highlighter-rouge">DRAFTS_DEPLOY_TOKEN</code> — a GitHub PAT with <code class="language-plaintext highlighter-rouge">repo</code> scope for cross-repo push</li>
</ul>

<p>The drafts repo needs GitHub Pages configured to “Deploy from a branch” → <code class="language-plaintext highlighter-rouge">main</code> → <code class="language-plaintext highlighter-rouge">/ (root)</code>. A concurrency group ensures only one drafts deployment runs at a time.</p>

<p>One setup trap: an empty repo has no <code class="language-plaintext highlighter-rouge">main</code> branch yet, so GitHub Pages setup fails until the repo has an initial commit. Initialize the repo with a <code class="language-plaintext highlighter-rouge">README.md</code> or create any file before trying to configure Pages.</p>

<h2 id="remaining-gaps">Remaining Gaps</h2>

<h3 id="public-vs-private-drafts-repo">Public vs Private Drafts Repo</h3>

<p>The <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> repo is a deployment target — it contains only built HTML (encrypted if using Staticrypt). Should it be public or private?</p>

<ul>
  <li><strong>Public</strong>: Free GitHub Pages. The HTML is encrypted. The repo has no source code of value.</li>
  <li><strong>Private</strong>: Requires GitHub Pro ($4/mo) for private repo Pages.</li>
</ul>

<p>Leaning public. The source is already public in the main repo. Encrypting the rendered HTML in a private repo adds no real security.</p>

<h3 id="full-site-mirror">Full Site Mirror</h3>

<p>The build includes the <strong>entire site</strong> — all 139+ published posts plus drafts and future posts. Reviewers see the full context of where a draft fits in the archive. Building only draft posts would require a custom Jekyll plugin and isn’t worth the complexity.</p>

<h3 id="absolute-links-in-content">Absolute Links in Content</h3>

<p>Draft posts that hardcode <code class="language-plaintext highlighter-rouge">https://mcgarrah.org/some-post/</code> will link to production, not the drafts site. The <code class="language-plaintext highlighter-rouge">url</code> override in <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code> handles Liquid’s <code class="language-plaintext highlighter-rouge">https://mcgarrah.org</code> references, but hardcoded URLs in markdown won’t be rewritten. I should use relative links (<code class="language-plaintext highlighter-rouge">/some-post/</code>) in drafts — which is good practice anyway.</p>

<h3 id="resume-sub-site">Resume Sub-Site</h3>

<p>The resume lives in a separate repo and builds independently. Links to <code class="language-plaintext highlighter-rouge">/resume/</code> from the drafts site will 404 or redirect to production. That’s fine — reviewers don’t need the resume.</p>

<h3 id="build-frequency">Build Frequency</h3>

<p>Two Jekyll builds per push to <code class="language-plaintext highlighter-rouge">main</code>. Both run in a public repo, so GitHub Actions minutes are unlimited. Total build time for the drafts workflow (Jekyll + Staticrypt on ~140 posts) should be 2-4 minutes.</p>

<h2 id="open-questions">Open Questions</h2>

<p>A few things I still need to decide before implementation:</p>

<ol>
  <li><strong>Staticrypt yes or no?</strong> — Leaning yes for the UX signal, but willing to drop it if the navigation friction is too high.</li>
  <li><strong>Update frequency</strong> — Every push to <code class="language-plaintext highlighter-rouge">main</code>, or only on-demand via <code class="language-plaintext highlighter-rouge">workflow_dispatch</code>? Starting with every push seems right.</li>
  <li><strong>Feedback mechanism</strong> — Giscus on the drafts repo is the plan, but I need to test it with Staticrypt. Fallback: a mailto link in the draft banner.</li>
  <li><strong>Drafts repo visibility</strong> — Leaning public. No real reason to pay for private.</li>
</ol>

<h2 id="whats-next">What’s Next</h2>

<p>Part 3 will cover the actual implementation — creating the repo, configuring DNS, the final workflow with all the edge cases I hit, Staticrypt testing results, and what I learned from the whole process.</p>

<hr />

<p><em>This is Part 2 of a three-part series on building a Jekyll draft preview site:</em></p>
<ul>
  <li><strong>Part 1</strong>: <a href="/jekyll-draft-preview-site-part-1/">Exploring every option I considered</a></li>
  <li><strong>Part 2</strong> (this post): Refining the design — config, workflow, feedback, and gaps</li>
  <li><strong>Part 3</strong>: <a href="/jekyll-draft-preview-site-part-3/">The complete implementation</a></li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="jekyll" /><category term="github-pages" /><category term="devops" /><category term="jekyll" /><category term="github-pages" /><category term="staticrypt" /><category term="drafts" /><category term="preview" /><category term="ci-cd" /><category term="github-actions" /><category term="giscus" /><summary type="html"><![CDATA[Detailed design for a Jekyll draft preview site using GitHub Pages, Staticrypt, and GitHub Actions. Covers config overlays, Giscus feedback on a separate repo, Staticrypt UX considerations, and remaining open questions. Part 2 of a three-part series.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-2.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-2.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Migrating 16 Domains from SquareSpace to Porkbun — Part 2: The Sprint Finish</title><link href="https://mcgarrah.org/name-service-registrars-part-2/" rel="alternate" type="text/html" title="Migrating 16 Domains from SquareSpace to Porkbun — Part 2: The Sprint Finish" /><published>2026-06-10T00:00:00+00:00</published><updated>2026-06-10T00:00:00+00:00</updated><id>https://mcgarrah.org/name-service-registrars-part-2</id><content type="html" xml:base="https://mcgarrah.org/name-service-registrars-part-2/"><![CDATA[<p>Sometimes the best migration strategy is “renewals are coming, just move.” Several domains were approaching their SquareSpace renewal dates in June, and paying full-year renewal to a registrar I was actively leaving felt like burning money. So this week became a sprint: get everything transferred before the charges hit, deal with the rough edges after.</p>

<blockquote>
  <p>“Done is better than perfect — unless you’re migrating a domain with 25 DNS records serving production traffic.”</p>
</blockquote>

<p>This is Part 2. <a href="/name-service-registrars/">Part 1</a> covered the evaluation, batching strategy, and first nine transfers. This covers the final push — and the one domain I’m deliberately leaving behind.</p>

<!-- excerpt-end -->

<h2 id="the-urgency">The Urgency</h2>

<p>The original plan from Part 1 was methodical: batch by renewal date, spread costs, migrate carefully. That plan assumed I’d have steady free time across several months. Reality intervened — between the Executive MBA coursework at UNCW and a full workload at Envestnet, those careful weekends never materialized. Then June arrived, and suddenly mcgarware.com (June 9), phonemes.org (June 16), and darkmagic.org (August 16) were all staring me down.</p>

<p>Renewing at SquareSpace just to transfer later would mean paying twice for the same year. The math was simple: move them now or eat the cost.</p>

<h2 id="what-got-migrated-this-week">What Got Migrated This Week</h2>

<table>
  <thead>
    <tr>
      <th>Domain</th>
      <th>Transfer Date</th>
      <th>New Expiration</th>
      <th>Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>mcgarware.com</td>
      <td>Jun 2026</td>
      <td>Jun 9, 2027</td>
      <td>Legacy domain, dead server records cleaned up</td>
    </tr>
    <tr>
      <td>phonemes.org</td>
      <td>Jun 2026</td>
      <td>Jun 16, 2027</td>
      <td>Email forwarding configured</td>
    </tr>
    <tr>
      <td>darkmagic.org</td>
      <td>Jun 2026</td>
      <td>Aug 16, 2027</td>
      <td>Legacy domain, minimal records</td>
    </tr>
    <tr>
      <td>mcgarrah.dev</td>
      <td>Jun 2026</td>
      <td>Mar 4, 2028</td>
      <td>Mailgun email retained</td>
    </tr>
    <tr>
      <td>mcgarrah.us</td>
      <td>Jun 2026</td>
      <td>Sep 22, 2028</td>
      <td>URL forwarding reconfigured</td>
    </tr>
    <tr>
      <td>mcgarrah.app</td>
      <td>Jun 2026</td>
      <td>Mar 4, 2029</td>
      <td>Mailgun email retained</td>
    </tr>
  </tbody>
</table>

<p><strong>Also: phonemes.biz</strong> — let expire as planned. No transfer, no renewal. One less domain to manage.</p>

<p><strong>Updated totals: 15 of 16 domains at Porkbun (93.75% complete)</strong></p>

<h2 id="the-transfer-process--abbreviated">The Transfer Process — Abbreviated</h2>

<p>Having done this nine times already in Batch 1 and 2, the process was mechanical:</p>

<ol>
  <li>Export DNS records from SquareSpace WebUI (copy/paste — still no export button)</li>
  <li>Request transfer authorization code from SquareSpace</li>
  <li>Wait for auth code email (24-48 hours)</li>
  <li>Initiate transfer at Porkbun with auth code</li>
  <li>Contact Porkbun support to request accelerated transfer</li>
  <li>Respond to verification email</li>
  <li>Recreate DNS records at Porkbun</li>
  <li>Verify propagation</li>
</ol>

<p>The bottleneck remained the same: SquareSpace’s initial 24 hour auth code delivery and the 5-day hold unless you contact their support for an accelerated transfer process which cuts that hold to round trip email responses, but I was still at the mercy of email round-trips, catching a support person and dealing with multiple domains simultaneously. Hectic but with organization, it is manageable.</p>

<h3 id="credit-where-its-due-squarespace-support">Credit Where It’s Due: SquareSpace Support</h3>

<p>I want to be clear — SquareSpace’s support team made this migration painless. Their chat and email support were responsive and genuinely helpful in accelerating the transfers. Get someone on the chat, set up the email verification to confirm ownership, and the transfers move quickly. No obstruction, no retention tricks, no guilt trips about leaving.</p>

<p>My issue with SquareSpace is architectural (no API, no DNS export, no automation hooks), not operational. Their support team is competent and professional. I’d use them again — and would happily recommend them for a consulting client — anytime the walled garden isn’t a constraint. If your requirements are “reliable registrar with good support and you’ll manage DNS through the web UI,” SquareSpace is fine. It’s only when you need programmatic access that the model breaks down.</p>

<p>No animosity here. That’s more than I can say about certain other hosting providers that turned domain exits into adversarial processes.</p>

<h3 id="porkbuns-inbound-experience">Porkbun’s Inbound Experience</h3>

<p>Porkbun has a greased track for incoming transfers. The process is well-designed, the UI is clear, and everything just works. Auth code goes in, payment processes, domain appears in your account, DNS management is immediately available. No complaints.</p>

<h2 id="dead-record-cleanup">Dead Record Cleanup</h2>

<p>This batch was an opportunity to clean house. Several domains had legacy records pointing to 162.192.161.17 — a server that hasn’t been online in years. Rather than faithfully recreating dead records at Porkbun, I dropped them:</p>

<p><strong>Records NOT migrated (intentionally):</strong></p>
<ul>
  <li>Wildcard A records pointing to dead servers (mcgarware.com, darkmagic.org)</li>
  <li>Root A records pointing to the same dead IP</li>
  <li>All <code class="language-plaintext highlighter-rouge">_domainconnect</code> CNAME records (registrar-specific, irrelevant at Porkbun)</li>
  <li>SquareSpace forwarding records for mcgarrah.us (replaced with Porkbun URL forwarding)</li>
  <li>Custom mail server MX records for mcgarware.com and darkmagic.org (servers offline)</li>
</ul>

<p>Carrying dead DNS records forward is how you accumulate security debt. Wildcard A records pointing to an IP you don’t control are an open invitation for subdomain takeover attacks.</p>

<h2 id="email-forwarding-at-porkbun">Email Forwarding at Porkbun</h2>

<p>The biggest operational change: moving off Mailgun for the simple forwarding domains and onto Porkbun’s built-in email forwarding.</p>

<p><strong>The good:</strong> Porkbun includes email forwarding for free. Twenty aliases per domain, which covers my needs comfortably. Setting up <code class="language-plaintext highlighter-rouge">michael@phonemes.org → mcgarrah@gmail.com</code> and <code class="language-plaintext highlighter-rouge">mcgarrah@phonemes.org → mcgarrah@gmail.com</code> took about thirty seconds in their UI.</p>

<p><strong>The annoying:</strong> No wildcard forwarding. Mailgun’s catch-all meant any address <code class="language-plaintext highlighter-rouge">*@domain.com</code> landed in my inbox. Porkbun requires explicit aliases. For domains where I only ever receive mail at one or two addresses, this is fine — 20 aliases is generous. But for domains where I’ve historically handed out unique addresses to different services (the <code class="language-plaintext highlighter-rouge">servicename@domain.com</code> pattern for tracking who sells your email), I’ll need to enumerate those aliases manually.</p>

<p><strong>The pragmatic decision:</strong> For domains still running Mailgun (mcgarrah.dev, mcgarrah.us, mcgarrah.app), I’m keeping it in place for now while I evaluate whether those domains actually <em>need</em> wildcard forwarding. The honest answer is probably “no” — I’ll audit each one and likely end up replacing Mailgun with Porkbun aliases on most of them. Twenty aliases per domain is generous enough to cover the addresses that actually receive mail, and eliminating Mailgun removes an external dependency and simplifies the DNS configuration.</p>

<h2 id="the-one-that-stays-behind-mcgarrahorg">The One That Stays Behind: mcgarrah.org</h2>

<p>The primary domain isn’t moving yet. The <a href="/assets/data/domain-migration/dns-records-reference.txt">DNS records reference</a> shows why — 25+ records across:</p>

<ul>
  <li><strong>GitHub Pages</strong> — 4 A records, www CNAME, challenge TXT (this blog)</li>
  <li><strong>Mailgun email</strong> — MX, SPF, DKIM for root domain</li>
  <li><strong>DigitalOcean apps</strong> — nutrition, quiz, shiny-quiz subdomains</li>
  <li><strong>Nutrition subdomain email</strong> — separate MX, SPF, DKIM, DMARC records</li>
  <li><strong>Legacy and verification records</strong></li>
</ul>

<p>This isn’t a “transfer and see what breaks” domain. This is a “block out a Saturday, have a rollback plan, and test every service” domain. Any mistake takes this blog offline, breaks email delivery, and disconnects four hosted applications simultaneously.</p>

<p>The renewal isn’t until August 2, 2027. There’s no financial pressure. I’m holding this until the Executive MBA wraps up and I can dedicate a proper weekend to the migration without competing priorities. The checklist from Part 1 still applies — I just need the focus time to execute it carefully.</p>

<h2 id="current-state">Current State</h2>

<table>
  <thead>
    <tr>
      <th>Domain</th>
      <th>Registrar</th>
      <th>Email</th>
      <th>Status</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>mathomancer.com</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>mathomancy.com</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyzone.com</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyzone.org</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>cshensley.com</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyz.one</td>
      <td>Porkbun ✅</td>
      <td>—</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyzone.app</td>
      <td>Porkbun ✅</td>
      <td>—</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyzone.net</td>
      <td>Porkbun ✅</td>
      <td>—</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>brainyzone.us</td>
      <td>Porkbun ✅</td>
      <td>—</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>mcgarware.com</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>phonemes.org</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>darkmagic.org</td>
      <td>Porkbun ✅</td>
      <td>Porkbun forwarding</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>mcgarrah.dev</td>
      <td>Porkbun ✅</td>
      <td>Mailgun</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>mcgarrah.us</td>
      <td>Porkbun ✅</td>
      <td>Mailgun</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>mcgarrah.app</td>
      <td>Porkbun ✅</td>
      <td>Mailgun</td>
      <td>Complete</td>
    </tr>
    <tr>
      <td>phonemes.biz</td>
      <td>—</td>
      <td>—</td>
      <td>Expired (abandoned)</td>
    </tr>
    <tr>
      <td><strong>mcgarrah.org</strong></td>
      <td><strong>SquareSpace</strong></td>
      <td><strong>Mailgun</strong></td>
      <td><strong>Pending — migrate last</strong></td>
    </tr>
  </tbody>
</table>

<h2 id="lessons-from-the-rush">Lessons From the Rush</h2>

<p><strong>Batch transfers create email race conditions.</strong> When you’re moving six domains in parallel and each has its own auth code email and verification email, it’s easy to miss a step. I tracked each domain’s transfer state in a simple checklist — without it, I’d have lost track of which domains were waiting on auth codes versus waiting on verification responses.</p>

<p><strong>Dead records are easier to drop during migration than after.</strong> Once records exist at the new registrar, there’s psychological resistance to deleting them (“what if something needs it?”). During transfer is the natural audit point — if you can’t explain what a record does, don’t recreate it.</p>

<p><strong>Porkbun’s 20-alias limit is fine for personal domains.</strong> I was initially concerned about losing Mailgun’s catch-all capability. In practice, I receive email at 2-3 addresses per domain. Twenty aliases is more than generous. The Forward Email evaluation I was considering in a <a href="/email-forwarding-evaluation/">draft post</a> may not be necessary at all for most of these domains.</p>

<p><strong>The accelerated transfer process is reliable.</strong> Every time I contacted Porkbun support, the verification email arrived within hours and the transfer completed the same day. Their support team is responsive and the process is straightforward. SquareSpace’s chat support deserves equal credit — they facilitated every transfer without friction once I got someone on the line.</p>

<h2 id="whats-next">What’s Next</h2>

<ol>
  <li>
    <p><strong>mcgarrah.org migration</strong> — Blocked until I have a clear weekend. Target: after Executive MBA coursework concludes. Full rollback plan, service-by-service verification, pre-staged DNS records at Porkbun ready to activate.</p>
  </li>
  <li>
    <p><strong>Porkbun API automation</strong> — With 15 domains now at Porkbun, automating DNS updates via their API is the next step. This enables the cert-manager DNS-01 challenges and external-dns integration that motivated this entire migration.</p>
  </li>
  <li>
    <p><strong>Email consolidation</strong> — Evaluating Mailgun on each remaining domain to determine if wildcard forwarding is actually needed. Most likely outcome: replace Mailgun with Porkbun aliases on domains that only receive mail at a handful of known addresses. Keep Mailgun only where I genuinely need catch-all or outbound sending capability.</p>
  </li>
</ol>

<p>The migration that started as a careful multi-month project ended as a one-week sprint driven by renewal deadlines. Not how I planned it, but the result is the same: vendor lock-in eliminated, API access gained, and the path to DNS automation is clear. One domain remains, and it’ll get the careful migration it deserves — just not this week.</p>

<p><strong>Part 3</strong> will cover the final mcgarrah.org transfer and a deeper look at Porkbun’s included services that I glossed over in the rush — automatic SSL/TLS certificates, email aliases, DNSSEC, API access for DNS automation, and other features I didn’t fully explore while focused on getting domains moved.</p>

<hr />

<h2 id="references">References</h2>

<ul>
  <li><a href="/name-service-registrars/">Part 1: Migrating 16 Domains from SquareSpace to Porkbun</a> — Original migration article with full strategy and tooling</li>
  <li><a href="/assets/data/domain-migration/dns-records-reference.txt">Complete DNS records reference</a> — Full record exports for all domains</li>
  <li><a href="https://porkbun.com/products/email_forwarding">Porkbun Email Forwarding</a> — Free forwarding service details</li>
  <li><a href="https://porkbun.com/api/json/v3/documentation">Porkbun API Documentation</a> — REST API for DNS automation</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="technical" /><category term="infrastructure" /><category term="dns" /><category term="domains" /><category term="porkbun" /><category term="squarespace" /><category term="migration" /><category term="homelab" /><category term="automation" /><category term="email" /><summary type="html"><![CDATA[Part 2 of the SquareSpace to Porkbun domain migration. Completing the transfer of 15 out of 16 domains in a rush week before renewal deadlines, dealing with email forwarding without wildcard support, and why mcgarrah.org stays behind until the Executive MBA wraps up.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/name-service-registrars-part-2.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/name-service-registrars-part-2.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Building a Draft Preview Site for Jekyll — Part 1: Exploring the Options</title><link href="https://mcgarrah.org/jekyll-draft-preview-site-part-1/" rel="alternate" type="text/html" title="Building a Draft Preview Site for Jekyll — Part 1: Exploring the Options" /><published>2026-06-08T00:00:00+00:00</published><updated>2026-06-08T00:00:00+00:00</updated><id>https://mcgarrah.org/jekyll-draft-preview-site-part-1</id><content type="html" xml:base="https://mcgarrah.org/jekyll-draft-preview-site-part-1/"><![CDATA[<p>I write a lot of drafts. At any given time I have 40-50 posts in various stages of completion sitting in my <code class="language-plaintext highlighter-rouge">_drafts/</code> folder. Some are nearly done, some are research notes, and some are half-baked ideas that might never see the light of day.</p>

<p>The problem: I want trusted reviewers to see the <em>rendered</em> site with drafts included — not raw markdown files on GitHub. I want them to give feedback before I publish. And I want Google to stay far away from unfinished content. Three hard constraints — no authentication on GitHub Pages, a public source repository, and no search engine indexing of drafts — drove every architectural decision that follows.</p>

<!-- excerpt-end -->

<p>This is Part 1 of a three-part series:</p>
<ul>
  <li><strong>Part 1</strong> (this post): Exploring every option I considered</li>
  <li><strong>Part 2</strong>: <a href="/jekyll-draft-preview-site-part-2/">Refining the approach — what survived and why</a></li>
  <li><strong>Part 3</strong>: <a href="/jekyll-draft-preview-site-part-3/">The complete implementation</a></li>
</ul>

<h2 id="the-problem">The Problem</h2>

<p>Jekyll has great built-in support for drafts (<code class="language-plaintext highlighter-rouge">_drafts/</code> folder) and future-dated posts. Running <code class="language-plaintext highlighter-rouge">bundle exec jekyll serve --drafts --future</code> locally shows everything. But “locally” doesn’t help when I want someone else to review my work.</p>

<p>What I need:</p>
<ol>
  <li>A separate site that builds with <code class="language-plaintext highlighter-rouge">--drafts --future</code></li>
  <li>Not indexed by Google</li>
  <li>Some form of access control — even minimal</li>
  <li>Automated — I don’t want to manually deploy every time I push</li>
  <li>No impact on the production site at <code class="language-plaintext highlighter-rouge">mcgarrah.org</code></li>
</ol>

<p>What I have to work with:</p>
<ul>
  <li>GitHub Pages (free, already hosting the production site)</li>
  <li>Porkbun (DNS registrar, already managing <code class="language-plaintext highlighter-rouge">mcgarrah.org</code>)</li>
  <li>GitHub Actions (already running CI/CD for the production build)</li>
  <li>A public repository (meaning the raw markdown is already visible to anyone)</li>
</ul>

<p>That last point matters more than you’d think.</p>

<h2 id="the-security-reality-check">The Security Reality Check</h2>

<p>Before diving into options, I had to be honest about what “access control” means here. My <code class="language-plaintext highlighter-rouge">mcgarrah.github.io</code> repository is <strong>public</strong>. Anyone can browse to <code class="language-plaintext highlighter-rouge">_drafts/</code> on GitHub right now and read every unpublished post in raw markdown.</p>

<p>So what am I actually protecting? Not the content — that ship has sailed. What I want is:</p>
<ul>
  <li><strong>A signal to reviewers</strong>: “This isn’t ready for public consumption”</li>
  <li><strong>A barrier to casual discovery</strong>: Someone Googling a topic shouldn’t land on my half-written draft</li>
  <li><strong>No indexing</strong>: Search engines must not crawl the preview site</li>
</ul>

<p>This reframes the entire problem. I don’t need enterprise authentication. I need a speed bump and a <code class="language-plaintext highlighter-rouge">noindex</code> tag.</p>

<h2 id="option-1-netlify-with-password-protection">Option 1: Netlify with Password Protection</h2>

<p>Netlify’s free tier includes site-wide password protection. The workflow would be:</p>
<ul>
  <li>GitHub Actions builds Jekyll with <code class="language-plaintext highlighter-rouge">--drafts --future</code></li>
  <li>Deploys to Netlify via CLI</li>
  <li>Netlify serves the site behind a shared password</li>
  <li>CNAME <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> points to Netlify</li>
</ul>

<p><strong>Pros:</strong> Dead simple. Real server-side password. Built-in <code class="language-plaintext highlighter-rouge">X-Robots-Tag: noindex</code>. Always-on.</p>

<p><strong>Why I eliminated it:</strong> It’s a new external service. I’d need a Netlify account, API tokens, and another platform to manage. My constraint was to stay within GitHub + Porkbun — the tools I already use. Netlify is a fine service, but adding it for a draft preview site felt like overkill.</p>

<h2 id="option-2-cloudflare-access">Option 2: Cloudflare Access</h2>

<p>Cloudflare Zero Trust (free tier, up to 50 users) can gate a subdomain behind email-based one-time-pin authentication. Real per-user auth, audit logs, the works.</p>

<p><strong>Pros:</strong> Proper authentication. Per-user access control. Free tier is generous.</p>

<p><strong>Why I eliminated it:</strong> Requires Cloudflare. I use Porkbun for DNS and don’t want to proxy traffic through Cloudflare or move DNS management. Adding Cloudflare for one subdomain introduces a dependency I’d rather avoid. Same “new external service” problem as Netlify.</p>

<h2 id="option-3-self-hosted-on-proxmox">Option 3: Self-Hosted on Proxmox</h2>

<p>I run a six-node Proxmox cluster with Ceph storage. I could serve the drafts site from Caddy with HTTP Basic Auth, or eventually from Kubernetes with ingress-nginx.</p>

<p><strong>Pros:</strong> Full control. Real authentication. Uses existing infrastructure.</p>

<p><strong>Why I eliminated it:</strong> My homelab isn’t ready for external-facing services yet. The Kubernetes cluster is still in the infrastructure build-out phase. Even if it were ready, tying reviewer access to my home internet uptime is a bad idea. And honestly, running a web server for a blog preview site is like using a sledgehammer to hang a picture frame.</p>

<p>I’ll revisit this when K8s is production-ready, but that’s months away.</p>

<h2 id="option-4-separate-branch-in-the-same-repo">Option 4: Separate Branch in the Same Repo</h2>

<p>Create a <code class="language-plaintext highlighter-rouge">drafts</code> branch, build with <code class="language-plaintext highlighter-rouge">--drafts --future</code>, deploy to a different GitHub Pages site.</p>

<p><strong>Why it doesn’t work:</strong> GitHub Pages deploys one branch to one domain per repo. You can’t have <code class="language-plaintext highlighter-rouge">main</code> → <code class="language-plaintext highlighter-rouge">mcgarrah.org</code> and <code class="language-plaintext highlighter-rouge">drafts</code> → <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> from the same repository. A separate branch helps with content management but doesn’t solve the hosting problem.</p>

<h2 id="option-5-project-page-at-mcgarrahorgdrafts">Option 5: Project Page at <code class="language-plaintext highlighter-rouge">mcgarrah.org/drafts/</code></h2>

<p>GitHub Pages project sites serve at <code class="language-plaintext highlighter-rouge">&lt;username&gt;.github.io/&lt;repo-name&gt;/</code>. My resume already works this way — the <code class="language-plaintext highlighter-rouge">resume</code> repo serves at <code class="language-plaintext highlighter-rouge">mcgarrah.org/resume/</code>. A repo named <code class="language-plaintext highlighter-rouge">drafts</code> would serve at <code class="language-plaintext highlighter-rouge">mcgarrah.org/drafts/</code>.</p>

<p><strong>Pros:</strong> No DNS changes. Familiar pattern. Works immediately.</p>

<p><strong>Why I deprioritized it:</strong> The <code class="language-plaintext highlighter-rouge">robots.txt</code> problem. The file lives at the domain root (<code class="language-plaintext highlighter-rouge">mcgarrah.org/robots.txt</code>) and is served by the main site. To block crawlers from <code class="language-plaintext highlighter-rouge">/drafts/</code>, I’d need to add <code class="language-plaintext highlighter-rouge">Disallow: /drafts/</code> to the <em>production</em> site’s <code class="language-plaintext highlighter-rouge">robots.txt</code> — which violates my constraint of not modifying the production site. The <code class="language-plaintext highlighter-rouge">noindex</code> meta tag on each page still works, but <code class="language-plaintext highlighter-rouge">robots.txt</code> is the first line of defense and I’d be giving that up.</p>

<p>Also, the path is guessable. Anyone who knows the main site exists might try <code class="language-plaintext highlighter-rouge">/drafts/</code> out of curiosity.</p>

<h2 id="option-6-staticrypt-on-github-pages">Option 6: Staticrypt on GitHub Pages</h2>

<p><a href="https://github.com/robinmoisson/staticrypt">Staticrypt</a> encrypts HTML files with AES-256-GCM using a password-derived key. Each page is replaced with a password prompt that decrypts the content in-browser. No server-side auth needed.</p>

<p><strong>Pros:</strong> Works on any static host including GitHub Pages. No external services. Password prompt signals “private preview.”</p>

<p><strong>Cons:</strong> Security through obscurity — but we already established that the source is public. The password is a UX signal, not a lock. Each page is encrypted independently, so navigation requires <code class="language-plaintext highlighter-rouge">--remember</code> to avoid re-prompting on every click.</p>

<p><strong>This one survived.</strong> Combined with a separate GitHub repo and automated deployment, it checks every box.</p>

<h2 id="option-7-separate-github-repo-without-any-auth">Option 7: Separate GitHub Repo WITHOUT Any Auth</h2>

<p>Same as Option 6 but skip Staticrypt entirely. The drafts site is publicly accessible but:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">robots.txt</code> blocks crawlers</li>
  <li><code class="language-plaintext highlighter-rouge">&lt;meta name="robots" content="noindex, nofollow"&gt;</code> on every page</li>
  <li>No links from the production site</li>
  <li>No Google Analytics, no sitemap, no RSS feed</li>
</ul>

<p><strong>This also survived.</strong> Given that the source markdown is already public, this is a defensible starting point. The password from Staticrypt is a nice-to-have, not a must-have.</p>

<h2 id="what-about-draftsgithubio">What About <code class="language-plaintext highlighter-rouge">drafts.github.io</code>?</h2>

<p>I briefly wondered if I could use <code class="language-plaintext highlighter-rouge">drafts.github.io</code> as the domain. Short answer: no. GitHub Pages <code class="language-plaintext highlighter-rouge">*.github.io</code> domains are tied to GitHub usernames. <code class="language-plaintext highlighter-rouge">drafts.github.io</code> would belong to whoever owns the <code class="language-plaintext highlighter-rouge">drafts</code> GitHub account. You get exactly one: <code class="language-plaintext highlighter-rouge">&lt;username&gt;.github.io</code>.</p>

<h2 id="where-i-landed">Where I Landed</h2>

<p>Two options survived the elimination process:</p>

<ol>
  <li><strong>Separate GitHub repo + Staticrypt</strong> — password-protected, fully automated, zero external services</li>
  <li><strong>Separate GitHub repo, no auth</strong> — even simpler, relies on <code class="language-plaintext highlighter-rouge">noindex</code> and obscurity</li>
</ol>

<p>Both use:</p>
<ul>
  <li>A new repo (<code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code>) with GitHub Pages enabled</li>
  <li>GitHub Actions in the main repo to build and push</li>
  <li>A <code class="language-plaintext highlighter-rouge">_config_drafts.yml</code> overlay to change the URL, disable analytics, and redirect Giscus comments</li>
  <li>DNS CNAME in Porkbun pointing <code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code> to GitHub Pages</li>
</ul>

<p>The subdomain approach (<code class="language-plaintext highlighter-rouge">drafts.mcgarrah.org</code>) won over the project page (<code class="language-plaintext highlighter-rouge">mcgarrah.org/drafts/</code>) because it gets its own <code class="language-plaintext highlighter-rouge">robots.txt</code>, its own cookie scope, and doesn’t require modifying the production site.</p>

<p>In <a href="/jekyll-draft-preview-site-part-2/">Part 2</a>, I’ll walk through the refined design — the config overlay, the GitHub Actions workflow, the Giscus feedback setup, and the gaps I found when I started thinking through the implementation details.</p>

<hr />

<p><em>This is Part 1 of a three-part series on building a Jekyll draft preview site:</em></p>
<ul>
  <li><strong>Part 1</strong> (this post): Exploring every option I considered</li>
  <li><strong>Part 2</strong>: <a href="/jekyll-draft-preview-site-part-2/">Refining the approach — what survived and why</a></li>
  <li><strong>Part 3</strong>: <a href="/jekyll-draft-preview-site-part-3/">The complete implementation</a></li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="jekyll" /><category term="github-pages" /><category term="devops" /><category term="jekyll" /><category term="github-pages" /><category term="staticrypt" /><category term="drafts" /><category term="preview" /><category term="ci-cd" /><category term="github-actions" /><summary type="html"><![CDATA[Exploration of options for creating a password-protected Jekyll draft preview site on GitHub Pages, including Staticrypt, Netlify, Cloudflare Access, self-hosting, and project pages. Part 1 of a three-part series.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-1.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/jekyll-draft-preview-site-part-1.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">ZFS Boot Mirrors on Proxmox 8 for the Homelab - Part 3</title><link href="https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-3/" rel="alternate" type="text/html" title="ZFS Boot Mirrors on Proxmox 8 for the Homelab - Part 3" /><published>2026-06-03T00:00:00+00:00</published><updated>2026-06-03T00:00:00+00:00</updated><id>https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-3</id><content type="html" xml:base="https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-3/"><![CDATA[<p><a href="/proxmox-zfs-boot-mirrors-part-1/">Part 1</a> covered replacing a failed ZFS boot mirror drive with one of the same size. <a href="/proxmox-zfs-boot-mirrors-part-2/">Part 2</a> covered the emergency recovery when both drives fail simultaneously. This is the planned version of that same fresh-install procedure — applied deliberately when your replacement drives are <em>smaller</em> than the originals, with a UEFI upgrade included.</p>

<p>The backup checklist and recovery steps here were refined across the Harlan emergency (Part 2), the Quell sequential drive swap, and the Edgar planned migration. Any planned infrastructure migration should have a documented rollback path before you start — the Golden Backup checklist below is that path. This is the procedure I’ll follow for the remaining cluster nodes as their spinning rust ages out.</p>

<p>In my case, the cluster nodes have 500GB or 1TB spinning rust HDDs as boot mirrors but only use 3-7GB of actual space — Ceph handles all the real storage. Replacing them with 128GB SSDs makes sense on cost, speed, and reliability grounds. But ZFS won’t let you add a smaller drive to an existing mirror:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">root@tanaka:~#</span><span class="w"> </span>zpool attach rpool ata-existing-part3 /dev/disk/by-id/ata-smaller-ssd-part3
<span class="go">cannot attach /dev/disk/by-id/ata-smaller-ssd-part3 to rpool: device is too small
</span></code></pre></div></div>

<p><a href="/assets/images/zfs-boot-mirror-proxmox8-001.png" target="_blank"><img src="/assets/images/zfs-boot-mirror-proxmox8-001.png" alt="Proxmox 8 ZFS Boot Mirror" width="40%" height="40%" style="display:block; margin-left:auto; margin-right:auto" /></a></p>

<!-- excerpt-end -->

<h2 id="why-fresh-install-instead-of-zfs-sendreceive">Why Fresh Install Instead of ZFS Send/Receive</h2>

<p>My original plan was to use <code class="language-plaintext highlighter-rouge">zfs send | zfs receive</code> to migrate data from the larger pool to a new smaller pool without reinstalling. There are documented approaches for this — a <a href="https://www.reddit.com/r/Proxmox/comments/1cr6wn7/tutorial_howto_migrate_a_pve_zfs_bootroot_mirror/">Reddit tutorial</a>, a <a href="https://github.com/kneutron/ansitest/blob/master/proxmox/proxmox-replace-zfs-mirror-boot-disks-with-smaller.sh">shell script</a>, and Aaron Lauterer’s <a href="https://aaronlauterer.com/blog/2021/proxmox-ve-migrate-to-smaller-root-disks/">migration guide</a>. I spent time researching and partially testing this path on tanaka before abandoning it.</p>

<p>The problem is that send/receive only solves one thing — the disk size mismatch. It leaves you on the same Legacy BIOS with GRUB, the same aging OS install, and the same accumulated configuration drift. After working through the Harlan and Quell recoveries documented in <a href="/proxmox-zfs-boot-mirrors-part-2/">Part 2</a>, I realized the fresh install approach solves multiple problems at once:</p>

<ul>
  <li><strong>UEFI upgrade.</strong> Legacy BIOS and GRUB are technical debt. Proxmox already prefers <code class="language-plaintext highlighter-rouge">systemd-boot</code> on UEFI installs, and PVE 9.x is coming. Every node left on Legacy BIOS is a node that will need extra attention during the major version upgrade. A fresh install is the clean path to UEFI — you can’t switch boot modes with a send/receive migration.</li>
  <li><strong>Clean OS state.</strong> Years of package upgrades, configuration changes, and accumulated cruft disappear. The fresh install starts from a known-good baseline.</li>
  <li><strong>The send/receive procedure is fragile.</strong> It requires creating a temporary pool, migrating snapshots, swapping pool names, and updating boot UUIDs — all on a live system. One mistake and you’re doing a fresh install anyway.</li>
  <li><strong>You’re already touching the hardware.</strong> If you’re pulling drives and swapping SSDs, the incremental effort of a fresh install is small.</li>
  <li><strong>The Ceph data is safe regardless.</strong> OSD drives are completely independent of the boot pool. A fresh install doesn’t touch them.</li>
</ul>

<p>The key insight from the Harlan recovery was that the <em>hard part</em> isn’t the install — it’s preserving the node’s identity so it can rejoin the cluster and reclaim its OSDs. Once you have a solid backup checklist, the fresh install path is actually less risky than the in-place migration.</p>

<h3 id="grub-is-the-real-problem">GRUB Is the Real Problem</h3>

<p>In a Proxmox ZFS-on-root setup, GRUB and Legacy BIOS are a constant source of friction. During the Quell recovery, <code class="language-plaintext highlighter-rouge">grub-install</code> failed with an “unknown filesystem” error because it doesn’t understand ZFS partitions directly — we had to work around it with <code class="language-plaintext highlighter-rouge">proxmox-boot-tool</code> every time. On UEFI with <code class="language-plaintext highlighter-rouge">systemd-boot</code>, this entire class of problem disappears. The boot partition is a simple FAT32 ESP that every tool understands.</p>

<p>With PVE 9.x on the horizon, staying on Legacy BIOS means risking boot issues during the major version upgrade. Converting one node at a time during planned SSD migrations is far less stressful than dealing with it during a cluster-wide upgrade.</p>

<h2 id="which-path-is-right-for-you">Which Path Is Right for You?</h2>

<table>
  <thead>
    <tr>
      <th>Situation</th>
      <th>Approach</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>One drive failed, replacement is same size</td>
      <td><a href="/proxmox-zfs-boot-mirrors-part-1/">Part 1</a> (zpool replace)</td>
    </tr>
    <tr>
      <td>Both drives failed simultaneously</td>
      <td><a href="/proxmox-zfs-boot-mirrors-part-2/">Part 2</a> (emergency fresh install)</td>
    </tr>
    <tr>
      <td>Planned migration to smaller drives</td>
      <td>This article (planned fresh install with UEFI upgrade)</td>
    </tr>
    <tr>
      <td>Node has no Ceph OSDs</td>
      <td>Simpler — skip the OSD reactivation steps</td>
    </tr>
  </tbody>
</table>

<p>The difference between this article and Part 2 is <em>timing</em>. Part 2 is an emergency recovery when the node is already dead. This article is a planned migration where you control the schedule and can prepare thoroughly.</p>

<h2 id="the-golden-backup-checklist">The Golden Backup Checklist</h2>

<p>This checklist was refined across the Harlan emergency recovery and the Quell and Edgar planned migrations. Every item earned its place by causing problems when it was missing.</p>

<p>Run these on the node <em>before</em> you power it down:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create backup directory on shared CephFS storage</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/pve/cephfs/backups/edgar/

<span class="c"># Capture the binary hostid — critical for ZFS pool reimport</span>
<span class="nb">hostid</span> <span class="o">&gt;</span> /mnt/pve/cephfs/backups/edgar/hostid.txt

<span class="c"># Bundle the critical system identity and mapping files</span>
<span class="nb">tar</span> <span class="nt">-cvzf</span> /mnt/pve/cephfs/backups/edgar/edgar_migration_bundle.tar.gz <span class="se">\</span>
  /etc/network/interfaces <span class="se">\</span>
  /etc/hosts <span class="se">\</span>
  /etc/hostname <span class="se">\</span>
  /etc/subuid <span class="se">\</span>
  /etc/subgid <span class="se">\</span>
  /etc/kernel/cmdline <span class="se">\</span>
  /etc/modprobe.d/ <span class="se">\</span>
  /etc/modules <span class="se">\</span>
  /etc/default/grub <span class="se">\</span>
  /etc/pve/user.cfg

<span class="c"># Capture OSD-to-disk mapping</span>
ceph-volume lvm list <span class="o">&gt;</span> /mnt/pve/cephfs/backups/edgar/ceph_lvm_layout.txt
lvs <span class="nt">-a</span> <span class="nt">-o</span> +devices <span class="o">&gt;</span> /mnt/pve/cephfs/backups/edgar/lvm_devices.txt

<span class="c"># List manually installed packages for post-install restoration</span>
apt-mark showmanual <span class="o">&gt;</span> /mnt/pve/cephfs/backups/edgar/apt-mark-showmanual.txt
</code></pre></div></div>

<h3 id="why-each-file-matters">Why Each File Matters</h3>

<table>
  <thead>
    <tr>
      <th>File</th>
      <th>Why it matters</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">hostid</code></td>
      <td>ZFS records the hostid that imported the pool. Mismatch causes warnings and can prevent clean import</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">/etc/subuid</code> / <code class="language-plaintext highlighter-rouge">/etc/subgid</code></td>
      <td>UID/GID mappings for unprivileged LXC containers — missing this breaks container startup with permission errors</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">/etc/default/grub</code></td>
      <td>Captures kernel boot parameters (USB quirks, etc.) that need to be translated to <code class="language-plaintext highlighter-rouge">/etc/kernel/cmdline</code> for UEFI</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ceph_lvm_layout.txt</code></td>
      <td>Maps OSD IDs to physical devices — your recovery map if <code class="language-plaintext highlighter-rouge">ceph-volume lvm activate --all</code> fails</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">/etc/pve/user.cfg</code></td>
      <td>Cluster-wide but good to have a local copy for reference</td>
    </tr>
  </tbody>
</table>

<p>The <code class="language-plaintext highlighter-rouge">/etc/subuid</code> and <code class="language-plaintext highlighter-rouge">/etc/subgid</code> files were the lesson learned from the Harlan recovery — the Jellyfin LXC container wouldn’t start after the rebuild because the UID mappings were missing. See <a href="/proxmox-zfs-boot-mirrors-part-2/">Part 2</a> for the full story.</p>

<h2 id="the-migration-procedure">The Migration Procedure</h2>

<h3 id="phase-1-pause-the-ceph-cluster">Phase 1: Pause the Ceph Cluster</h3>

<p>Run from any healthy cluster node:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd <span class="nb">set </span>noout
ceph osd <span class="nb">set </span>nobackfill
</code></pre></div></div>

<p>This prevents the cluster from rebalancing data while the node is offline.</p>

<h3 id="phase-2-hardware-and-bios">Phase 2: Hardware and BIOS</h3>

<ol>
  <li><strong>Power down</strong> the node</li>
  <li><strong>Swap drives</strong> — pull the old HDDs, install the new SSDs</li>
  <li><strong>Verify OSD cables</strong> — since the case is open, confirm SATA and power cables for the OSD drives and WAL SSD are secure</li>
  <li><strong>Enter BIOS setup:</strong>
    <ul>
      <li>Set Boot Mode to <strong>UEFI Only</strong> (disable CSM/Legacy Boot)</li>
      <li>Ensure <strong>Secure Boot is OFF</strong> (avoids complications with Proxmox/ZFS)</li>
      <li>Enable <strong>SATA AHCI Mode</strong> if not already set</li>
      <li>Set the new SSDs as primary boot priority</li>
    </ul>
  </li>
</ol>

<h3 id="phase-3-fresh-proxmox-install">Phase 3: Fresh Proxmox Install</h3>

<p>Boot from the Proxmox installer USB:</p>

<ol>
  <li><strong>Target disks</strong>: Select the two new SSDs. Choose <strong>ZFS (RAID1)</strong>.
    <ul>
      <li><strong>WARNING</strong>: Do NOT select the OSD drives or the WAL SSD. The installer will show all connected drives.</li>
    </ul>
  </li>
  <li><strong>Network</strong>: Use the exact same <strong>hostname</strong> and <strong>IP address</strong> as the original node.</li>
  <li><strong>Boot mode</strong>: Because you switched to UEFI in the BIOS, the installer will automatically create the ESP (EFI System Partition) and set up <code class="language-plaintext highlighter-rouge">systemd-boot</code> instead of GRUB.</li>
</ol>

<h3 id="phase-4-post-install-identity-restoration">Phase 4: Post-Install Identity Restoration</h3>

<p>Once the fresh install is up and you can SSH in:</p>

<p><strong>Restore the HostID:</strong></p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Use the hex value from your backup</span>
zgenhostid <span class="si">$(</span><span class="nb">cat</span> /mnt/pve/cephfs/backups/edgar/hostid.txt<span class="si">)</span>
</code></pre></div></div>

<p><strong>Restore LXC UID/GID mappings:</strong></p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">tar</span> <span class="nt">-xvzf</span> /mnt/pve/cephfs/backups/edgar/edgar_migration_bundle.tar.gz <span class="se">\</span>
  <span class="nt">-C</span> / etc/subuid etc/subgid
</code></pre></div></div>

<p><strong>Translate boot quirks from GRUB to systemd-boot:</strong></p>

<p>If your old <code class="language-plaintext highlighter-rouge">/etc/default/grub</code> had kernel parameters (like USB storage quirks for OSD drives), they need to move to the UEFI equivalent:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Check the old grub config from your backup</span>
<span class="nb">tar</span> <span class="nt">-xvzf</span> /mnt/pve/cephfs/backups/edgar/edgar_migration_bundle.tar.gz <span class="se">\</span>
  <span class="nt">-C</span> /tmp etc/default/grub
<span class="nb">grep </span>GRUB_CMDLINE /tmp/etc/default/grub

<span class="c"># If you had usb-storage.quirks or other parameters, add them to:</span>
nano /etc/kernel/cmdline
<span class="c"># Append the parameters to the existing line</span>

<span class="c"># Apply the change</span>
proxmox-boot-tool refresh
</code></pre></div></div>

<p><strong>Run the Proxmox post-install script</strong> (optional but recommended):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash <span class="nt">-c</span> <span class="s2">"</span><span class="si">$(</span>wget <span class="nt">-qLO</span> - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/post-pve-install.sh<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>This fixes the no-subscription repository warning, disables the enterprise repo, and applies CPU microcode updates.</p>

<h3 id="phase-5-rejoin-the-cluster">Phase 5: Rejoin the Cluster</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pvecm add &lt;IP-of-healthy-node&gt;
</code></pre></div></div>

<p>If you get a mount error about <code class="language-plaintext highlighter-rouge">/etc/pve</code> being busy:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mv</span> /etc/pve /etc/pve.local-backup
systemctl restart pve-cluster
</code></pre></div></div>

<p>The cluster filesystem will mount and replicate the shared configuration from the other nodes.</p>

<h3 id="phase-6-reactivate-ceph-osds">Phase 6: Reactivate Ceph OSDs</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph-volume lvm activate <span class="nt">--all</span>
</code></pre></div></div>

<p>Watch the OSD status:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd tree
</code></pre></div></div>

<p>The OSDs should transition from <code class="language-plaintext highlighter-rouge">down</code> to <code class="language-plaintext highlighter-rouge">up</code>. If automatic activation fails, use the LVM layout backup to activate manually:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># From ceph_lvm_layout.txt, find the OSD fsid</span>
ceph-volume lvm activate <span class="nt">--bluestore</span> &lt;osd-id&gt; &lt;osd-fsid&gt;
</code></pre></div></div>

<h3 id="phase-7-unpause-the-cluster">Phase 7: Unpause the Cluster</h3>

<p>Once the OSDs are <code class="language-plaintext highlighter-rouge">up</code> and cluster health looks good:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd <span class="nb">unset </span>noout
ceph osd <span class="nb">unset </span>nobackfill
ceph crash archive-all
</code></pre></div></div>

<h2 id="the-uefi-difference">The UEFI Difference</h2>

<p>The main practical difference after switching from Legacy BIOS to UEFI:</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Legacy BIOS</th>
      <th>UEFI</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Bootloader</td>
      <td>GRUB</td>
      <td><code class="language-plaintext highlighter-rouge">systemd-boot</code></td>
    </tr>
    <tr>
      <td>Kernel parameters</td>
      <td><code class="language-plaintext highlighter-rouge">/etc/default/grub</code> + <code class="language-plaintext highlighter-rouge">update-grub</code></td>
      <td><code class="language-plaintext highlighter-rouge">/etc/kernel/cmdline</code> + <code class="language-plaintext highlighter-rouge">proxmox-boot-tool refresh</code></td>
    </tr>
    <tr>
      <td>Boot partition</td>
      <td>BIOS boot (1007K)</td>
      <td>EFI System Partition (1GB, FAT32)</td>
    </tr>
    <tr>
      <td>Boot tool</td>
      <td><code class="language-plaintext highlighter-rouge">proxmox-boot-tool init &lt;part&gt; grub</code></td>
      <td><code class="language-plaintext highlighter-rouge">proxmox-boot-tool init &lt;part&gt;</code></td>
    </tr>
    <tr>
      <td>Max boot disk size</td>
      <td>2TB (MBR limitation)</td>
      <td>No practical limit</td>
    </tr>
  </tbody>
</table>

<p>The <code class="language-plaintext highlighter-rouge">proxmox-boot-tool</code> commands for managing the boot mirror are the same — <code class="language-plaintext highlighter-rouge">format</code>, <code class="language-plaintext highlighter-rouge">init</code>, <code class="language-plaintext highlighter-rouge">status</code>, <code class="language-plaintext highlighter-rouge">clean</code> — the tool detects the boot mode automatically. The only workflow change is where kernel parameters live.</p>

<h2 id="verify-the-migration">Verify the Migration</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ZFS pool healthy</span>
zpool status <span class="nt">-v</span> rpool

<span class="c"># Both SSDs registered as boot targets</span>
proxmox-boot-tool status

<span class="c"># Scrub to verify data integrity</span>
zpool scrub rpool

<span class="c"># Cluster membership</span>
pvecm status

<span class="c"># Ceph OSDs active</span>
ceph osd tree
</code></pre></div></div>

<h2 id="references">References</h2>

<ul>
  <li><a href="https://aaronlauterer.com/blog/2021/proxmox-ve-migrate-to-smaller-root-disks/">Migrate Proxmox VE to smaller root disks</a> — Aaron Lauterer (send/receive approach for reference)</li>
  <li><a href="https://www.reddit.com/r/Proxmox/comments/1cr6wn7/tutorial_howto_migrate_a_pve_zfs_bootroot_mirror/">Tutorial: migrate PVE ZFS boot mirror to smaller disks</a> — Reddit (send/receive shell script)</li>
  <li><a href="https://community-scripts.github.io/ProxmoxVE/">Proxmox community post-install scripts</a> — Post-install automation</li>
</ul>

<h2 id="related-articles">Related Articles</h2>

<ul>
  <li><a href="/proxmox-zfs-boot-mirrors-part-1/">ZFS Boot Mirrors on Proxmox 8 - Part 1</a> — Same-size drive replacement</li>
  <li><a href="/proxmox-zfs-boot-mirrors-part-2/">ZFS Boot Mirrors on Proxmox 8 - Part 2</a> — Emergency recovery from catastrophic dual-drive failure</li>
  <li><a href="/proxmox-zfs-boot-mirror-smart-analysis/">Monitoring ZFS Boot Mirror Health in Proxmox 8 Clusters</a> — SMART monitoring and alerting</li>
  <li><a href="/proxmox-ceph-guide/">Proxmox &amp; Ceph Homelab Guide</a> — All my Proxmox and Ceph articles in one place</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="proxmox" /><category term="zfs" /><category term="storage" /><category term="homelab" /><category term="proxmox" /><category term="zfs" /><category term="storage" /><category term="homelab" /><category term="hardware" /><category term="boot" /><category term="mirror" /><category term="ssd" /><category term="uefi" /><summary type="html"><![CDATA[How to migrate a Proxmox ZFS boot mirror to smaller replacement drives using a fresh install approach, covering the Golden Backup checklist, UEFI upgrade from Legacy BIOS, cluster rejoin, and Ceph OSD reactivation. Based on real migrations across a six-node homelab cluster.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/proxmox-zfs-boot-mirrors-part-3.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/proxmox-zfs-boot-mirrors-part-3.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Managing Context and Rules Across Multiple AI Coding Assistants</title><link href="https://mcgarrah.org/managing-cross-ai-agent-context/" rel="alternate" type="text/html" title="Managing Context and Rules Across Multiple AI Coding Assistants" /><published>2026-06-01T00:00:00+00:00</published><updated>2026-06-01T00:00:00+00:00</updated><id>https://mcgarrah.org/managing-cross-ai-agent-context</id><content type="html" xml:base="https://mcgarrah.org/managing-cross-ai-agent-context/"><![CDATA[<p>As the landscape of AI coding assistants continues to explode — with tools like Amazon Q, Claude Code, GitHub Copilot, Cursor, Windsurf, and Gemini Code Assist — managing how these agents interact with your codebase is becoming a new kind of maintenance headache.</p>

<p>Each of these tools has its own proprietary way of defining repository rules and context. Amazon Q looks in <code class="language-plaintext highlighter-rouge">.amazonq/rules/</code>, Claude Code reads <code class="language-plaintext highlighter-rouge">CLAUDE.md</code>, GitHub Copilot wants <code class="language-plaintext highlighter-rouge">.github/copilot-instructions.md</code>, Cursor uses <code class="language-plaintext highlighter-rouge">.cursor/rules/*.mdc</code>, and so on. If you bounce between tools, or if your team members use different assistants, maintaining separate rule files for each tool is a recipe for drift and frustration.</p>

<p>For a detailed reference of what each agent expects, see the companion article <a href="/ai-coding-agent-context-files-reference/">AI Coding Agent Context Files: A Reference Guide</a>.</p>

<!-- excerpt-end -->

<h2 id="the-problem-n-tools--m-rules--drift">The Problem: N Tools × M Rules = Drift</h2>

<p>This is fundamentally the same configuration management problem as maintaining Terraform modules, Ansible roles, or Kubernetes manifests across environments — a single source of truth that needs to be projected into multiple vendor-specific formats. The tooling is different, but the drift pattern is identical.</p>

<p>Here’s what a repository looks like when you try to support every agent natively:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>project-root/
├── .amazonq/rules/
│   ├── coding-standards.md
│   └── project-context.md
├── .github/
│   └── copilot-instructions.md
├── .cursor/rules/
│   └── general.mdc
├── .windsurf/rules/
│   └── general.md
├── .gemini/
│   └── styleguide.md
├── .clinerules
├── .aider.conf.yml
├── CLAUDE.md
└── ... your actual code
</code></pre></div></div>

<p>That’s eight places containing roughly the same information. When you update a coding convention, you need to update all eight. You won’t. Nobody will. The files will drift, and different agents will enforce different rules on the same codebase.</p>

<h2 id="why-cross-referencing-doesnt-work">Why Cross-Referencing Doesn’t Work</h2>

<p>My initial thought was to just cross-reference the files. For example, creating a <code class="language-plaintext highlighter-rouge">CLAUDE.md</code> that says “Please adhere to the coding standards defined in the <code class="language-plaintext highlighter-rouge">.amazonq/rules/</code> directory.”</p>

<p>The problem with this approach is that it relies on the AI recognizing the path, deciding to crawl those specific files, and successfully fetching the text into its active context window. While agents are getting better at full-workspace awareness, relying on an AI to chase down directory references designed for a competitor’s tool is fragile. It often results in the agent hallucinating or ignoring your standards completely because the actual text of the rules never made it into the prompt’s context window.</p>

<p>Some agents handle this better than others — Claude Code will actually read files you reference — but it’s not reliable across all tools.</p>

<h2 id="strategy-1-tool-agnostic-centralization">Strategy 1: Tool-Agnostic Centralization</h2>

<p>The most robust approach is to decouple your instructions from any specific AI’s proprietary folder structure.</p>

<h3 id="step-1-create-a-root-level-source-of-truth">Step 1: Create a Root-Level Source of Truth</h3>

<p>Move the actual content of your rules out of the proprietary folders and consolidate them into a well-structured file at the root of your repository:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>project-root/
├── CONVENTIONS.md          # The single source of truth
├── .amazonq/rules/
│   └── conventions.md      # → Points to or copies from CONVENTIONS.md
├── CLAUDE.md               # → Points to or copies from CONVENTIONS.md
├── .github/
│   └── copilot-instructions.md  # → Points to or copies from CONVENTIONS.md
└── ...
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">CONVENTIONS.md</code> (or <code class="language-plaintext highlighter-rouge">RULES.md</code>, <code class="language-plaintext highlighter-rouge">CONTRIBUTING.md</code>, or <code class="language-plaintext highlighter-rouge">AI-CONTEXT.md</code> — the name doesn’t matter) contains your actual coding standards, project context, build commands, and architectural decisions. Human developers read this file too.</p>

<h3 id="step-2-make-agent-files-thin-wrappers">Step 2: Make Agent Files Thin Wrappers</h3>

<p>Each agent-specific file becomes a thin wrapper. For agents that can follow file references (Claude Code), point to the source:</p>

<div class="language-markdown highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh"># CLAUDE.md</span>
Read and follow all instructions in CONVENTIONS.md in this repository root.
</code></pre></div></div>

<p>For agents that don’t reliably follow references (most of them), you have two options:</p>

<p><strong>Option A: Duplicate with a generation note.</strong> Copy the content and add a header:</p>

<div class="language-markdown highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- AUTO-GENERATED: Source of truth is CONVENTIONS.md. Do not edit directly. --&gt;</span>

<span class="gh"># Coding Standards</span>
...
</code></pre></div></div>

<p><strong>Option B: Use a build script.</strong> Generate the agent-specific files from the source:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># sync-ai-rules.sh — Generate agent-specific files from CONVENTIONS.md</span>

<span class="nv">SOURCE</span><span class="o">=</span><span class="s2">"CONVENTIONS.md"</span>

<span class="c"># Amazon Q</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> .amazonq/rules
<span class="nb">cp</span> <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> .amazonq/rules/conventions.md

<span class="c"># Claude Code</span>
<span class="nb">cp</span> <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> CLAUDE.md

<span class="c"># GitHub Copilot</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> .github
<span class="nb">cp</span> <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> .github/copilot-instructions.md

<span class="c"># Gemini</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> .gemini
<span class="nb">cp</span> <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> .gemini/styleguide.md

<span class="c"># Cursor (needs MDC frontmatter)</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> .cursor/rules
<span class="nb">echo</span> <span class="s1">'---
description: Project conventions
alwaysApply: true
---'</span> <span class="o">&gt;</span> .cursor/rules/conventions.mdc
<span class="nb">cat</span> <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> <span class="o">&gt;&gt;</span> .cursor/rules/conventions.mdc

<span class="nb">echo</span> <span class="s2">"AI context files synced from </span><span class="nv">$SOURCE</span><span class="s2">"</span>
</code></pre></div></div>

<p>Run this as a pre-commit hook or part of your CI pipeline.</p>

<h3 id="step-3-explicitly-pass-context-when-it-matters">Step 3: Explicitly Pass Context When It Matters</h3>

<p>When starting a task where repository rules matter, pull them into the chat context using the <code class="language-plaintext highlighter-rouge">@</code>-mention feature available in most IDEs:</p>

<blockquote>
  <p>“Refactor this module, and please follow the guidelines in @CONVENTIONS.md”</p>
</blockquote>

<p>This guarantees the full text is loaded into the LLM’s context window regardless of which agent you’re using.</p>

<h2 id="strategy-2-layered-architecture">Strategy 2: Layered Architecture</h2>

<p>For larger projects, a single file gets unwieldy. Use a layered approach where the root file is a summary and agent-specific files add detail:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>project-root/
├── CONVENTIONS.md              # High-level standards (all agents read this)
├── docs/
│   ├── ARCHITECTURE.md         # Detailed architecture (referenced as needed)
│   └── CODING-STANDARDS.md     # Detailed style guide
├── .amazonq/rules/
│   ├── conventions.md          # Symlink or copy of CONVENTIONS.md
│   └── project-context.md     # Amazon Q-specific extras (memory bank, etc.)
├── CLAUDE.md                   # Points to CONVENTIONS.md + Claude-specific notes
└── .github/
    └── copilot-instructions.md # Copy of CONVENTIONS.md
</code></pre></div></div>

<p>This works well when you have one primary agent (with rich rules) and want other agents to at least get the basics.</p>

<h2 id="strategy-3-accept-the-fragmentation">Strategy 3: Accept the Fragmentation</h2>

<p>Sometimes the pragmatic answer is to pick your primary agent and maintain its rules properly, while giving other agents minimal context. Not every agent needs the full picture if you only use it occasionally.</p>

<p>For example, if Amazon Q is your daily driver:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">.amazonq/rules/</code> — Full, detailed rules (maintained)</li>
  <li><code class="language-plaintext highlighter-rouge">CLAUDE.md</code> — One-paragraph project summary + “see .amazonq/rules/ for details”</li>
  <li><code class="language-plaintext highlighter-rouge">.github/copilot-instructions.md</code> — One-paragraph project summary</li>
  <li>Everything else — Don’t bother</li>
</ul>

<p>This is honest about how most developers actually work. The risk is that when you do switch agents, the context gap shows.</p>

<h2 id="handling-exclusions">Handling Exclusions</h2>

<p>Just as you need to feed the AI the right rules, you need to keep it away from the wrong files. The exclusion landscape is fragmented:</p>

<table>
  <thead>
    <tr>
      <th>Agent</th>
      <th>Exclusion File</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Cursor</td>
      <td><code class="language-plaintext highlighter-rouge">.cursorignore</code></td>
    </tr>
    <tr>
      <td>Windsurf</td>
      <td><code class="language-plaintext highlighter-rouge">.windsurfignore</code></td>
    </tr>
    <tr>
      <td>Gemini</td>
      <td><code class="language-plaintext highlighter-rouge">.aiexclude</code></td>
    </tr>
    <tr>
      <td>Cline</td>
      <td><code class="language-plaintext highlighter-rouge">.clineignore</code></td>
    </tr>
    <tr>
      <td>Aider</td>
      <td><code class="language-plaintext highlighter-rouge">.aiderignore</code></td>
    </tr>
    <tr>
      <td>Amazon Q / Claude Code / Copilot</td>
      <td><code class="language-plaintext highlighter-rouge">.gitignore</code> (reused)</td>
    </tr>
  </tbody>
</table>

<p>All of these use <code class="language-plaintext highlighter-rouge">.gitignore</code> syntax. If you need AI-specific exclusions (files tracked in git but hidden from AI), you can maintain a single source and symlink:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create the canonical AI exclusion list</span>
<span class="nb">cat</span> <span class="o">&gt;</span> .ai-exclude <span class="o">&lt;&lt;</span><span class="sh">'</span><span class="no">EOF</span><span class="sh">'
*.env
*.pem
*.key
secrets/
credentials/
</span><span class="no">EOF

</span><span class="c"># Symlink for each agent that supports it</span>
<span class="nb">ln</span> <span class="nt">-s</span> .ai-exclude .cursorignore
<span class="nb">ln</span> <span class="nt">-s</span> .ai-exclude .windsurfignore
<span class="nb">ln</span> <span class="nt">-s</span> .ai-exclude .aiexclude
<span class="nb">ln</span> <span class="nt">-s</span> .ai-exclude .clineignore
<span class="nb">ln</span> <span class="nt">-s</span> .ai-exclude .aiderignore
</code></pre></div></div>

<h2 id="a-real-example-my-blog-repository">A Real Example: My Blog Repository</h2>

<p>Here’s how I handle this in practice for this Jekyll blog. Amazon Q is my primary agent, so it gets the full treatment:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mcgarrah.github.io/
├── .amazonq/rules/
│   ├── git-credentials-context.md    # Git auth troubleshooting
│   └── memory-bank/
│       ├── guidelines.md             # Coding standards
│       ├── product.md                # Project overview
│       ├── structure.md              # Directory layout
│       └── tech.md                   # Technology stack
└── ... (no other agent files currently)
</code></pre></div></div>

<p>If I wanted to support Claude Code and Copilot without duplicating everything, I’d add:</p>

<div class="language-markdown highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh"># CLAUDE.md</span>
This is a Jekyll 4.4.1 blog hosted on GitHub Pages at mcgarrah.org.
Read .amazonq/rules/memory-bank/ for detailed project context,
coding standards, and architecture documentation.

<span class="gu">## Quick Reference</span>
<span class="p">-</span> <span class="sb">`bundle exec jekyll serve`</span> for local dev
<span class="p">-</span> Posts in _posts/ with YYYY-MM-DD-title.md naming
<span class="p">-</span> Custom plugins in _plugins/ must set <span class="sb">`safe true`</span>
<span class="p">-</span> JavaScript: IIFEs, strict mode, const/let only
</code></pre></div></div>

<p>Claude Code will actually read those referenced files. For Copilot, I’d put a condensed version of the standards directly in <code class="language-plaintext highlighter-rouge">.github/copilot-instructions.md</code> since it won’t follow the reference.</p>

<h2 id="what-about-mcp">What About MCP?</h2>

<p>Model Context Protocol (MCP) standardizes how AI agents connect to external tools and data sources — databases, APIs, file systems, and more. It’s an important step toward interoperability.</p>

<p>However, MCP does not address project-level rules or coding conventions. It solves the “how does the agent access external resources” problem, not the “how does the agent know my coding standards” problem. These are complementary concerns, and the rules-file fragmentation remains an unsolved problem in the ecosystem.</p>

<h2 id="what-id-like-to-see">What I’d Like to See</h2>

<ul>
  <li><strong>A cross-agent standard</strong> for project rules, similar to how <code class="language-plaintext highlighter-rouge">.editorconfig</code> standardized editor settings. One file, one format, every agent reads it</li>
  <li><strong>MCP extension for conventions</strong> — A protocol-level way to serve project rules to any compliant agent</li>
  <li><strong>Agent-aware <code class="language-plaintext highlighter-rouge">.editorconfig</code></strong> — Extending the existing standard to include AI-specific directives</li>
</ul>

<p>Until then, the build-script approach (Strategy 1, Option B) is the most maintainable solution for teams that genuinely use multiple agents.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Future-proofing your repository for AI isn’t about catering to one specific vendor — it’s about making your codebase’s rules and boundaries easily readable by any agent, or human, that comes along. The centralized source of truth pattern works today, even if the tooling hasn’t caught up to make it seamless.</p>

<p>Pick your primary agent, maintain its rules properly, and give the others enough context to be useful. When a cross-agent standard eventually emerges, you’ll be ready to adopt it because your conventions are already documented in one place.</p>

<h2 id="related-articles">Related Articles</h2>

<ul>
  <li><a href="/ai-coding-agent-context-files-reference/">AI Coding Agent Context Files: A Reference Guide</a> — Detailed reference for each agent’s file format</li>
  <li><a href="/claude-code-setup-guide/">Getting Started with Claude Code</a> — Setup guide for Claude Code’s CLI workflow</li>
</ul>

<h2 id="revision-history">Revision History</h2>

<ul>
  <li><strong>2026-06-30</strong>: Expanded to cover all major agents, added practical strategies and real examples</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="technical" /><category term="development" /><category term="ai" /><category term="gemini" /><category term="amazon-q" /><category term="github-copilot" /><category term="claude-code" /><category term="cursor" /><category term="windsurf" /><category term="productivity" /><category term="tooling" /><summary type="html"><![CDATA[Practical strategies for managing project context and coding rules across multiple AI coding assistants without maintaining duplicate files for each tool.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/managing-cross-ai-agent-context.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/managing-cross-ai-agent-context.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">ZFS Boot Mirrors on Proxmox 8 for the Homelab - Part 2</title><link href="https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-2/" rel="alternate" type="text/html" title="ZFS Boot Mirrors on Proxmox 8 for the Homelab - Part 2" /><published>2026-05-30T00:00:00+00:00</published><updated>2026-05-30T00:00:00+00:00</updated><id>https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-2</id><content type="html" xml:base="https://mcgarrah.org/proxmox-zfs-boot-mirrors-part-2/"><![CDATA[<p><a href="/proxmox-zfs-boot-mirrors-part-1/">Part 1</a> covers replacing a failed drive with one of the same size. This part covers the scenario that procedure can’t fix — when both drives in the mirror fail at the same time and you’re doing an emergency recovery. <a href="/proxmox-zfs-boot-mirrors-part-3/">Part 3</a> covers the planned migration path — downsizing from large HDDs to smaller SSDs with a fresh install and UEFI upgrade, applying the lessons learned here.</p>

<p>That’s what happened to harlan, one of the six nodes in my homelab cluster. Both 500GB HDDs developed simultaneous checksum errors and permanent data corruption. The root cause wasn’t two independent drive deaths — it was a shared failure point, most likely a bad SATA cable or failing controller causing I/O errors that corrupted both sides of the mirror before ZFS could self-heal. This is why I treat shared failure points — cables, controllers, backplanes — as the first thing to investigate when correlated failures appear. Independent drive deaths are statistically rare; shared infrastructure failures are common.</p>

<p>The recovery path is a fresh Proxmox install. The goal is to get the OS back without touching the Ceph OSD drives, then rejoin the cluster and reactivate the OSDs from their existing LVM metadata.</p>

<p><a href="/assets/images/zfs-boot-mirror-proxmox8-001.png" target="_blank"><img src="/assets/images/zfs-boot-mirror-proxmox8-001.png" alt="Proxmox 8 ZFS Boot Mirror status in the web UI" width="40%" height="40%" style="display:block; margin-left:auto; margin-right:auto" /></a></p>

<!-- excerpt-end -->

<h2 id="recognizing-the-catastrophic-failure">Recognizing the Catastrophic Failure</h2>

<p>The first sign was a <code class="language-plaintext highlighter-rouge">dpkg</code> error during a routine package install:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dpkg: unrecoverable fatal error, aborting:
loading files list for package 'ssh': cannot open
/var/lib/dpkg/info/ssh.list (Invalid exchange)
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">Invalid exchange</code> (errno 52) from a filesystem operation means the kernel hit a block it couldn’t read. Running <code class="language-plaintext highlighter-rouge">zpool status -v</code> confirmed the worst:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">root@harlan:~#</span><span class="w"> </span>zpool status <span class="nt">-v</span>
<span class="go">  pool: rpool
 state: DEGRADED
status: Mismatch between pool hostid and system hostid on imported pool.
  scan: resilvered 17.5M in 00:00:08 with 7 errors on Thu Jan 15 06:47:57 2026
config:
        NAME                                        STATE     READ WRITE CKSUM
        rpool                                       DEGRADED     0     0     0
          mirror-0                                  DEGRADED     0     0     0
            ata-ST500DM002-1BD142_Z3TGX1AS-part3    DEGRADED     0     0   226  too many errors
            ata-ST500DM002-1SB10A_ZA45K50E-part3    DEGRADED     0     0  3.14K too many errors

errors: Permanent errors have been detected in the following files:
/var/lib/dpkg/info/ssh.list
</span><span class="c">...
</span></code></pre></div></div>

<p>Both drives show <code class="language-plaintext highlighter-rouge">CKSUM</code> errors — 226 and 3,140 respectively. ZFS cannot self-heal when both sides of the mirror are corrupted. The <code class="language-plaintext highlighter-rouge">Permanent errors</code> section lists files that are unrecoverable from either disk.</p>

<p><strong>This is not a normal drive failure.</strong> When you see high CKSUM counts on both sides of a mirror simultaneously, suspect:</p>

<ul>
  <li>A bad SATA cable (most common — a loose cable causes CRC errors that corrupt both drives before ZFS notices)</li>
  <li>A failing HBA or SATA controller</li>
  <li>Bad RAM (ECC errors corrupting data in flight)</li>
</ul>

<p>Check <code class="language-plaintext highlighter-rouge">dmesg | grep -E "ata|error"</code> for <code class="language-plaintext highlighter-rouge">exception Emask</code> or <code class="language-plaintext highlighter-rouge">SATA link down</code> messages. If you see those, you have a physical connection problem, not two independent drive deaths.</p>

<h2 id="the-decision-fresh-install">The Decision: Fresh Install</h2>

<p>Once you have permanent errors on both sides of a ZFS mirror, your options are:</p>

<ol>
  <li><strong>Attempt repair</strong> — only viable if the corruption is limited to non-critical files and the hardware problem is fixed</li>
  <li><strong>Fresh install</strong> — the clean path when system files are corrupted</li>
</ol>

<p>For a Proxmox cluster node with Ceph OSDs, a fresh install is actually less scary than it sounds. The Ceph data lives on separate drives and is completely independent of the OS. The cluster configuration is replicated across all nodes. What you need to preserve is the node’s <em>identity</em> so it can rejoin the cluster and reclaim its OSDs.</p>

<h2 id="what-to-back-up-before-the-drives-die-completely">What to Back Up Before the Drives Die Completely</h2>

<p>Ideally you run these backups while the node is still limping along. If the node is already dead, most of this can be reconstructed from other cluster nodes — but having it locally is faster and safer.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create backup directory on shared CephFS storage</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/pve/cephfs/backups/harlan/

<span class="c"># Capture the binary hostid — critical for ZFS pool reimport</span>
<span class="nb">hostid</span> <span class="o">&gt;</span> /mnt/pve/cephfs/backups/harlan/harlan-hostid.txt

<span class="c"># Bundle the cluster identity files</span>
<span class="nb">tar</span> <span class="nt">-cvzf</span> /mnt/pve/cephfs/backups/harlan/harlan_backup_<span class="si">$(</span><span class="nb">date</span> +%Y-%m-%d<span class="si">)</span>.tar.gz <span class="se">\</span>
  /etc/network/interfaces <span class="se">\</span>
  /etc/hosts <span class="se">\</span>
  /etc/resolv.conf <span class="se">\</span>
  /etc/hostname <span class="se">\</span>
  /etc/subuid <span class="se">\</span>
  /etc/subgid <span class="se">\</span>
  /etc/pve/ceph.conf <span class="se">\</span>
  /etc/pve/storage.cfg <span class="se">\</span>
  /etc/corosync/corosync.conf <span class="se">\</span>
  /etc/corosync/authkey <span class="se">\</span>
  /etc/ceph/ceph.client.admin.keyring <span class="se">\</span>
  /var/lib/ceph/bootstrap-osd/ceph.keyring <span class="se">\</span>
  /var/lib/pve-cluster/config.db

<span class="c"># Capture OSD-to-disk mapping</span>
ceph-volume lvm list <span class="o">&gt;</span> /mnt/pve/cephfs/backups/harlan/ceph_lvm_layout.txt
pvs <span class="o">&gt;</span> /mnt/pve/cephfs/backups/harlan/ceph_pv_layout.txt

<span class="c"># Capture /var/lib/ceph/osd symlinks (block device paths)</span>
<span class="nb">tar</span> <span class="nt">-cvzf</span> /mnt/pve/cephfs/backups/harlan/ceph_osd_var_lib.tar.gz <span class="se">\</span>
  /var/lib/ceph/osd/

<span class="c"># Capture any kernel boot quirks (important for USB OSD drives)</span>
<span class="nb">cp</span> /etc/default/grub.d/<span class="k">*</span>.cfg <span class="se">\</span>
  /mnt/pve/cephfs/backups/harlan/ 2&gt;/dev/null <span class="o">||</span> <span class="nb">true</span>

<span class="c"># List manually installed packages for post-install restoration</span>
apt-mark showmanual <span class="o">&gt;</span> /mnt/pve/cephfs/backups/harlan/harlan-apt-mark-showmanual.txt
</code></pre></div></div>

<h3 id="why-each-file-matters">Why Each File Matters</h3>

<table>
  <thead>
    <tr>
      <th>File</th>
      <th>Why it matters</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">hostid</code></td>
      <td>ZFS records the hostid that imported the pool. Mismatch causes warnings and can prevent clean import</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">corosync/authkey</code></td>
      <td>Without this, the node cannot rejoin the cluster — it’s the shared secret</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">corosync/corosync.conf</code></td>
      <td>Node list, cluster name, config version — needed to reconstruct if cluster is also degraded</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ceph.client.admin.keyring</code></td>
      <td>Ceph admin authentication — needed to run <code class="language-plaintext highlighter-rouge">ceph</code> commands post-install</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">bootstrap-osd/ceph.keyring</code></td>
      <td>Required for <code class="language-plaintext highlighter-rouge">ceph-volume lvm activate</code> to authenticate with the cluster</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ceph_lvm_layout.txt</code></td>
      <td>Maps OSD IDs to physical devices — your recovery map if <code class="language-plaintext highlighter-rouge">activate --all</code> fails</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">subuid</code> / <code class="language-plaintext highlighter-rouge">subgid</code></td>
      <td>UID/GID mappings for unprivileged LXC containers — missing this breaks container startup</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">grub.d/*.cfg</code></td>
      <td>USB storage quirks — without these, USB-attached OSD drives may not be accessible at boot</td>
    </tr>
  </tbody>
</table>

<h3 id="the-usb-quirks-file">The USB Quirks File</h3>

<p>harlan’s Ceph OSDs are on Seagate USB3 portable drives. Without a kernel parameter to disable UAS (USB Attached SCSI) for these specific drives, the kernel’s UAS driver causes I/O errors. The quirks file looked like this:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/default/grub.d/usb-quirks.cfg</span>
<span class="nv">GRUB_CMDLINE_LINUX</span><span class="o">=</span><span class="s2">"</span><span class="nv">$GRUB_CMDLINE_LINUX</span><span class="s2"> usb_storage.quirks=0bc2:ac2b:,0bc2:2344:,0bc2:ac41:,0bc2:ab9a:"</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">0bc2:*</code> vendor IDs are Seagate’s USB bridge chips. If your OSDs are on USB drives, check <code class="language-plaintext highlighter-rouge">lsusb</code> and <code class="language-plaintext highlighter-rouge">dmesg | grep -i uas</code> to identify whether you need this. Without it, the drives may appear but throw I/O errors under load.</p>

<h2 id="pause-the-ceph-cluster">Pause the Ceph Cluster</h2>

<p>Before taking the node down, tell Ceph not to rebalance data while the node is offline. Run this from any healthy cluster node:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd <span class="nb">set </span>noout
ceph osd <span class="nb">set </span>nobackfill
</code></pre></div></div>

<p>This prevents the cluster from treating harlan’s OSDs as permanently lost and starting a potentially hours-long rebalance. You’ll unset these flags after the node rejoins.</p>

<h2 id="the-fresh-install">The Fresh Install</h2>

<p>Boot from the Proxmox installer USB. The key decisions:</p>

<ul>
  <li><strong>Target disks</strong>: Select only the new boot SSDs. Do <strong>not</strong> touch the OSD drives or the WAL SSD.</li>
  <li><strong>Filesystem</strong>: ZFS RAID1 (mirror)</li>
  <li><strong>Hostname</strong>: Use the exact same hostname (<code class="language-plaintext highlighter-rouge">harlan</code>) and IP (<code class="language-plaintext highlighter-rouge">192.168.86.11</code>)</li>
  <li><strong>Boot mode</strong>: harlan was reinstalled in Legacy BIOS mode to match the existing cluster. See the note below on UEFI.</li>
</ul>

<p>The installer will create a fresh rpool on the new SSDs. The OSD drives are untouched.</p>

<h3 id="legacy-bios-vs-uefi">Legacy BIOS vs UEFI</h3>

<p>harlan was kept on Legacy BIOS for the reinstall because the other cluster nodes are also Legacy BIOS. Mixing boot modes in a cluster is fine — each node boots independently — but switching a live node from Legacy to UEFI requires repartitioning the boot disks, which adds risk during an already stressful recovery.</p>

<p>The practical rule: <strong>keep the same boot mode as the original install</strong> for an emergency recovery. Switch to UEFI on a planned fresh install when you have time to do it deliberately — see <a href="/proxmox-zfs-boot-mirrors-part-3/">Part 3</a> for the planned migration path that includes the UEFI upgrade.</p>

<h2 id="post-install-identity-restoration">Post-Install Identity Restoration</h2>

<p>Once the fresh install is up and you can SSH in, restore the node’s identity before joining the cluster.</p>

<h3 id="1-restore-the-hostid">1. Restore the HostID</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Set the hostid to match the original</span>
zgenhostid a8c00b56
</code></pre></div></div>

<p>Verify it took:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">hostid</span>
<span class="c"># should output: a8c00b56</span>
</code></pre></div></div>

<p>Without this step, ZFS will import the rpool with a hostid mismatch warning. The pool still works, but the warning persists across reboots and can cause confusion. More importantly, if you ever need to import the pool on another node for recovery, the mismatch can complicate things.</p>

<h3 id="2-restore-network-configuration">2. Restore Network Configuration</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Copy back from the backup tarball</span>
<span class="nb">tar</span> <span class="nt">-xvzf</span> /path/to/harlan_backup_2026-01-15.tar.gz <span class="se">\</span>
  <span class="nt">-C</span> / <span class="se">\</span>
  etc/network/interfaces <span class="se">\</span>
  etc/hosts <span class="se">\</span>
  etc/resolv.conf
</code></pre></div></div>

<p>Verify the interfaces match what the installer configured. The installer may have set up the bridge correctly already if you used the same hostname and IP — but confirm <code class="language-plaintext highlighter-rouge">vmbr0</code> and <code class="language-plaintext highlighter-rouge">vmbr1</code> (SAN network) are both present.</p>

<h3 id="3-restore-lxc-uidgid-mappings">3. Restore LXC UID/GID Mappings</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">tar</span> <span class="nt">-xvzf</span> /path/to/harlan_backup_2026-01-15.tar.gz <span class="se">\</span>
  <span class="nt">-C</span> / <span class="se">\</span>
  etc/subuid <span class="se">\</span>
  etc/subgid
</code></pre></div></div>

<p>If you skip this, unprivileged LXC containers will fail to start with permission errors. The Jellyfin LXC on harlan hit exactly this problem — the container started but couldn’t write to its data directories because the UID mapping was wrong.</p>

<h3 id="4-restore-usb-boot-quirks">4. Restore USB Boot Quirks</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cp</span> /path/to/etc-default-grub.d-usb-quirks.cfg <span class="se">\</span>
  /etc/default/grub.d/usb-quirks.cfg

update-grub
</code></pre></div></div>

<p>This must be done before the OSDs are activated, or the USB drives may not be accessible.</p>

<h3 id="5-run-the-proxmox-post-install-script-optional">5. Run the Proxmox Post-Install Script (Optional)</h3>

<p>The <a href="https://community-scripts.github.io/ProxmoxVE/">Proxmox community post-install script</a> fixes the no-subscription repository warning, disables the enterprise repo, and applies CPU microcode updates:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash <span class="nt">-c</span> <span class="s2">"</span><span class="si">$(</span>wget <span class="nt">-qLO</span> - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/post-pve-install.sh<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<h2 id="rejoin-the-cluster">Rejoin the Cluster</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pvecm add 192.168.86.12
</code></pre></div></div>

<p>Use the IP of any healthy cluster node. You’ll be prompted for the root password of that node.</p>

<h3 id="the-etcpve-mount-issue">The <code class="language-plaintext highlighter-rouge">/etc/pve</code> Mount Issue</h3>

<p>After <code class="language-plaintext highlighter-rouge">pvecm add</code>, Proxmox mounts the cluster filesystem at <code class="language-plaintext highlighter-rouge">/etc/pve</code>. If the directory already has files from the fresh install (it will), you may see:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount: /etc/pve: special device pmxcfs already mounted or /etc/pve busy.
</code></pre></div></div>

<p>The fix:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Move the local pve config out of the way</span>
<span class="nb">mv</span> /etc/pve /etc/pve.local-backup

<span class="c"># Let the cluster mount take over</span>
systemctl restart pve-cluster

<span class="c"># Merge any node-specific config from the backup if needed</span>
<span class="c"># (most config is cluster-wide and will come from pmxcfs)</span>
</code></pre></div></div>

<p>Once the cluster filesystem is mounted, the node’s configuration — VM/LXC definitions, storage config, user permissions — is all replicated from the cluster. You don’t need to restore <code class="language-plaintext highlighter-rouge">config.db</code> manually unless the entire cluster was down.</p>

<h3 id="verify-cluster-membership">Verify Cluster Membership</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pvecm status
</code></pre></div></div>

<p>harlan should appear in the node list with quorum. If the cluster had 6 nodes before and now shows 6 again, you’re in good shape.</p>

<h2 id="reactivate-the-ceph-osds">Reactivate the Ceph OSDs</h2>

<p>The OSD data is intact on the drives. Ceph just needs to be told to activate them on the new OS install.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph-volume lvm activate <span class="nt">--all</span>
</code></pre></div></div>

<p>This scans the LVM metadata on all drives, finds the Ceph OSD volumes, and starts the OSD daemons. For harlan, this brought back osd.0, osd.3, and osd.6.</p>

<p>Watch the OSD status from any cluster node:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd tree
</code></pre></div></div>

<p>The OSDs should transition from <code class="language-plaintext highlighter-rouge">down</code> to <code class="language-plaintext highlighter-rouge">up</code>. They’ll stay <code class="language-plaintext highlighter-rouge">in</code> because the cluster map still has them — they were never removed, just offline.</p>

<h3 id="if-activate---all-fails">If <code class="language-plaintext highlighter-rouge">activate --all</code> Fails</h3>

<p>If the automatic activation doesn’t find the OSDs, use the LVM layout backup to activate them manually:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># From ceph_lvm_layout.txt, find the OSD fsid and activate by ID</span>
ceph-volume lvm activate <span class="nt">--bluestore</span> &lt;osd-id&gt; &lt;osd-fsid&gt;

<span class="c"># Example for osd.0:</span>
ceph-volume lvm activate <span class="nt">--bluestore</span> 0 3f49e837-c410-4025-bcf5-af5e6cd2c173
</code></pre></div></div>

<p>The OSD fsids are in the <code class="language-plaintext highlighter-rouge">ceph_lvm_layout.txt</code> backup under <code class="language-plaintext highlighter-rouge">osd fsid</code>.</p>

<h2 id="unpause-the-cluster">Unpause the Cluster</h2>

<p>Once harlan’s OSDs are <code class="language-plaintext highlighter-rouge">up</code> and the cluster health is <code class="language-plaintext highlighter-rouge">HEALTH_OK</code> (or <code class="language-plaintext highlighter-rouge">HEALTH_WARN</code> with only expected warnings):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ceph osd <span class="nb">unset </span>noout
ceph osd <span class="nb">unset </span>nobackfill

<span class="c"># Archive any crash reports from the outage</span>
ceph crash archive-all
</code></pre></div></div>

<h2 id="verify-the-new-boot-mirror">Verify the New Boot Mirror</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@harlan:~# zpool status <span class="nt">-v</span>
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B <span class="k">in </span>00:00:12 with 0 errors on Sun Apr 12 00:24:13 2026
config:
        NAME                                STATE     READ WRITE CKSUM
        rpool                               ONLINE       0     0     0
          mirror-0                          ONLINE       0     0     0
            ata-SSD_YS202010015363AA-part3  ONLINE       0     0     0
            ata-SSD_YS202010025083AA-part3  ONLINE       0     0     0

errors: No known data errors

root@harlan:~# proxmox-boot-tool status
System currently booted with legacy bios
D91B-B982 is configured with: grub <span class="o">(</span>versions: 6.8.12-18-pve<span class="o">)</span>
D91C-8216 is configured with: grub <span class="o">(</span>versions: 6.8.12-18-pve<span class="o">)</span>
</code></pre></div></div>

<p>Both SSDs listed, both registered with grub, no errors. The recovery is complete.</p>

<h3 id="the-residual-hostid-warning">The Residual HostID Warning</h3>

<p>After the fresh install and <code class="language-plaintext highlighter-rouge">zgenhostid</code>, harlan still shows this on <code class="language-plaintext highlighter-rouge">zpool status</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>status: Mismatch between pool hostid and system hostid on imported pool.
</code></pre></div></div>

<p>This is a cosmetic warning from the initial boot before <code class="language-plaintext highlighter-rouge">zgenhostid</code> was run. It clears permanently after a clean export and reimport of the pool, which requires a live migration of any running VMs/LXCs off the node first. It does not affect pool operation or data integrity. You can leave it or clear it during a future maintenance window with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Migrate all VMs/LXCs off harlan first, then:</span>
zpool <span class="nb">export </span>rpool
zpool import rpool
</code></pre></div></div>

<h2 id="the-backup-checklist">The Backup Checklist</h2>

<p>Based on the harlan recovery, here’s the minimum backup set every Proxmox Ceph node should have on shared storage:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Run on each node, store to /mnt/pve/cephfs/backups/&lt;nodename&gt;/</span>
<span class="nv">NODE</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span><span class="si">)</span>
<span class="nv">BACKUP_DIR</span><span class="o">=</span><span class="s2">"/mnt/pve/cephfs/backups/</span><span class="k">${</span><span class="nv">NODE</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">"</span>

<span class="nb">hostid</span> <span class="o">&gt;</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">NODE</span><span class="k">}</span><span class="s2">-hostid.txt"</span>

<span class="nb">tar</span> <span class="nt">-czf</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">NODE</span><span class="k">}</span><span class="s2">_backup_</span><span class="si">$(</span><span class="nb">date</span> +%Y-%m-%d<span class="si">)</span><span class="s2">.tar.gz"</span> <span class="se">\</span>
  /etc/network/interfaces <span class="se">\</span>
  /etc/hosts <span class="se">\</span>
  /etc/resolv.conf <span class="se">\</span>
  /etc/hostname <span class="se">\</span>
  /etc/subuid <span class="se">\</span>
  /etc/subgid <span class="se">\</span>
  /etc/pve/ceph.conf <span class="se">\</span>
  /etc/pve/storage.cfg <span class="se">\</span>
  /etc/corosync/corosync.conf <span class="se">\</span>
  /etc/corosync/authkey <span class="se">\</span>
  /etc/ceph/ceph.client.admin.keyring <span class="se">\</span>
  /var/lib/ceph/bootstrap-osd/ceph.keyring <span class="se">\</span>
  /var/lib/pve-cluster/config.db <span class="se">\</span>
  2&gt;/dev/null

ceph-volume lvm list <span class="o">&gt;</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/ceph_lvm_layout.txt"</span>
pvs <span class="o">&gt;</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/ceph_pv_layout.txt"</span>
<span class="nb">tar</span> <span class="nt">-czf</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/ceph_osd_var_lib.tar.gz"</span> /var/lib/ceph/osd/
<span class="nb">cp</span> /etc/default/grub.d/<span class="k">*</span>.cfg <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/"</span> 2&gt;/dev/null <span class="o">||</span> <span class="nb">true
</span>apt-mark showmanual <span class="o">&gt;</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BACKUP_DIR</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">NODE</span><span class="k">}</span><span class="s2">-apt-mark-showmanual.txt"</span>
</code></pre></div></div>

<p>Run this as a cron job or after any significant configuration change. The backup is small (under 50KB for most nodes) and the CephFS storage is replicated across the cluster.</p>

<h2 id="related-articles">Related Articles</h2>

<ul>
  <li><a href="/proxmox-zfs-boot-mirrors-part-1/">ZFS Boot Mirrors on Proxmox 8 - Part 1</a> — Same-size drive replacement</li>
  <li><a href="/proxmox-zfs-boot-mirrors-part-3/">ZFS Boot Mirrors on Proxmox 8 - Part 3</a> — Planned migration to smaller SSDs with fresh install and UEFI upgrade</li>
  <li><a href="/proxmox-zfs-boot-mirror-smart-analysis/">Monitoring ZFS Boot Mirror Health in Proxmox 8 Clusters</a> — SMART monitoring and alerting</li>
  <li><a href="/proxmox-ceph-guide/">Proxmox &amp; Ceph Homelab Guide</a> — All my Proxmox and Ceph articles in one place</li>
</ul>]]></content><author><name>Michael McGarrah</name><email>mcgarrah@gmail.com</email><uri>https://mcgarrah.org/about/</uri></author><category term="proxmox" /><category term="zfs" /><category term="storage" /><category term="homelab" /><category term="ceph" /><category term="proxmox" /><category term="zfs" /><category term="storage" /><category term="homelab" /><category term="hardware" /><category term="boot" /><category term="mirror" /><category term="ceph" /><category term="recovery" /><category term="disaster-recovery" /><summary type="html"><![CDATA[Step-by-step guide to recovering a Proxmox node after catastrophic dual-drive ZFS boot mirror failure, covering pre-failure backup strategy, fresh install, hostid restoration, cluster rejoin, and Ceph OSD reactivation without data loss.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://mcgarrah.org/assets/images/og/proxmox-zfs-boot-mirrors-part-2.png" /><media:content medium="image" url="https://mcgarrah.org/assets/images/og/proxmox-zfs-boot-mirrors-part-2.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>