StoryFolder Help

Appearance

Image pipeline: automatic responsive + WebP variants for every image in public/ ​

Status: implementation plan, 2026-09-09. Branch feat/image-pipeline, worktree /Users/jeffjassky/Projects/scenedetect-images.

Goal ​

Every .jpg/.jpeg/.png under public/ automatically gets a ladder of resized variants plus WebP equivalents, with zero manual steps, and every place that renders an image (marketing components, blog heroes, blog body HTML) serves that ladder. Free (no paid CDN features). Incremental: only regenerate what changed.

Non-goals ​

  • No change to OG image generation (server/router/og-image.js). Untouched.
  • No change to serving/caching (server/main.js Cache-Control is already done).
  • No AVIF. WebP only.
  • No SVG/GIF/MP4 processing.

Design in one paragraph ​

One script (scripts/images.js) walks public/, hashes each source image, and writes <base>@w<N><ext> + <base>@w<N>.webp beside it for each rung of the ladder, skipping anything whose hash and outputs are already present. It writes public/image-manifest.json describing what exists. Derived files and the manifest are gitignored (regenerated on every yarn build/yarn serve via prebuild/preserve hooks, and in CI with an actions/cache so steady-state runs take seconds). Three consumers read the manifest: ResponsiveImage.vue, Capture.vue, and an isomorphic helper upgradeImages(html, manifest) that rewrites <img> tags in blog body HTML into <picture> — used both server-side (blog-meta-inject.js) and client-side (PageBlogArticle.vue).

Fixed contracts (every agent depends on these — do not deviate) ​

Derived file naming ​

For source /blog-media/x/hero.jpg (width W):

  • Rungs: [600, 1000, 1600] filtered to < W, plus min(W, 1600) appended if not already present. Examples: W=2400 → [600,1000,1600]; W=1600 → [600,1000,1600]; W=900 → [600,900]; W=500 → [500]; W=1437 → [600,1000,1437].
  • For each rung N: /blog-media/x/hero@w<N>.jpg (same format as source; PNG stays PNG) and /blog-media/x/hero@w<N>.webp.
  • Never upscale. Never modify the source file.
  • The existing convention is exactly this naming (@w600/@w1000/@w1600), and 38 database featureImage values point at …@w1600.jpg, so the rung set and the @w naming must be preserved exactly.

Manifest: public/image-manifest.json ​

json
{
  "version": 1,
  "images": {
    "/blog-media/x/hero.jpg": {
      "width": 2400,
      "height": 1260,
      "ext": ".jpg",
      "widths": [600, 1000, 1600],
      "webp": true
    }
  }
}
  • Keys are site-absolute URL paths (leading /, forward slashes, relative to public/).
  • version is the script's SCRIPT_VERSION constant. If it changes, everything regenerates.
  • Keys are sorted so the file diff is stable.
  • Source sha1 hashes are NOT in the manifest (it is inlined into the client bundle; 647 hashes cost 18 KB gzipped). They live in a sidecar public/.image-cache.json ({version, hashes: {key: sha1}}) that only scripts/images.js reads.
  • Consumers must treat a missing key as "no variants, use the plain src".

Client-side manifest import ​

Vue components import it as import manifest from '@/../public/image-manifest.json'; (webpack resolves @ to src/). The prebuild/preserve hooks guarantee it exists before webpack runs. Do not add fallbacks for a missing file in components.

Server-side manifest load ​

server/utils/upgrade-images.js exports loadManifest(distDir) which reads <distDir>/image-manifest.json once, returns {version, images} and returns {version: 0, images: {}} (never throws) if missing or unparsable.

Work packages ​

Four packages, disjoint files, run in parallel. Agents do not commit. The reviewer commits after review. Agents run yarn test (node --test) freely. Agents must NOT run yarn build, yarn serve or yarn prerender — the reviewer does that once at the end.

WP-A: generator script (owner: agent A) ​

Files: scripts/images.js (new), scripts/images.test.js (new), package.json (scripts only), .gitignore, scripts/set-feature-images.js (one error-message line), and the git index for public/**/*@w* (see step 5).

  1. scripts/images.js, CommonJS, requires only sharp, node:fs, node:path, node:crypto, node:os. Export run({ publicDir, log, concurrency }) returning { scanned, generated, skipped, removed, manifestPath }. CLI guard if (require.main === module) runs run({ publicDir: path.join(__dirname, '..', 'public') }) and prints one summary line; --verbose prints per-file lines; exits non-zero on any sharp error.
  2. Algorithm:
    • Walk publicDir recursively. Source = file matching /\.(jpe?g|png)$/i whose basename does NOT match /@w\d+\.[a-z0-9]+$/i. Skip .DS_Store, skip nothing else.
    • Load existing manifest if present (tolerate missing/corrupt → empty). If manifest.version !== SCRIPT_VERSION, treat as empty.
    • For each source: hash = sha1(bytes). Compute expected rung list from the source width (read via sharp(file).metadata() — only when needed; if the manifest entry hash matches, reuse its stored width/height/widths and only stat the expected outputs). If hash matches AND every expected output file exists → skipped++, keep entry. Otherwise generate all outputs with sharp (.resize({ width: N, withoutEnlargement: true }); jpeg { quality: 80, mozjpeg: true }; png { compressionLevel: 9 }; webp { quality: 80 }), strip metadata (sharp default), write entry, generated++.
    • Concurrency: a simple promise pool of os.cpus().length (min 2).
    • Orphans: any file in publicDir matching /@w\d+\.(jpe?g|png|webp)$/i whose source (strip @w\d+; for .webp try .jpg, .jpeg, .png in that order) no longer exists → delete, removed++. Manifest entries whose source is gone → drop.
    • Write manifest (2-space JSON, sorted keys) only if its content changed.
  3. scripts/images.test.js (node:test, .js so the repo's yarn test glob picks it up):
    • Uses fs.mkdtempSync and sharp to create fixtures: a 2400×1260 jpg, a 900×600 png, a 400×300 jpg, and a stray orphan [email protected] with no source.
    • Run once: assert outputs exactly [600,1000,1600] jpg+webp for the first; [600,900] png+webp for the second; [400] for the third; orphan removed; manifest keys/widths correct; hash is 40 hex chars.
    • Run twice: assert generated === 0, skipped === 3, and output mtimes unchanged.
    • Modify the 900px png (rewrite with different pixels), run: generated === 1.
    • Delete the 400px source, run: its variants removed, manifest entry gone.
    • Delete one output file only, run: that source regenerates (generated === 1).
  4. package.json scripts: add "images": "node scripts/images.js", "prebuild": "yarn images", "preserve": "yarn images". Touch nothing else in package.json.
  5. .gitignore — append:
    # Derived image variants + manifest. Regenerated by `yarn images` (scripts/images.js),
# which also runs before every build/serve. Sources stay committed; these never are.
public/**/*@w[0-9]*.*
public/image-manifest.json
    Then git rm -r --cached --quiet every currently tracked file matching public/**/*@w* (291 files: 202 jpg, 89 png). Leave them on disk. Confirm with git ls-files 'public/**/*@w*' | wc -l → 0.
  6. scripts/set-feature-images.js: its "Not on disk, refusing to wire" error should add one line: Run \yarn images` first — @w variants are generated, not committed.`
  7. Run node scripts/images.js --verbose once in the worktree so the manifest and the full ladder exist for the reviewer. Report the summary line and wall time in your final message. Then run it a second time and report that summary line too (expect generated 0).

Completion criteria: yarn test passes including the new test; second run of the script reports generated: 0; git status shows the 291 removals staged and no derived files untracked; public/image-manifest.json exists and is gitignored (git check-ignore confirms).

WP-B: blog body <img> → <picture> (owner: agent B) ​

Files: server/utils/upgrade-images.js (new), server/utils/upgrade-images.test.js (new), server/middlewares/blog-meta-inject.js (call site), src/pages/PageBlogArticle.vue (call site).

  1. server/utils/upgrade-images.js, CommonJS, no dependencies, no Node built-ins in the exported transform (it is bundled into the browser by webpack). Exports:
    • upgradeImages(html, manifest, { sizes } = {}) → string.
    • srcsetFor(entry, base, ext) and webpSrcsetFor(entry, base) helpers (exported for tests and for the Vue components to stay consistent).
    • loadManifest(distDir) — the only function that uses fs/path; require them inside the function so webpack's browser bundle does not pull them in.
    • DEFAULT_SIZES = '(min-width: 1000px) 940px, 100vw'.
  2. upgradeImages behaviour:
    • Find every <img …> tag (/<img\b[^>]*>/gi). Parse attributes with a tolerant regex (double-quoted, single-quoted, unquoted, bare). If parsing fails, return the tag unchanged.
    • Skip if the tag has data-optimized.
    • Look up attrs.src in manifest.images. If absent, or widths is empty → unchanged.
    • Otherwise emit:
      html
      <picture><source type="image/webp" srcset="<base>@w600.webp 600w, …" sizes="<sizes>"><img … data-optimized="1"></picture>
      where the inner <img> keeps every original attribute except srcset and sizes, which are replaced by the manifest-derived jpg/png srcset and the resolved sizes (attrs.sizes if present, else the option, else DEFAULT_SIZES). If the original has no width/height, add them from the manifest entry. Attribute order: original order, then the added ones.
    • <base>/<ext> derive from src exactly as ResponsiveImage.vue does today: /^(.*)(\.[a-zA-Z0-9]+)$/.
    • Never throws. If manifest is falsy or has no images → return html unchanged.
  3. Tests (node:test): empty manifest passthrough; unknown src passthrough; known src → picture with correct webp/jpg srcsets, original alt/class/loading preserved, sizes default applied; existing Ghost srcset/sizes replaced; width/height added when missing and preserved when present; idempotent (running twice yields identical output); PNG source keeps .png fallback srcset; malformed tag left alone; loadManifest on a missing dir returns {version: 0, images: {}}.
  4. server/middlewares/blog-meta-inject.js: load the manifest once at module scope with loadManifest(path.join(__dirname, '..', '..', 'dist')) (same dist resolution server/router/og-image.js line ~99 uses) and wrap the body at the single interpolation site (${article.body} in the article body builder, ~line 185) as ${upgradeImages(article.body, manifest)}. Do not touch anything else in that file. Read the file header first; it is dense and deliberate.
  5. src/pages/PageBlogArticle.vue: import { upgradeImages } from '../../server/utils/upgrade-images'; and import manifest from '@/../public/image-manifest.json';. At the END of the existing processedBody() computed (after all existing regex passes, before the return), add html = upgradeImages(html, manifest);. Nothing else changes.

Completion criteria: yarn test passes with the new test file; grep -n upgradeImages server/middlewares/blog-meta-inject.js src/pages/PageBlogArticle.vue shows exactly one call site each; the transform module has no top-level require of Node built-ins.

WP-C: components (owner: agent C) ​

Files: src/components/atoms/ResponsiveImage.vue, src/components/marketing/sections/appframe/Capture.vue.

Both import manifest from '@/../public/image-manifest.json' and { srcsetFor, webpSrcsetFor } from '../../../server/utils/upgrade-images' (adjust relative depth per file; do not add a webpack alias). WP-B owns that helper and its signature is fixed above: srcsetFor(entry, base, ext) returns the jpg/png srcset string, webpSrcsetFor(entry, base) the webp one, both from entry.widths. Until WP-B lands, write against that contract; do not stub it in your files.

  1. ResponsiveImage.vue:
    • Root becomes <picture class="responsive-image-picture"> containing <source v-if="webpSrcset" type="image/webp" :srcset="webpSrcset" :sizes="sizes"> then the existing <img> unchanged in attributes.
    • New computed entry = manifest.images[this.src] || null.
    • srcsetAttr: if entry and entry.widths.length → srcsetFor(entry, base, ext); else keep the current behaviour (derive from the widths prop) so any image not yet in the manifest renders exactly as today.
    • webpSrcset: entry && entry.webp ? webpSrcsetFor(entry, base) : ''.
    • resolvedWidth/resolvedHeight: prop if given, else entry.width/entry.height, else null.
    • Styles: keep .responsive-image rules on the <img>; add .responsive-image-picture { display: block; }.
    • Keep every existing prop, comment and the fetchPriorityAttr logic verbatim.
  2. Capture.vue:
    • Add <source v-if="webpSrcset" type="image/webp" :srcset="webpSrcset" :sizes="media.sizes || null">after the existing mobileSrc <source> and before CaptureImage/<img>. Order matters: the browser takes the first matching <source>, and the mobile crop must win below 600px.
    • Computed entry = manifest.images[media.src] || null.
    • srcset: if entry and entry.widths.length → srcsetFor(...); else current logic (from media.widths), unchanged.
    • webpSrcset as above. Also add entry-derived width/height fallbacks on the <img> only when media.width/media.height are absent.
    • Keep the header comment; append one sentence explaining the manifest-driven webp source.

Completion criteria: yarn lint --no-fix reports no errors for the two files (npx vue-cli-service lint --no-fix src/components/atoms/ResponsiveImage.vue src/components/marketing/sections/appframe/Capture.vue); both files still compile with node -e "require('vue-template-compiler').parseComponent(require('fs').readFileSync(process.argv[1],'utf8'))" without throwing; no other files changed.

WP-D: CI (owner: agent D) ​

File: .github/workflows/aws.yml only.

  1. After the Checkout step and before Make envfile, add:
    yaml
    - name: Use Node 20
  uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: yarn

# Derived image variants (see scripts/images.js and docs/plans/image-pipeline.md)
# are gitignored and regenerated here. The cache holds last run's outputs so
# only sources that changed since then are re-encoded. The key hashes the
# sources and the script; restore-keys falls back to the newest cache so a
# miss on the exact key still restores everything except the changed files.
- name: Restore derived image variants
  uses: actions/cache@v4
  with:
    path: |
      public/**/*@w[0-9]*.*
      public/image-manifest.json
    key: images-v1-${{ hashFiles('public/**/*.jpg', 'public/**/*.jpeg', 'public/**/*.png', 'scripts/images.js') }}
    restore-keys: |
      images-v1-

- name: Install dependencies
  run: YOUTUBE_DL_SKIP_PYTHON_CHECK=1 yarn install --frozen-lockfile --ignore-engines

- name: Generate image variants
  run: yarn images --verbose
    Mirror the exact yarn install flags the Dockerfile uses (they are quoted above).
  2. Purge step (last step, "Purge changed media from Cloudflare's cache"): derived variants are no longer in git, so git diff -- public/ will never list them. After CHANGED is computed, expand it: for each changed path that is a source image (.jpg/.jpeg/.png, not @w), read public/image-manifest.json and append each <base>@w<N><ext> and <base>@w<N>.webp from that entry's widths. Do this in the existing Node fetch block (it already has the list in memory) rather than in bash. Keep the 30-per-request batching. Keep the secrets-unset early exit. Keep every existing comment.
  3. Validate: python3 -c "import yaml; yaml.safe_load(open('.github/workflows/aws.yml'))". Also run node --check on the extracted Node block if you touched it (extract to a temp file under the scratchpad).

Completion criteria: YAML parses; the diff touches only the four new steps and the purge step's Node block; no secrets.* appears in any if:.

Reviewer checklist (after all four land) ​

  1. yarn test green.
  2. node scripts/images.js second run → generated: 0, under 15 s.
  3. yarn build succeeds (runs prebuild → images → vue-cli build). Then:
    • dist/image-manifest.json exists.
    • grep -c 'image/webp' dist/index.html is irrelevant (SPA shell); instead compile-check by grepping the built JS for image/webp and @w.
  4. Every featureImage in Mongo that ends in @wN.ext has that file under public/ after generation (script in reviewer's notes).
  5. Server SSR: node -e require blog-meta-inject's body builder with a fake article and a loaded manifest and confirm <picture> appears with both srcsets.
  6. git status: only intended files; 291 deletions staged; no derived files tracked.
  7. Commit on feat/image-pipeline, merge --no-ff into development, push.
  8. Post-deploy: curl -sI https://storyfolder.com/blog-media/<slug>/[email protected] → 200, content-type: image/webp, cf-cache-status HIT on second request.

WP-E: coverage — heroes, captures, cards, section fallbacks (added after review) ​

The first four packages covered blog bodies and Capture's fixture branch. An audit of the live pages found the actual LCP elements still bypass the manifest, plus one bug. Files: server/utils/upgrade-images.js (+test), src/components/marketing/CaptureImage.vue, src/components/marketing/sections/appframe/Capture.vue, src/components/atoms/ResponsiveImage.vue, src/pages/PageBlogArticle.vue, src/pages/PageBlog.vue, and the section components listed in step 5.

  1. resolveEntry(manifest, src) exported from upgrade-images.js: returns { entry, base, ext } | null. Looks up src in manifest.images; if absent and src matches /^(.*)@w\d+(\.[a-zA-Z0-9]+)$/, looks up <base><ext> instead. base/ext are always the source's. upgradeImages and both Vue components use it, so the derived-path handling lives in one place (37 Article.featureImage values are …/[email protected]). Tests: direct hit, derived hit, miss, malformed.
  2. Bug fix in Capture.vue. The webp <source> is keyed on media.src, the fallback image; when media.capture is set that source precedes CaptureImage, so the browser shows the fallback's webp instead of the app capture. Fix: webpSrcset returns '' when media.capture is truthy. CaptureImage handles its own webp (next step).
  3. CaptureImage.vue becomes manifest-aware. Root becomes <picture>; a webp <source> and an <img srcset> are emitted only when resolveEntry(manifest, captureSrc) hits and the component is NOT in usingFallback. In fallback state render exactly today's plain <img> so the @error → fallback flow is untouched. Keep every prop, data-capture* attribute, class, eager behaviour and comment. Width/height: prop, else entry. sizes prop (String, default '(min-width: 1000px) 560px, 100vw') added and passed to both.
  4. Blog hero and cards. PageBlogArticle.vue line ~42: replace the raw <img> with <ResponsiveImage :src="article.featureImage" :alt="article.title" loading="eager" sizes="(min-width: 1200px) 1160px, 100vw" /> (register the component). PageBlog.vue lines ~57 and ~83: <ResponsiveImage :src="…featureImage" :alt="…title" sizes="(min-width: 900px) 33vw, 100vw" /> (lazy default). ResponsiveImage must resolve derived srcs via resolveEntry; when the manifest misses AND src is a derived path, emit no srcset at all (today it would build hero@[email protected]). Check the .feature-image img and .card-image-link img LESS still applies (they target img, which is still inside the picture — confirm, don't restyle).
  5. Section fallback branches → ResponsiveImage. In SplitA, AnnotatedA, AnnotatedB, SeamA, HeroThumbnailWall, WallLifted, ArtifactFan, AutofillBoard: every raw <img> whose src is a /marketing-media/… or /blog-media/… path becomes <ResponsiveImage>, passing through alt, width, height, sizes, loading/fetchpriority semantics (HeroThumbnailWall's hero is loading="eager"). Where a component computes its own srcset from widths, delete that computed; ResponsiveImage derives it. Do NOT touch <img>s whose src is an /api/… URL, an inline SVG, or a data URI. Keep all classes on the <img> by passing them via img-class — add that String prop to ResponsiveImage and bind it as :class="['responsive-image', imgClass]" on the <img> — because several of these components' LESS rules target a class on the <img> itself.
  6. Verification: yarn test green; lint of every touched .vue clean apart from the pre-existing vue/no-v-html warning; template-compile check on each; yarn build green (reviewer runs it); grep -rn '<img' src/components/marketing/sections src/pages/PageBlog*.vue lists only the exclusions from step 5 and the two atoms.