Runbook

• Email subject: [auction-intel] WARN — Google Drive 80% full
• Storage state changed to YELLOW

[storage-failover] state GREEN -> YELLOW

Cold-tier Drive crossed the 80% threshold. No system disruption — this is an early warning to upgrade Google One to a 5 TB plan within ~30 days before reaching ORANGE (95%).

1. Log into [email protected] Google account at https://one.google.com
2. Upgrade Google One to 5 TB plan
3. Wait one storage-failover tick (~30 min). State auto-recovers to GREEN.

• app/services/storage_failover.py
• /home/auction/gdrive (rclone mount)

• Email subject: [auction-intel] URGENT — Google Drive 95% full
• Photo tier mover skipping with /var/lib/auction-intel/.tier-mover-paused
• VPS hot tier filling up

[storage-failover] tier mover PAUSED
[storage-failover] state YELLOW -> ORANGE

Drive cold tier crossed 95%. Storage failover paused photo_tier_mover.sh automatically — new photos stay on the VPS hot tier. Scrapers continue. If left, the VPS will fill in a few weeks and trip RED.

1. Upgrade Google One to a larger plan immediately (https://one.google.com)
2. Once Drive < 90%: tier mover auto-resumes within 30 min
3. Manual resume: rm /var/lib/auction-intel/.tier-mover-paused

• /var/lib/auction-intel/.tier-mover-paused
• /opt/auction-intel/scripts/photo_tier_mover.sh

• Photos 404 from /static/
• ls /opt/auction-intel/static/ shows fewer dirs than expected

Transport endpoint is not connected

mergerfs unifies /var/lib/auction-intel/static-hot/ + /home/auction/gdrive/AI Projects/auction-intel/static/. If rclone (Drive) is unmounted, mergerfs degrades silently and shows only hot tier.

1. Check rclone: mount | grep gdrive
2. If missing: systemctl restart rclone-gdrive.service
3. Then: systemctl restart mergerfs-static.service
4. Verify: ls /opt/auction-intel/static/ | wc -l

• /etc/systemd/system/rclone-gdrive.service
• /etc/systemd/system/mergerfs-static.service

• VPS hot tier growing while Drive shows free space
• Pause flag exists despite Drive < 90%

[skip] tier mover paused

/var/lib/auction-intel/.tier-mover-paused was set during ORANGE/RED and never auto-cleared (e.g. storage_failover never ran while Drive recovered).

1. Confirm Drive % via df -h /home/auction/gdrive
2. If < 90%: rm /var/lib/auction-intel/.tier-mover-paused
3. Or run storage check manually: python3 -c 'from app.services.storage_failover import run_storage_check; print(run_storage_check())'

• /var/lib/auction-intel/.tier-mover-paused
• /opt/auction-intel/scripts/photo_tier_mover.sh

• df -h /home/auction/gdrive errors with 'Transport endpoint not connected'
• Cold-tier photos 404
• Backups failing

fuse: bad mount point
rclone: token expired

FUSE mount can drop after long uptime, network blip, or expired OAuth token.

1. fusermount -u /home/auction/gdrive (if half-mounted)
2. systemctl restart rclone-gdrive.service
3. If token expired: as user 'auction' run rclone config reconnect gdrive:
4. Verify: ls /home/auction/gdrive | head

• /etc/systemd/system/rclone-gdrive.service
• /home/auction/.config/rclone/rclone.conf

• Email subject: [auction-intel] CRITICAL — System paused, storage full
• Scrapers all show 'disabled' on /scrapes/ page
• scrapers_enabled = False in settings.json

[storage-failover] scrapers_enabled = False
[storage-failover] state ORANGE -> RED

Drive ≥95% AND VPS hot tier ≥90%. Both safety valves exhausted. Storage failover automatically set scrapers_enabled=False to prevent the VPS from running out of disk.

1. Upgrade Google One plan (https://one.google.com)
2. Wait — once Drive < 90% AND VPS < 80%, scrapers auto-resume within 30 min
3. Force resume (only after confirming space): set scrapers_enabled=True via /admin/health or settings.json

• data/settings.json (scrapers_enabled key)
• app/services/storage_failover.py:run_storage_check

• How to read storage states / thresholds

4-state machine: GREEN < 80% | YELLOW 80-95% | ORANGE ≥95% | RED Drive≥95% AND VPS≥90%. Recovery uses hysteresis (drop below DRIVE_RECOVER=90 / VPS_RECOVER=80). Email alerts fire on transitions only.

1. Snapshot: from app.services.storage_failover import classify_state; classify_state()
2. Force tick: run_storage_check()
3. State persists in data/settings.json under storage_state key
4. Alert recipient: [email protected]

• app/services/storage_failover.py

• Screenshots / photos accumulate for status='expired'/'deleted'/'superseded' listings indefinitely
• Disk usage growing slowly with no cleanup hitting these tiers

app/services/storage_policy.py only handles `status == "sold"` (the `_cleanup_sold_statutory` rule). The other terminal statuses do NOT trigger screenshot/photo cleanup: - expired (Manheim 241, Pickles 19, IAAI 631 as of 2026-05-04) - deleted (Carsales 1203, Manheim 3, Pickles 5) - superseded (Grays 1) - relisted (Pickles 1563, Manheim 70, IAAI 6) — should NOT be cleaned, may revert - referred (Pickles 179, IAAI 11) — should NOT be cleaned, may revert Net effect: ~2,000+ historical listings never have their screenshots reclaimed. At ~200 KB per screenshot, that's ~400 MB sitting forever. Not a crisis but worth fixing for long-term unattended operation.

1. Extend storage_policy._cleanup_sold_statutory to also include status in ('expired','deleted','superseded')
2. Use a 60-day grace period (vs 30 for sold) to allow late re-list / data corrections
3. Per feedback_never_delete_data.md: photos/listings stay; we only delete SCREENSHOTS, never the canonical images
4. Test on a small batch first: add a `--dry-run` flag to confirm the SQL targets the right rows

• app/services/storage_policy.py
• app/services/storage_policy.py:_cleanup_sold_statutory

• [scheduler] grays bid monitor failed: <some sqlalchemy/asyncpg error>
• Traceback originates at scheduler.py _run_grays_monitor → monitor_grays_closing

_run_grays_monitor
monitor_grays_closing
grays bid monitor failed

The Grays bid-monitor 60s sweep wraps `monitor_grays_closing()` in a try/except and logs at WARN level. The actual root cause is whatever the inner await blew up on — usually transient (cached-statement after a migration, a brief connection pool blip, etc.). One-off failures auto-recover on the next 60s tick.

1. Confirm it's a one-off: grep 'grays bid monitor failed' /var/log/auction-intel/server.log | tail -10
2. If repeating every minute: pull the full traceback from the archive — likely a real query bug in monitor_grays_closing
3. Restart the service if the same row keeps failing: systemctl restart auction-intel
4. If asyncpg cache-related: see database-asyncpg-cached-statement-stale

• app/scrapers/auction/grays/monitor.py:monitor_grays_closing
• app/services/scheduler.py:_run_grays_monitor

• AVIBM bot crashes with -9 / SIGKILL
• session not created: This version of ChromeDriver only supports Chrome version X

signal -9
killed with signal -9
SIGKILL
session not created: This version of ChromeDriver only supports Chrome version
chrome not reachable

Chrome auto-updates monthly; webdriver-manager's cached chromedriver version drifts. Self-healer detects this in /var/log/auction-intel/avibm.log and auto-fixes.

1. Self-healer auto-runs every cycle. Wait 1 healer tick.
2. Manual fix: rm -rf /home/auction/.wdm/drivers/chromedriver
3. Then: sudo -n /usr/bin/systemctl restart avibm-bot.service
4. Cooldown: 1 hour between auto-attempts (.avibm-chromedriver-heal-attempted flag)
5. ONLY applies to AVIBM (Selenium) — Velocicast uses Playwright, immune.

• app/services/self_healer.py:_heal_avibm_chromedriver_drift
• /home/auction/.wdm/drivers/chromedriver
• /etc/sudoers.d/auction-intel-healer

• AVIBM bot stuck retrying same vehicle
• Update Booking flow shows 'check CRN/email' error

WOVI: please verify CRN and email

WOVI sometimes silently fails to deliver the confirmation email. Bot now detects this, escalates to customer email after 2 fails, and pauses vehicle via search_after_date sentinel.

1. Check AVIBM admin panel — paused vehicles show search_after_date in future
2. Customer email escalation logged in vehicle row
3. Manual unpause: clear search_after_date in AVIBM DB
4. Ground truth: contact WOVI support if pattern repeats for one customer

• feedback_avibm_wovi_email_mismatch.md
• AVIBM admin panel

• [carsales-walker] make discovery HTTP 403
• Walker logs '0 listings' for many makes in a row

HTTP 403
make discovery HTTP 403 — using hardcoded list

Carsales aggressively bot-blocks. We use curl_cffi with chrome120/safari impersonation; bot-block is per-IP and rotates. Fall-through is normal — walker skips blocked makes and continues.

1. Confirm rate is < 100 listings/min — slow down if higher
2. Wait 1-2 hours for IP cooldown
3. Discovery falls back to hardcoded AU_CAR_MAKES list automatically
4. Don't promise antibot fixes — see feedback_no_guarantees_antibot.md

• app/scrapers/market/carsales.py
• scripts/scrape_carsales.py

• Grays scrape finishes with 0 listings
• ScrapeLog status 'completed' but new_count=0

[grays] no lots found on category page

Grays uses Algolia-backed Next.js. We extract __NEXT_DATA__ from the category page. If they redesign the page or change the JSON shape, the parser silently returns empty.

1. Open https://www.grays.com/search/automotive-trucks-and-marine/motor-vehicles in browser
2. Inspect DOM: should have <script id='__NEXT_DATA__'>
3. If shape changed: update GraysScraper.parse_search_page
4. All 7 categories scraped: salvage + 7 NS subcategories

• app/scrapers/auction/grays/scraper.py
• app/scrapers/auction/grays/monitor.py

• Need to verify or debug IAAI hammer-price capture
• Comparing what each source contributed

IAAI live auctions send hammer prices to the AuctionNow UI but DO NOT populate `HighPrebidValue` on the public search endpoint. Single-source capture (public endpoint only) caught ~47% of hammers on sale 209 (2026-05-04 / scrape_log 1784) — 130 sold, 62 with prices. Architecture (live as of 2026-05-04): • monitor_iaai_auction (auction_monitor.py) — public search endpoint, 8s poll, persists status flips + pre-bid values. Backstop for the cases where AuctionNow misses a lot. • capture_auctionnow (auctionnow_capture.py) — Playwright on the AuctionNow portal, 2s poll, dual-source within itself: - on-the-block panel → tracks current stock_number per lane - messages feed → "Run #N: Sold for $X" announcements Pairs the messages-feed run number to the on-the-block stock_number captured the previous poll (before the price wipe). Run numbers repeat across lanes (per feedback_iaai_lane_capture.md), so we match by stock_number which is globally unique. Both run in parallel under iaai_session_service._runner via asyncio.gather. AuctionNow's on_lot_sold writes to listing.sold_price via the same on_lot_sold callback used by the public monitor. Last- write-wins on price, but AuctionNow polls 4x faster + has the actual hammer, so it usually wins.

1. Verify both ran for an auction: SELECT raw_data FROM scrape_log WHERE source='iaai' AND raw_data::jsonb ? 'auctionnow' ORDER BY id DESC LIMIT 1; — auctionnow.total_captured / auctionnow.sold should be set
2. Compare hit rates: SELECT COUNT(*) FILTER (WHERE sold_price IS NOT NULL)::float / NULLIF(COUNT(*),0) FROM auction_listings WHERE source='iaai' AND raw_data::jsonb->>'AuctionId'=<id> AND status='sold'
3. If AuctionNow logs show 0 lanes parsed: DOM selectors in _DOM_SCRIPT need updating (IAAI portal markup evolves). Run sudo -u auction xvfb-run -a .venv/bin/python -c '...' against a live auction URL and inspect document.querySelectorAll output
4. Per-lane fallback: if lane matching fails (matched_lane=''), the price will still be captured but stock_number is null — listing won't get its sold_price set. Look at scrape_log auctionnow.captures[*].lane to see how often this happens
5. DOM selectors are in _DOM_SCRIPT (Playwright page.evaluate). Update there if portal markup changes

• app/scrapers/auction/iaai/auctionnow_capture.py
• app/scrapers/auction/iaai/auction_monitor.py
• app/services/iaai_session_service.py

• raw_data.last_poll_at older than 5 min on active IAAI session
• No new hammers captured during running auction

[iaai-monitor] heartbeat stale
[iaai-watcher] restarting hung monitor

IAAI AuctionNow Playwright session can silently hang. Watcher detects stale last_poll_at (>5 min), cancels the asyncio task, and restarts. Auto-recovers in ~6 min.

1. Wait 6 min — watcher auto-restarts
2. If still hung after 10 min: systemctl restart auction-intel
3. Check Playwright browser is alive: pgrep -f 'iaai_monitor'

• app/scrapers/auction/iaai/monitor.py
• app/services/iaai_watcher.py

• IAAI auction reports N lots sold but DB has sold_price NULL on half of them
• scrape_log raw_data.result.total_sold > listings with sold_price set

Per feedback_iaai_prices_lost_on_close.md, IAAI's "on the block" panel clears the hammer price the moment a lot sells. The bid scraper has only the time between when a lot becomes the block and when it gets cleared to read the price. Default poll_interval_s on IAAI was 120 seconds — IAAI auctions sell multiple lots per minute, so most hammer prices were gone by the next poll. The "messages" feed (last 2 sold per lane) backstopped some captures, recovering ~50% of prices. Fixed 2026-05-04: dropped poll_interval_s 120 → 8 seconds in iaai_session_service.py. Next IAAI live auction should capture near 100%.

1. Verify: grep poll_interval_s app/services/iaai_session_service.py — should show 8
2. If prices still missing: drop further to 5s (IAAI doesn't appear to throttle aggressively)
3. Compare runs: SELECT COUNT(*) FILTER (WHERE sold_price IS NOT NULL) / COUNT(*)::float AS hit_rate FROM auction_listings WHERE source='iaai' AND raw_data::jsonb->>'AuctionId'='<id>' AND status='sold'
4. Today's already-finished auctions can't be backfilled — IAAI doesn't expose post-auction prices

• app/services/iaai_session_service.py
• app/scrapers/auction/iaai/auction_monitor.py
• feedback_iaai_prices_lost_on_close.md

• /scrapes/ shows manheim_industrial firing repeatedly within hours
• Same sale_number processed multiple times

[manheim] session X already scraped — skip

Bulk scraper stores ScrapeLog with source='manheim_industrial' but the per-session dedup guard was filtering on source=='manheim'. Mismatch made dedup miss every time. Fixed via _source_for_division helper.

1. Confirm fix is in place: grep 'source.in_((' app/services/manheim_session_service.py
2. Should see: ScrapeLog.source.in_(('manheim','manheim_ns','manheim_industrial'))
3. If missing: re-apply the patch from session digest

• app/services/manheim_session_service.py

• [manheim_simulcast] PASA058 initial enrichment: 0/N upserted
• [manheim_timed] NAD05 initial enrichment: 0/N upserted
• All N lots fail upsert with the same error
• scrape_log.listings_updated stays at 0 throughout the monitor run
• Live auction completes with no bids captured

manheim_simulcast.*upsert failed.*Object of type datetime is not JSON serializable
manheim_timed.*upsert failed.*Object of type datetime is not JSON serializable
TypeError: Object of type datetime is not JSON serializable
manheim_simulcast.*0/[0-9]+ upserted
manheim_timed.*0/[0-9]+ upserted

Both manheim_session_service.py upsert paths (simulcast around line 728, timed around line 1005) build `raw_data._all_fields = extra` where `extra` is the parse_detail() output. `extra` contains datetime fields (e.g. last_seen_at). raw_data is a JSON column → SQLAlchemy serializes via stdlib json.dumps which doesn't know about datetime → entire upsert fails. The exception is caught + logged at WARNING but the listing row never gets enriched, so the monitor sits there polling 0 active lots and reports "X listings_updated=0" forever. Why it didn't break on the Mac: the bug was probably present on Mac too but either (a) raw_data wasn't being serialized in the same code path, or (b) the lots that hit this path didn't include datetime fields in extra. On VPS the data shape changed slightly post-migration and the bug surfaced for every Manheim live + timed run.

1. Verify: grep '_json_safe' app/services/manheim_session_service.py — should show 2 functions (one in each path, ~lines 728 + 1005)
2. If missing: re-apply by adding a recursive _json_safe() helper that converts datetime → isoformat() and assigns raw['_all_fields'] = _json_safe(extra); item['raw_data'] = _json_safe(raw)
3. Restart auction-intel + reset orphan sessions to retrigger watcher: UPDATE auction_sessions SET status='scheduled' WHERE status='monitoring' AND id IN (<affected sessions>)
4. Confirm next monitor run: scrape_log.listings_updated should reflect actual lot count, no TypeError in logs

• app/services/manheim_session_service.py
• function _json_safe — defined inline in both upsert paths

• Listing shows auction_type='timed_online' but session is 'bid_live'
• Wrong scraper path picked for sweep

Pickles listings carry their own auction_type field that can disagree with the session. SESSION IS SOURCE OF TRUTH. Drift causes wrong scrape path on sweeps.

1. See feedback_listing_auction_type_drift.md for bulk-sync SQL
2. On scrape: use auction_session.auction_type, not listing.auction_type
3. After bulk-sync: re-run downstream sweeps

• feedback_listing_auction_type_drift.md

• download_pickles_images.py log shows 'no new listings'
• Run history identical photo counts day-over-day

[pickles-images] no missing photos

Script downloads photos for listings WITHOUT photos. If main scraper already pulled them inline, this is a no-op — which is correct. Not a bug.

1. Don't 'fix' if everything has photos — that's success
2. If a specific listing is missing photos: check listing.photos JSONB
3. Force re-download for one listing: scripts/redownload_listing_photos.py <id>

• scripts/download_pickles_images.py

• Upcoming sales page shows past dates
• Auction monitor never fires for a session

sync_upcoming_sessions reads the sale-info page (authoritative). Concluded sales sometimes show stale start times. MUST keep past-date guard or scheduler tries to monitor sessions that already happened.

1. Confirm past-date guard exists in pickles_session_service.sync_upcoming_sessions
2. Force re-sync: from app.services.pickles_session_service import sync_upcoming_sessions; await sync_upcoming_sessions()
3. Sale-info page is overwritten every 30 min in normal operation

• app/services/pickles_session_service.py
• feedback_pickles_sale_info_authoritative.md

• Listings present yesterday gone today with no SOLD status

Unsold Pickles lots roll to a follow-up sale with NEW sale_number. If our monitor doesn't follow rollovers, lots vanish. Bid Live hook on ended sale routes to rollover event while it's live.

1. Verify rollover follower runs in scheduler
2. Manually check on Pickles website: search lot title — usually rolled to next sale
3. See reference_pickles_rollover.md

• app/services/pickles_session_service.py
• reference_pickles_rollover.md

• Searching the runbook for an R-XXXXXX code returns nothing
• Confusion about whether R-C292C0 is a vehicle or a scrape run

Two different systems use the SAME R-<6-hex> format: • Vehicle refs (project_vehicle_refs.md): R = Caravan/RV/Motorhome. Hash of VIN|colour|make|model|year. Stable per vehicle. • Scrape-run refs (app/models/scrape_log.py): R = scrape Run. Hash of run|log_id|source|started_at. Different on every scrape. When you see R-XXXXXX in self_healer log lines, it's the scrape-run ref. When you see it on /vehicle/<id> pages or in vehicle taxonomy, it's the vehicle ref.

1. Context tells you which: log line with 'self_healer' = run ref; vehicle page = vehicle ref
2. Run ref → look up by ID: SELECT * FROM scrape_log WHERE id=<id-derived-from-context>;
3. Vehicle ref → SELECT * FROM vehicles WHERE ref='R-XXXXXX';
4. Future cleanup: rename run-ref prefix to S- (Scrape) or J- (Job) to remove collision

• app/models/scrape_log.py:ref
• project_vehicle_refs.md

• Same row's [permanent error] log line keeps printing every 5 min
• scrape_log.raw_data->heal is NULL despite logs claiming it was set
• Earlier rows in the same healer cycle had a SQLAlchemy parse error

permanent error [parse]
This Session's transaction has been rolled back due to a previous exception during flush
InvalidRequestError

Self-healer iterates all failed scrape_log rows and tries to write a heal-status mark on each. If ONE row triggers a flush exception (e.g. attempts to write a value violating a constraint), the AsyncSession enters rollback-only state. Every subsequent db.commit() in that cycle fails to persist — but the loop keeps going and emits 'first-time permanent error' log lines that imply the mark was set. Next healer tick, same rows look unmarked, same logs print again.

1. Refactor _try_heal to use a fresh session per row, OR catch InvalidRequestError + db.rollback() before continuing
2. Until that's fixed, manually mark the affected row to silence the loop:
3. UPDATE scrape_log SET raw_data = jsonb_set(coalesce(raw_data::jsonb,'{}'::jsonb), '{heal}', '{"status":"permanent","category":"watchdog"}'::jsonb) WHERE id=<id>;
4. Check no constraint violations are queued: tail server.log for 'IntegrityError' or 'StringDataRightTruncation'

• app/services/self_healer.py:_try_heal
• app/db/session.py

• /auctions/ returns 500 for some requests (often googlebot)
• Listings page server-side render fails

jinja2.exceptions.TemplateAssertionError: No filter named 'source_label'
No filter named 'source_label'
Exception in ASGI application

Jinja filters are registered per-Templates instance. Each router that creates its own `Jinja2Templates(directory=...)` must call `register_source_filter(templates)` if any of the templates it renders uses `| source_label`. Auctions router was missing this call; the filter only got picked up the first time another router (rules, runbook) was hit, then cached — race-condition'd 500s for /auctions/.

1. Add to the router file's import + setup block:
2. from app.services.source_label import register_source_filter
3. register_source_filter(templates)
4. Templates that use the filter: auctions.html, scrape_history.html, scrape_listings.html, scrape_status.html
5. Verify all routers handling those templates register the filter:
6. grep -L register_source_filter $(grep -lE 'Jinja2Templates' app/routers/*.py)

• app/services/source_label.py
• app/routers/auctions.py

• 0 hammers captured for a Pickles Bid Live session that just ended

[velocicast] 0 lots returned

When a Pickles Bid Live sale closes, Velocicast purges the tile. Data is GONE FOREVER. One 30-min retry max, then permanent skip. Don't add long retry cooldowns — see feedback_velocicast_gone_means_gone.md

1. Confirm hammer monitor was running during the auction window
2. Check raw_data.last_poll_at on the auction_session row
3. Future: ensure capture_velocicast_prices runs every 60s during auction
4. Accept the loss — there's no recovery path

• app/services/velocicast_capture.py
• feedback_velocicast_gone_means_gone.md

• Velocicast dump file is fat (real data inside) but DB unchanged
• No new hammer rows after a known-good capture

ValueError: invalid literal for int() with base 10: '115A'

Some Pickles lot numbers carry alphabetic suffixes (e.g. '115A'). save_velocicast_results used int(lotNumber) which throws on those. Fixed with regex digit extract.

1. Replay dumps from data/velocicast/dumps/<sale>.json into save_velocicast_results
2. Confirm fix uses re.search(r'\d+', lotNumber).group()
3. Don't drop suffix lots — they're real listings

• app/services/velocicast_capture.py:save_velocicast_results
• data/velocicast/dumps/

• ScrapeLog row stuck in 'running' for >2 hours
• Self-healer flags it [watchdog] every 5 min
• Same R-XXXXXX run-ref repeating in logs

[self_healer] R-XXXXXX (<source>): permanent error [watchdog]
Watchdog: marked as failed after 125min with no completion
Likely app crash or unhandled exception

A scrape subprocess (or in-process job) started, never wrote a completion record to scrape_log, and the watchdog's 125-min ceiling kicked in. Common causes: process killed by 23:00 daily restart while mid-scrape, OOM, network stall on long Algolia/HTTP fetch, deadlock on Postgres pool, or unhandled exception that didn't propagate to the cleanup handler.

1. Inspect the row: SELECT id, source, started_at, raw_data FROM scrape_log WHERE id=<id>;
2. Check scrape subprocess log if any: ls /opt/auction-intel/logs/scrape_<id>.log
3. If the subprocess log shows a traceback: that's your root cause — fix and re-run
4. If no log: scrape ran in-process; check journalctl -u auction-intel for that timeframe
5. If it's noisy (re-firing every 5 min): see 'Self-healer commit silently fails — session rolled back'
6. Manual close: UPDATE scrape_log SET status='error', error_message='manually closed' WHERE id=<id>;
7. Manual re-trigger of that source: /scrapes/ page > the source > Run Now

• app/services/self_healer.py
• /opt/auction-intel/logs/scrape_<id>.log

• AVIBM admin Save fails to persist
• Tempted to disable RLS as 'fix'

permission denied for table bot_settings

Bot + every Next.js route use service_role (which bypasses RLS). RLS being enabled is fine — don't disable. Original 2026-04-21 disable was misdiagnosis; real bug was uncommitted route file. Re-enabled 2026-04-29.

1. Verify Next.js routes use SUPABASE_SERVICE_ROLE_KEY, not anon key
2. Re-deploy AVIBM admin to Vercel after env-var changes
3. RLS stays ENABLED on all public.* tables
4. See feedback_avibm_rls_gotcha.md

• feedback_avibm_rls_gotcha.md
• AVIBM Vercel env vars

• After ~3 page refreshes, every endpoint hangs
• psql 'SELECT state FROM pg_stat_activity' shows many 'idle in transaction'
• Workers wedged

asyncpg.exceptions.TooManyConnectionsError
TimeoutError waiting for connection from pool

Some FastAPI handlers don't close their AsyncSession properly. Mitigated by 4-hourly auto-restart. Real fix is per-handler audit.

1. Immediate: systemctl restart auction-intel
2. Permanent fix needed: see project_auction_intel_db_session_leak.md
3. Audit handlers using AsyncSession without async context manager
4. Tighten 4-hourly restart cron if leak gets worse

• /etc/cron.d/auction-intel (4-hourly restart)
• app/db/session.py

• First request after ALTER TABLE fails with 500
• [scheduler] grays bid monitor failed: InvalidCachedStatementError
• Random asyncpg/sqlalchemy traceback with no obvious user-code root cause
• Subsequent requests succeed (dialect auto-recovers)

asyncpg.exceptions.InvalidCachedStatementError
cached statement plan is invalid due to a database schema or configuration change
SQLAlchemy asyncpg dialect will now invalidate all prepared caches
asyncpg/prepared_stmt.py
_prepare_and_execute
dialects/postgresql/asyncpg.py.*greenlet_spawn

After we ALTER TABLE on a column, asyncpg's prepared-statement cache holds the old plan. Next query fails once with InvalidCachedStatementError. SQLAlchemy's asyncpg dialect catches this and invalidates ALL caches — second attempt succeeds. So it's self-recovering but emits one ugly traceback per affected connection. Stack frames in asyncpg/prepared_stmt.py or sqlalchemy/dialects/postgresql/asyncpg.py with no clear user-code root usually mean this family.

1. Nothing to do — it self-recovers on the next query
2. If recurring beyond a few minutes after a migration: systemctl restart auction-intel
3. Prevent: when running ALTER TABLE during business hours, plan for a brief restart after
4. If you want the FULL traceback of an instance: open the archive entry on /admin/runbook
5. Watch noise: grep InvalidCachedStatementError /var/log/auction-intel/server.log | wc -l

• /var/log/auction-intel/server.log

• Scheduled scrape ends with status=error after ~10 min of running
• Error message: 'This Session's transaction has been rolled back due to a previous exception during flush'
• Underlying cause: '(asyncpg) connection is closed' / InterfaceError
• Most often hits the longer-running scrapes (Manheim NS, Carsales, full Pickles)

asyncpg.exceptions._base.InterfaceError
connection is closed
Session's transaction has been rolled back due to a previous exception during flush
asyncpg.dialects.postgresql.asyncpg.InterfaceError

The scraper holds one AsyncSession open for the entire scrape run. Between DB writes it makes HTTP calls (search pages, lot details, bid history). If ANY gap between two DB ops exceeds Postgres's idle_in_transaction_session_timeout, PG kills the backend connection. Next db.commit() throws InterfaceError → SQLAlchemy session enters rollback-only state → every subsequent commit silently fails. Default PG cluster setting on this VPS was 10 min — Manheim NS scrape takes 11+ min, so it hit this every run. Bumped to 0 (disabled) on 2026-05-04 because our scraper workloads can legitimately hold long transactions and we have other safety nets (4-hourly auto-restart, per-row session in self_healer).

1. Verify cluster setting: sudo -u postgres psql -c 'SHOW idle_in_transaction_session_timeout;' (should be 0)
2. If set: sudo -u postgres psql -c "ALTER SYSTEM SET idle_in_transaction_session_timeout = 0;" && sudo -u postgres psql -c 'SELECT pg_reload_conf();'
3. Existing sessions retain old value — restart auction-intel for new setting to apply pool-wide
4. Long-term refactor: split scrapers into HTTP-fetch phase (no DB) + commit phase (short tx). Don't hold a session across HTTP calls.
5. If a scrape failed once: it auto-re-fires at the next scheduled slot (no manual catchup per feedback_dont_restart_during_scrape_window.md)
6. If you must catch up today's data: /scrapes/ → Run Now button on the source row

• /opt/auction-intel/app/services/auction_service.py:run_scraper
• /opt/auction-intel/app/database.py
• PG config: ALTER SYSTEM ... idle_in_transaction_session_timeout

• Detail page shows 'Part-Time' instead of 'Part-Time 4WD'
• Truncated values across many rows

value too long for type character varying(N)

Original schema had drive_type VARCHAR(10), transmission VARCHAR(30), body_type VARCHAR(50). New scrapers send longer values that get truncated (or rejected, depending on PG strictness).

1. ALTER TABLE auction_listings ALTER COLUMN drive_type TYPE VARCHAR(30);
2. ALTER TABLE auction_listings ALTER COLUMN transmission TYPE VARCHAR(80);
3. ALTER TABLE auction_listings ALTER COLUMN body_type TYPE VARCHAR(100);
4. Backfill truncated rows by re-scraping affected listings

• app/models/auction_listing.py

• VIN healer reports 0 recoveries despite many candidates
• IntegrityError: vehicles_vin_key

duplicate key value violates unique constraint "vehicles_vin_key"

_safe_set_vin tries to write a VIN that already exists on a different vehicle row. Original code threw on conflict. Fix: merge — repoint AuctionListing to the canonical vehicle row.

1. Confirm merge logic exists in vin_healer._safe_set_vin
2. Re-run healer: scripts/heal_vins.py
3. Last successful run recovered 645 VINs (2026-05-01)

• app/services/vin_healer.py
• scripts/heal_vins.py

• systemctl status auction-intel shows 'failed'
• Site returns 502

ModuleNotFoundError
ImportError
AttributeError at import time

Usually a Python import error in newly-deployed code. Service exits before binding port. Logs in journalctl reveal the traceback.

1. journalctl -u auction-intel -n 100 --no-pager
2. Look for the LAST traceback before 'failed'
3. Fix the import / syntax error
4. systemctl restart auction-intel
5. Verify: curl -sf http://127.0.0.1:8000/admin/health.json

• /etc/systemd/system/auction-intel.service
• /opt/auction-intel/.venv/bin/uvicorn

• Where is X on the VPS?

Reference for navigating the deployment.

1. Code: /opt/auction-intel
2. Venv: /opt/auction-intel/.venv/bin/python
3. Logs: /var/log/auction-intel/
4. Hot tier photos: /var/lib/auction-intel/static-hot/
5. Cold tier (Drive): /home/auction/gdrive/AI Projects/auction-intel/static/
6. Union mount: /opt/auction-intel/static/
7. Settings: /opt/auction-intel/data/settings.json
8. DB: postgres on localhost:5432, db 'auction_intel'
9. Service: systemctl status auction-intel
10. Scrape logs: /opt/auction-intel/logs/scrape_<id>.log

• /opt/auction-intel
• /var/log/auction-intel
• /var/lib/auction-intel

• Need to apply code changes
• Need to clear stuck state

Reference for safe restart order.

1. 1. systemctl restart auction-intel
2. 2. Wait 5s
3. 3. Verify: curl -sf http://127.0.0.1:8000/admin/health.json
4. 4. Check journalctl -u auction-intel -n 30 for clean startup
5. Auto-restart runs nightly at 23:00 ACST via /etc/cron.d/auction-intel
6. 4-hourly safety restart for connection-leak mitigation

• /etc/cron.d/auction-intel
• /etc/systemd/system/auction-intel.service

• df -h / shows ≥ 90%
• Service fails to write logs
• Photos failing to download

OSError: [Errno 28] No space left on device

Photos pile up on hot tier (/var/lib/auction-intel/static-hot) when tier mover is paused or behind. Logs also accumulate.

1. du -sh /var/lib/auction-intel/static-hot/* | sort -h | tail -10
2. Confirm tier mover not paused: ls /var/lib/auction-intel/.tier-mover-paused
3. Force tier mover: /opt/auction-intel/scripts/photo_tier_mover.sh
4. Check log sizes: du -sh /var/log/auction-intel/
5. Logs auto-rotate via /etc/logrotate.d/auction-intel (14-day retention)

• /var/lib/auction-intel/static-hot/
• /var/log/auction-intel/

• ps aux | grep python shows orphaned scrape subprocesses
• CPU pinned by stuck job

Manual scrape subprocess can outlive its parent if killed mid-flight. self_healer reaps zombies every cycle.

1. Self-healer auto-reaps zombies on next tick
2. Manual: pkill -f 'scripts/scrape_'
3. Verify no important jobs got killed: tail /opt/auction-intel/logs/scrape_*.log

• app/services/self_healer.py
• /opt/auction-intel/logs/scrape_*.log

• Need to give someone access to auction-intel.com

Reference procedure.

1. Cloudflare dash > Zero Trust > Access > Applications
2. Click auction-intel.com > Policies
3. Edit policy 'Allow' > add Email: <[email protected]>
4. Save. User then visits https://auction-intel.com and gets OTP/Google flow.
5. For Google OAuth: Zero Trust > Settings > Authentication > Add Login Method

• Cloudflare Zero Trust > Access

• User enters email, no OTP arrives
• Cloudflare Access logs empty for that user

Could be: email filter blocking [email protected], user typed wrong domain, or Access policy doesn't include them. We hit this with Faraz initially — incognito worked, eventually resolved.

1. User checks spam folder for [email protected]
2. Confirm Access policy includes user's email: Zero Trust > Access > Applications > auction-intel.com > Policies
3. Try Google OAuth identity provider instead of one-time-pin
4. Test in incognito to rule out cookies

• Cloudflare Zero Trust > Access > Applications

• Need tunnel name / target

Reference.

1. Tunnel runs as systemd service: cloudflared.service
2. Routes auction-intel.com -> http://127.0.0.1:8000 (local uvicorn)
3. Auth: Cloudflare Access (OTP or Google) for [email protected] + [email protected]
4. Tunnel name viewable: Zero Trust > Networks > Tunnels

• /etc/cloudflared/config.yml
• /etc/systemd/system/cloudflared.service

• auction-intel.com returns 502 / 521
• Cloudflare dash shows tunnel offline

cloudflared: connection refused

cloudflared process crashed or systemd unit failed.

1. systemctl status cloudflared
2. systemctl restart cloudflared
3. tail -f /var/log/cloudflared.log
4. Verify on Cloudflare dash: Zero Trust > Networks > Tunnels — tunnel HEALTHY

• /etc/systemd/system/cloudflared.service
• Cloudflare Zero Trust dash > Tunnels

• avibm.vercel.app/admin Save click does nothing
• DB row unchanged

Open bug. RLS fixed 2026-04-21, but Next.js API path still not writing. Likely Vercel env-var issue (SUPABASE_SERVICE_ROLE_KEY missing or stale).

1. Vercel dashboard > avibm project > Settings > Environment Variables
2. Confirm SUPABASE_SERVICE_ROLE_KEY is set for Production
3. Redeploy: Vercel > Deployments > Redeploy latest
4. Test Save again — check Network tab for 200 vs 500 on POST /api/...

• Vercel avibm project
• project_avibm_admin_save_broken.md

• Nav badge shows AVIBM stopped
• No new bookings being attempted

avibm-bot.service: Failed

Bot crashed (often chromedriver drift) or was manually stopped.

1. systemctl status avibm-bot
2. tail -100 /var/log/auction-intel/avibm.log
3. If SIGKILL: see avibm-chromedriver-sigkill entry
4. Manual restart: sudo systemctl restart avibm-bot.service
5. Or use the AVIBM control button in the auction-intel nav bar

• /etc/systemd/system/avibm-bot.service
• /var/log/auction-intel/avibm.log

• Where does AVIBM code live?

Reference.

1. VPS bot code: /opt/avibm
2. Bot service: /etc/systemd/system/avibm-bot.service
3. Bot log: /var/log/auction-intel/avibm.log
4. Web admin: avibm.vercel.app/admin (Next.js + Supabase)
5. Bot uses Selenium + chromedriver via webdriver-manager
6. Drivers cached at /home/auction/.wdm/drivers/
7. Full system context: project_avibm.md

• /opt/avibm
• project_avibm.md

• Detail page for IAAI listing shows 0 photos
• Listing dir under /opt/auction-intel/static/ exists but empty

Open issue (~30-50% missing). IAAI photo download path failing silently. Older queue item — see project_storage_anomalies_todo.md.

1. Confirm scope: scripts/audit_iaai_photos.py
2. Manual re-download for one listing: scripts/redownload_iaai_listing.py <id>
3. Root-cause investigation pending — needs new session

• project_storage_anomalies_todo.md

• Sample of iaai_<id>/photos shows 0 files
• Audit script reports 27%+ of active IAAI listings missing photos
• But the website renders photos fine

iaai photo missing
iaai photos failing to download

Photos live across two tiers — hot (/var/lib/auction-intel/static-hot/uploads) and cold (/home/auction/gdrive/AI Projects/auction-intel/static/uploads). They're unioned via mergerfs at /opt/auction-intel/static. Photos newer than ~14 days sit on the hot tier; photo_tier_mover.sh drains them to cold over time. An audit script that only walks ONE tier will report missing dirs that actually exist on the other tier. Always audit via the union mount. Also note: IAAI photo dir names use raw_data.StockId (IAAI internal numeric ID), NOT external_id (StockNum). Pattern: /static/uploads/iaai_<StockId>/photos/<n>.jpg

1. Always audit via the union: cd /opt/auction-intel/static/uploads
2. If a specific listing is genuinely missing photos: scripts/redownload_iaai_listing.py <id>
3. Confirm both mounts are alive: mount | grep -E 'mergerfs|gdrive' should show 2 lines
4. Last verified 2026-05-04: all 1886 active IAAI listings have photos via union

• /var/lib/auction-intel/static-hot/uploads
• /home/auction/gdrive/AI Projects/auction-intel/static/uploads
• /opt/auction-intel/static/uploads
• app/services/photo_tier_mover.sh

• About to write a cleanup script that deletes anything

2,382-listing photo deletion incident on 2026-04-15. Never delete photos / listings / DB rows / folders without explicit per-scope approval. Carsales is the ONLY exception (minimal shape: 1 thumb only).

1. Refuse the request and confirm scope first
2. List exactly what will be deleted (count, source, age range)
3. Only proceed after explicit user 'yes'
4. See feedback_never_delete_data.md and feedback_non_salvage_market_shape.md

• feedback_never_delete_data.md
• feedback_non_salvage_market_shape.md

• Wondering if Pickles photos are bloated
• Concern about Drive filling up

As of 2026-05-04: ~50 GB total Pickles photos (39 GB hot tier + 11 GB cold tier). ~7,134 active + ~9,500 historical listings. Average 24-26 photos per listing, 844x633 JPEG, ~80 KB each → ~2 MB per listing total. Growth rate is ~5 GB/week sustained. With 2 TB Drive and storage_failover auto-pause at 95%, we have 6+ months runway before any action needed. Per feedback_non_salvage_market_shape.md: NEVER trim or delete Pickles photos (or any auction-source photos). They're sellable archive data. Carsales is the only minimal-shape exception.

1. No action needed at current footprint
2. If Drive ever fills (95%): storage_failover RED state auto-pauses cold-tier writes — see storage-red-halted runbook
3. If you want exact current numbers: du -sh /var/lib/auction-intel/static-hot/uploads/pickles_* and du -sh '/home/auction/gdrive/AI Projects/auction-intel/static/uploads/pickles_*'
4. Long-term plan: upgrade Google One to 5 TB before Drive crosses ORANGE (95%) — currently at ~5%

• /var/lib/auction-intel/static-hot/uploads/pickles_*
• /home/auction/gdrive/AI Projects/auction-intel/static/uploads/pickles_*

• Thumbnail missing on a listing card

Only /thumbs/<source>_<id>.jpg is canonical. Legacy fix_hero_thumbnails.py is DISABLED. auction_service post-scrape spawn handles new thumbs; self_healer maps existing ones. Don't revive the old script.

1. Manual rebuild for one listing: python3 -c 'from app.services.thumb_generator import generate_thumb; generate_thumb(<id>)'
2. Bulk: don't run fix_hero_thumbnails.py — it's deliberately disabled
3. Self-healer maps any orphaned thumbnails on each cycle

• app/services/thumb_generator.py
• /var/lib/auction-intel/thumbs/
• feedback_thumbnail_system_migration.md

• Looking for a specific listing's photos

Reference.

1. Hot tier (recent): /var/lib/auction-intel/static-hot/<source>/<id>/
2. Cold tier (Drive): /home/auction/gdrive/AI Projects/auction-intel/static/<source>/<id>/
3. Union (read-through): /opt/auction-intel/static/<source>/<id>/
4. URL pattern: /static/<source>/<id>/<filename>.jpg
5. Tier mover runs every N min via /opt/auction-intel/scripts/photo_tier_mover.sh
6. Move criteria: listing > 30 days old AND not currently in active auction

• /opt/auction-intel/scripts/photo_tier_mover.sh

• /home/auction/gdrive/.../backups/ missing today's dump

pg_dump: error
rclone: failed to upload

Either pg_dump errored, or rclone Drive mount lost during write.

1. Check log: tail /var/log/auction-intel/db-backup.log
2. Run manually: /opt/auction-intel/scripts/db_backup.sh
3. If rclone-related: see storage-rclone-drive-unmounted
4. Step 4 also tars car_makes_models/ if changed (60-day retention)

• /opt/auction-intel/scripts/db_backup.sh
• /var/log/auction-intel/db-backup.log

• /var/log/auction-intel/git-autocommit.log shows 'PUSH FAILED — auth needed'

PUSH FAILED — auth needed
Permission denied (publickey)

SSH deploy key missing / expired, or GitHub repo deploy-key list out of date.

1. Test: sudo -u auction ssh -T [email protected]
2. If fails: regenerate ~auction/.ssh/id_ed25519, add public key to GitHub repo Deploy Keys (with write)
3. Manual run: sudo -u auction /opt/auction-intel/scripts/git_autocommit.sh
4. Cron: 23:30 ACST nightly via /etc/cron.d/auction-intel

• /opt/auction-intel/scripts/git_autocommit.sh
• /var/log/auction-intel/git-autocommit.log
• /home/auction/.ssh/

• Want to mirror something to local SSD

Pre-VPS pattern: LaunchAgent on user's Mac auto-synced to plugged-in SSD. Now data is VPS-only; SSD sync is mostly historical.

1. User leaving for 1-2 years overseas — don't depend on Mac sync
2. Backup chain is now: Postgres -> Drive (rclone) + code -> GitHub
3. If Mac sync is needed: ~/Library/LaunchAgents/com.auctionintel.ssdsync.plist

• session_2026_04_17_18_digest.md

• DB corruption / lost data / fresh VPS

Reference for restoring.

1. Drive backup path: /home/auction/gdrive/.../backups/db-YYYY-MM-DD.sql.gz
2. Restore: gunzip -c db-YYYY-MM-DD.sql.gz | psql auction_intel
3. car_makes_models: tar xzf car_makes_models-YYYY-MM-DD.tar.gz -C /opt/auction-intel/data/
4. Code: git clone <repo> + restore .env from password manager
5. Photos: read directly from /home/auction/gdrive/.../static/

• /home/auction/gdrive/AI Projects/auction-intel/backups/
• /opt/auction-intel/scripts/db_backup.sh

• Expected job missing in journalctl
• ScrapeLog has no entry for that slot

Either cron unit not loaded, file syntax error, or another job overlapped/killed it. Daily app-restart at 23:00 ACST will kill anything running through that minute.

1. systemctl status cron
2. sudo crontab -u auction -l (user crontab)
3. cat /etc/cron.d/auction-intel (system crontab)
4. journalctl -u cron -n 50
5. If conflict with restart: stagger the slot away from 23:00

• /etc/cron.d/auction-intel

• Daily slots fire but ScrapeLog shows 0 work done
• /scrapes/ history empty for hours

[scheduler] scrapers disabled — skip

settings.json has scrapers_enabled=False. Set during a RED storage event or manual toggle. Storage failover should auto-flip back to True on recovery, but a stuck flag is possible.

1. Check current state: jq .scrapers_enabled /opt/auction-intel/data/settings.json
2. Toggle on: from app.services.app_settings import set_scrapers_enabled; set_scrapers_enabled(True)
3. Or via /admin/health page if a button exists
4. Verify storage state isn't RED before re-enabling

• /opt/auction-intel/data/settings.json
• app/services/app_settings.py

• Time on schedule page differs from expected

All scheduled work uses Australia/Adelaide timezone (ACST = UTC+9:30, ACDT = UTC+10:30 during DST). Pages render local times via register_adelaide_filters Jinja filter.

1. Confirm: from zoneinfo import ZoneInfo; ZoneInfo('Australia/Adelaide')
2. Adelaide observes DST — ACST in winter, ACDT in summer
3. Code uses ZoneInfo, not pytz
4. Schedule view: /scrapes/ — Schedule ▾ dropdown shows fire times in ACST/ACDT

• app/services/scheduler.py
• app/services/tz_format.py

• What runs when?

Reference for the _DAILY scheduler list.

1. 00:00 Pickles (S)
2. 00:30 Pickles (NS)
3. 01:00 Manheim (S)
4. 01:30 Manheim (NS)
5. 02:00 Manheim (Ind)
6. 02:30 Pickles (Ind)
7. 03:00 Grays (NS)
8. 03:30 Grays (S)
9. 04:00 Carsales (full walk)
10. 04:30 IAAI
11. 23:00 service restart
12. 23:30 git auto-commit + push
13. + continuous: live monitors, storage-failover (every 30 min), self-healer

• app/services/scheduler.py
• /etc/cron.d/auction-intel

• apscheduler job 'Auction session sync (Pickles + Manheim, every 30m)' raises every 30 min
• Storage failover scheduler tick logs '[scheduler] storage failover failed'
• Same NameError text repeating with very high counts (39+ in a day)

NameError: name 'grays' is not defined
NameError: name 'state' is not defined
NameError: name 'drive_pct' is not defined
scheduler.py.*NameError

f-string log lines in scheduler.py used bare identifiers as dict keys (e.g. g.get(grays, 0) or result.get(state)) when they should have been string literals (g.get('grays', 0)). Python evaluates them as variables, throws NameError, the whole job aborts. Fixed 2026-05-03.

1. Quote the keys: g.get('grays', 0), result.get('state'), etc.
2. Verify with: cd /opt/auction-intel && .venv/bin/python -c "import ast; ast.parse(open('app/services/scheduler.py').read())"
3. systemctl restart auction-intel
4. Watch one cycle: journalctl -u auction-intel -f | grep '\[scheduler\]'

• app/services/scheduler.py:340 (storage state log)
• app/services/scheduler.py:542 (session sync log)

• [auction] <source> scraper was cancelled
• apscheduler 'Daily scrape — <source>' raised an exception (CancelledError downstream)
• Happens around the daily restart minute

scraper was cancelled
CancelledError
Daily scrape.*raised an exception
Daily scrape — 

When systemd restarts auction-intel, in-flight asyncio tasks get cancelled. If a scrape was running, asyncio.gather propagates the CancelledError as an exception. Cosmetic — the scrape will fire again at the next scheduled slot. This used to fire frequently because the daily restart was at 03:00 ACST, colliding with the 03:00 Grays NS slot. Restart was moved to 23:00 on 2026-05-03 to clear the conflict.

1. Verify the conflict is resolved: grep -E '(cron|daily restart)' /etc/cron.d/auction-intel
2. Should show: 0 23 * * * root /usr/bin/systemctl restart auction-intel.service
3. If a specific scrape didn't run because of cancellation: hit /scrapes/ > Run Now for that source
4. Don't run code-based catchup — APScheduler doesn't replay missed cron fires (per feedback_dont_restart_during_scrape_window.md)

• /etc/cron.d/auction-intel
• app/services/scheduler.py

• Need to know what Faraz can see/do

Reference for handover state.

1. Email: [email protected]
2. Cloudflare: full admin role (can manage tunnel, Access policies, DNS)
3. VPS VNC: full access (can run commands, restart services)
4. Cloudflare Access policy on auction-intel.com: includes his email
5. Storage alert recipient: gets all storage_failover transition emails

• Cloudflare account members
• Storage alert: app/services/storage_failover.py ALERT_EMAIL

• Key compromised / new admin

Reference procedure.

1. On new admin's machine: ssh-keygen -t ed25519
2. Copy public key: cat ~/.ssh/id_ed25519.pub
3. On VPS: append to /root/.ssh/authorized_keys (or /home/auction/.ssh/authorized_keys)
4. Test: ssh -i ~/.ssh/id_ed25519 [email protected]
5. If revoking old key: remove its line from authorized_keys

• /root/.ssh/authorized_keys
• /home/auction/.ssh/authorized_keys

• Self-healer can't restart avibm-bot

sudo: a password is required

Self-healer runs as 'auction' user, restarts avibm-bot.service via passwordless sudo. One specific command is granted via sudoers.d.

1. cat /etc/sudoers.d/auction-intel-healer
2. Should contain: auction ALL=(root) NOPASSWD: /usr/bin/systemctl restart avibm-bot.service
3. Permissions must be 0440
4. Test: sudo -u auction sudo -n /usr/bin/systemctl restart avibm-bot.service

• /etc/sudoers.d/auction-intel-healer