Sample log line
2026-05-04 12:53:30,005 INFO self_healer — [self_healer] R-941178 (manheim_ns): permanent error [unknown] — 'Auto-healed: no heartbeat for >30min'
Recent occurrences
loading…
Searchable manual covering every system we've built. Type anything — error code, log message, scraper name, or just a keyword.
2026-05-04 12:53:30,005 INFO self_healer — [self_healer] R-941178 (manheim_ns): permanent error [unknown] — 'Auto-healed: no heartbeat for >30min'
loading…
2026-05-04 14:12:52,454 INFO self_healer — [self_healer] R-175EE8 (manheim_industrial): permanent error [unknown] — 'Auto-healed: no heartbeat for >30min'
loading…
2026-05-06 11:22:19,785 INFO self_healer — [self_healer] R-DF8CAB (iaai): permanent error [unknown] — 'Auto-healed: no heartbeat for >30min'
loading…
2026-05-06 00:13:07,197 INFO self_healer — [self_healer] R-1B2674 (pickles): permanent error [unknown] — 'Cancelled by asyncio'
loading…
2026-05-04 13:57:52,434 INFO self_healer — [self_healer] R-BEE84D (manheim_industrial): permanent error [unknown] — 'killed by service restart 13:52'
loading…
2026-05-04 13:57:52,426 INFO self_healer — [self_healer] R-611E9A (manheim_ns): permanent error [unknown] — 'killed by service restart 13:52'
loading…
2026-05-04 13:57:52,406 INFO self_healer — [self_healer] R-57CF05 (manheim): permanent error [unknown] — 'killed by service restart 13:52'
loading…
2026-05-04 16:52:52,471 INFO self_healer — [self_healer] R-8784E3 (manheim): permanent error [unknown] — 'Auto-healed: no heartbeat for >30min'
loading…
2026-05-06 03:02:03,358 ERROR app.services.auction_service — [auction] Upsert failed for 0001-60050706: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
loading…
Traceback (most recent call last):
File "/opt/auction-intel/app/services/auction_service.py", line 1183, in run_scraper
listing_obj, is_new = await upsert_listing(upsert_db, item)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/auction_service.py", line 1028, in upsert_listing
listing.opportunity_score = await score_listing(db, listing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scoring_service.py", line 81, in score_listing
roi_score = await _roi_score(db, listing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scoring_service.py", line 213, in _roi_score
market = await get_market_summary(db, listing.vehicle_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/market_service.py", line 57, in get_market_summary
result = await db.execute(
^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2228, in _execute_internal
loading…
Traceback (most recent call last):
File "/opt/auction-intel/app/services/auction_service.py", line 1183, in run_scraper
listing_obj, is_new = await upsert_listing(upsert_db, item)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/auction_service.py", line 601, in upsert_listing
vehicle = await find_or_create_vehicle(db, data, existing_vehicle_id=existing.vehicle_id if existing else None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/vehicle_service.py", line 263, in find_or_create_vehicle
await db.flush()
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 787, in flush
await greenlet_spawn(self.sync_session.flush, objects=objects)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4331, in flush
self._flush(objects)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4466, in _flush
with util.safe_reraise():
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 121, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4427, in _flush
flush_context.execute()
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
rec.execute(self)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
util.preloaded.orm_persistence.save_obj(
loading…
Traceback (most recent call last):
File "/opt/auction-intel/app/services/auction_service.py", line 1183, in run_scraper
listing_obj, is_new = await upsert_listing(upsert_db, item)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/auction_service.py", line 1028, in upsert_listing
listing.opportunity_score = await score_listing(db, listing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scoring_service.py", line 153, in score_listing
relist_penalty = await _relist_penalty(db, listing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scoring_service.py", line 172, in _relist_penalty
r = await db.execute(
^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
result = context.switch(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2239, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2108, in _connection_for_bind
loading…
2026-05-07 10:51:51,127 ERROR apscheduler.executors.default — Job "Auto Pickles bid-history sweep (3m) (trigger: interval[0:03:00], next run at: 2026-05-07 10:54:44 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 1136, in _auto_bid_sweep
await _aio.gather(*[_one(*j) for j in jobs])
File "/opt/auction-intel/app/services/scheduler.py", line 1104, in _one
bids = await fetch_bid_history(client, sale, stock, asset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/scrapers/auction/pickles/timed/monitor.py", line 130, in fetch_bid_history
resp = await client.get(url, headers=_BID_HEADERS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1768, in get
return await self.request(
^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
loading…
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 420, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1163, in __call__
await super().__call__(scope, receive, send)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/applications.py", line 90, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 191, in __call__
with recv_stream, send_stream, collapse_excgroups():
File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
raise exc
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 193, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/main.py", line 92, in no_cache_html
response: Response = await call_next(request)
loading…
2026-05-06 16:14:49,818 ERROR app.services.iaai_session_service — [iaai_session] auctionnow crashed: cannot access local variable 'poll_count' where it is not associated with a value
Traceback (most recent call last):
File "/opt/auction-intel/app/services/iaai_session_service.py", line 287, in _auctionnow_monitor
return await capture_auctionnow(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/scrapers/auction/iaai/auctionnow_capture.py", line 353, in capture_auctionnow
f"({sold} sold, {referred} referred, {passed} passed) in {poll_count} polls"
^^^^^^^^^^
UnboundLocalError: cannot access local variable 'poll_count' where it is not associated with a value
loading…
2026-05-06 21:39:00,964 ERROR apscheduler.executors.default — Job "Auction session sync (Pickles + Manheim, every 30m) (trigger: interval[0:30:00], next run at: 2026-05-06 22:08:38 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 563, in _sync_all_sessions
File "/opt/auction-intel/app/services/manheim_session_service.py", line 224, in sync_manheim_sessions
r2 = await db.execute(
^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2258, in _execute_internal
result = conn.execute(
^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
return connection._execute_clauseelement(
loading…
2026-05-05 23:50:58,830 ERROR apscheduler.executors.default — Job "Health checks (5m) (trigger: interval[0:05:00], next run at: 2026-05-05 23:55:29 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 1467, in _run_health_checks
results = await run_all_checks(db, include_live_probes=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/health.py", line 87, in run_all_checks
result = await _run_all_checks_uncached(db, include_live_probes=include_live_probes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/health.py", line 131, in _run_all_checks_uncached
results_by_name[name] = await _run_one(name, fn, sev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/health.py", line 112, in _run_one
r = await fn(db)
^^^^^^^^^^^^
File "/opt/auction-intel/app/services/health.py", line 808, in _session_listings_findable
cnt = await db.scalar(
^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 505, in scalar
return await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
loading…
2026-05-07 22:15:04,126 ERROR apscheduler.executors.default — Job "Auction session watcher (60s) (trigger: interval[0:01:00], next run at: 2026-05-07 22:16:03 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 752, in _watch_sessions
(l.status = 'running' AND l.started_at > now() - interval '4 hours')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/manheim_session_service.py", line 1254, in manheim_session_watcher
r = await db.execute(
^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2239, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2108, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
loading…
2026-05-07 10:38:52,591 ERROR apscheduler.executors.default — Job "Velocicast bid-live capture (5m) (trigger: interval[0:05:00], next run at: 2026-05-07 10:42:45 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 1270, in _run_velocicast_capture
batch_results = await capture_all_velocicast_sales(sales_input, headless=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/scrapers/auction/pickles/bidlive/capture.py", line 931, in capture_all_velocicast_sales
if not modal_visible:
^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/async_api/_generated.py", line 9045, in goto
await self._impl_obj.goto(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_page.py", line 552, in goto
return await self._main_frame.goto(**locals_to_params(locals()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 153, in goto
await self._channel.send(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 69, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 557, in wrap_api_call
return await cb()
^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 123, in _inner_send
done, _ = await asyncio.wait(
loading…
2026-05-08 00:00:54,834 ERROR app.scrapers.auction.pickles.scraper — [pickles] API request failed (skip=300):
loading…
2026-05-06 16:14:49,063 ERROR app.scrapers.auction.iaai.auctionnow_capture — [auctionnow] crashed: Page.goto: net::ERR_TOO_MANY_REDIRECTS at https://portal.auctionnow.iaai.com/Bid?j=%5B%7B%22T%22%3A%20%22iaaaus%22%2C%20%22Id%22%3A%20%22212L1%22%2C%20%22Lang%22%3A%20%22en-au%22%2C%20%22Timezone%22%3A%20%2210%3A00%3A00%22%2C%20%22CrId%22%3A%20null%7D%2C%20%7B%22T%22%3A%20%22iaaaus%22%2C%20%22Id%22%3A%20%22212L2%22%2C%20%22Lang%22%3A%20%22en-au%22%2C%20%22Timezone%22%3A%20%2210%3A00%3A00%22%2C%20%22CrId%22%3A%20null%7D%2C%20%7B%22T%22%3A%20%22iaaaus%22%2C%20%22Id%22%3A%20%22212L3%22%2C%20%22Lang%22%3A%20%22en-au%22%2C%20%22Timezone%22%3A%20%2210%3A00%3A00%22%2C%20%22CrId%22%3A%20null%7D%2C%20%7B%22T%22%3A%20%22iaaaus%22%2C%20%22Id%22%3A%20%22212L4%22%2C%20%22Lang%22%3A%20%22en-au%22%2C%20%22Timezone%22%3A%20%2210%3A00%3A00%22%2C%20%22CrId%22%3A%20null%7D%5D
loading…
2026-05-07 00:42:24,480 ERROR app.scrapers.auction.iaai.session_fetcher — [iaai_session] Playwright fetch failed: Page.goto: Timeout 30000ms exceeded.
loading…
Traceback (most recent call last):
File "/opt/auction-intel/app/scrapers/auction/iaai/session_fetcher.py", line 167, in fetch_iaai_upcoming
html = await _fetch_page_html()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/scrapers/auction/iaai/session_fetcher.py", line 112, in _fetch_page_html
await page.goto(AUCTIONS_URL, wait_until="domcontentloaded", timeout=30_000)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/async_api/_generated.py", line 9045, in goto
await self._impl_obj.goto(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_page.py", line 552, in goto
return await self._main_frame.goto(**locals_to_params(locals()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 153, in goto
await self._channel.send(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 69, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 559, in wrap_api_call
raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.TimeoutError: Page.goto: Timeout 30000ms exceeded.
loading…
2026-05-06 12:11:50,304 ERROR apscheduler.executors.default — Job "Velocicast bid-live capture (5m) (trigger: interval[0:05:00], next run at: 2026-05-06 12:12:19 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 1216, in _run_velocicast_capture
batch_results = await capture_all_velocicast_sales(sales_input, headless=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/scrapers/auction/pickles/bidlive/capture.py", line 957, in capture_all_velocicast_sales
await bid_btn.wait_for(state="visible", timeout=30000)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/async_api/_generated.py", line 17993, in wait_for
await self._impl_obj.wait_for(timeout=timeout, state=state)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_locator.py", line 710, in wait_for
await self._frame.wait_for_selector(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 369, in wait_for_selector
await self._channel.send(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 69, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 557, in wrap_api_call
return await cb()
^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 123, in _inner_send
done, _ = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/tasks.py", line 464, in wait
loading…
2026-05-04 12:10:44,983 ERROR apscheduler.executors.default — Job "Auction session sync (Pickles + Manheim, every 30m) (trigger: interval[0:30:00], next run at: 2026-05-04 12:38:55 ACST)" raised an exception
Traceback (most recent call last):
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/apscheduler/executors/base.py", line 181, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/scheduler.py", line 563, in _sync_all_sessions
m = await sync_manheim_sessions(db)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/app/services/manheim_session_service.py", line 224, in sync_manheim_sessions
r2 = await db.execute(
^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2258, in _execute_internal
result = conn.execute(
^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
loading…
PermissionError: [Errno 13] Permission denied: '/opt/avibm.log'
loading…
Traceback (most recent call last):
File "/opt/auction-intel/app/scrapers/auction/iaai/auctionnow_capture.py", line 241, in capture_auctionnow
await page.goto(portal_url, wait_until="domcontentloaded", timeout=60000)
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/async_api/_generated.py", line 9045, in goto
await self._impl_obj.goto(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_page.py", line 552, in goto
return await self._main_frame.goto(**locals_to_params(locals()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 153, in goto
await self._channel.send(
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 69, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/auction-intel/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 559, in wrap_api_call
raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
loading…
[storage-failover] state GREEN -> YELLOW
Cold-tier Drive crossed the 80% threshold. No system disruption — this is an early warning to upgrade Google One to a 5 TB plan within ~30 days before reaching ORANGE (95%).
[storage-failover] tier mover PAUSED [storage-failover] state YELLOW -> ORANGE
Drive cold tier crossed 95%. Storage failover paused photo_tier_mover.sh automatically — new photos stay on the VPS hot tier. Scrapers continue. If left, the VPS will fill in a few weeks and trip RED.
Transport endpoint is not connected
mergerfs unifies /var/lib/auction-intel/static-hot/ + /home/auction/gdrive/AI Projects/auction-intel/static/. If rclone (Drive) is unmounted, mergerfs degrades silently and shows only hot tier.
[skip] tier mover paused
/var/lib/auction-intel/.tier-mover-paused was set during ORANGE/RED and never auto-cleared (e.g. storage_failover never ran while Drive recovered).
fuse: bad mount point rclone: token expired
FUSE mount can drop after long uptime, network blip, or expired OAuth token.
[storage-failover] scrapers_enabled = False [storage-failover] state ORANGE -> RED
Drive ≥95% AND VPS hot tier ≥90%. Both safety valves exhausted. Storage failover automatically set scrapers_enabled=False to prevent the VPS from running out of disk.
4-state machine: GREEN < 80% | YELLOW 80-95% | ORANGE ≥95% | RED Drive≥95% AND VPS≥90%. Recovery uses hysteresis (drop below DRIVE_RECOVER=90 / VPS_RECOVER=80). Email alerts fire on transitions only.
app/services/storage_policy.py only handles `status == "sold"` (the `_cleanup_sold_statutory` rule). The other terminal statuses do NOT trigger screenshot/photo cleanup: - expired (Manheim 241, Pickles 19, IAAI 631 as of 2026-05-04) - deleted (Carsales 1203, Manheim 3, Pickles 5) - superseded (Grays 1) - relisted (Pickles 1563, Manheim 70, IAAI 6) — should NOT be cleaned, may revert - referred (Pickles 179, IAAI 11) — should NOT be cleaned, may revert Net effect: ~2,000+ historical listings never have their screenshots reclaimed. At ~200 KB per screenshot, that's ~400 MB sitting forever. Not a crisis but worth fixing for long-term unattended operation.
_run_grays_monitor monitor_grays_closing grays bid monitor failed
The Grays bid-monitor 60s sweep wraps `monitor_grays_closing()` in a try/except and logs at WARN level. The actual root cause is whatever the inner await blew up on — usually transient (cached-statement after a migration, a brief connection pool blip, etc.). One-off failures auto-recover on the next 60s tick.
signal -9 killed with signal -9 SIGKILL session not created: This version of ChromeDriver only supports Chrome version chrome not reachable
Chrome auto-updates monthly; webdriver-manager's cached chromedriver version drifts. Self-healer detects this in /var/log/auction-intel/avibm.log and auto-fixes.
WOVI: please verify CRN and email
WOVI sometimes silently fails to deliver the confirmation email. Bot now detects this, escalates to customer email after 2 fails, and pauses vehicle via search_after_date sentinel.
HTTP 403 make discovery HTTP 403 — using hardcoded list
Carsales aggressively bot-blocks. We use curl_cffi with chrome120/safari impersonation; bot-block is per-IP and rotates. Fall-through is normal — walker skips blocked makes and continues.
[grays] no lots found on category page
Grays uses Algolia-backed Next.js. We extract __NEXT_DATA__ from the category page. If they redesign the page or change the JSON shape, the parser silently returns empty.
IAAI live auctions send hammer prices to the AuctionNow UI but DO NOT populate `HighPrebidValue` on the public search endpoint. Single-source capture (public endpoint only) caught ~47% of hammers on sale 209 (2026-05-04 / scrape_log 1784) — 130 sold, 62 with prices. Architecture (live as of 2026-05-04): • monitor_iaai_auction (auction_monitor.py) — public search endpoint, 8s poll, persists status flips + pre-bid values. Backstop for the cases where AuctionNow misses a lot. • capture_auctionnow (auctionnow_capture.py) — Playwright on the AuctionNow portal, 2s poll, dual-source within itself: - on-the-block panel → tracks current stock_number per lane - messages feed → "Run #N: Sold for $X" announcements Pairs the messages-feed run number to the on-the-block stock_number captured the previous poll (before the price wipe). Run numbers repeat across lanes (per feedback_iaai_lane_capture.md), so we match by stock_number which is globally unique. Both run in parallel under iaai_session_service._runner via asyncio.gather. AuctionNow's on_lot_sold writes to listing.sold_price via the same on_lot_sold callback used by the public monitor. Last- write-wins on price, but AuctionNow polls 4x faster + has the actual hammer, so it usually wins.
[iaai-monitor] heartbeat stale [iaai-watcher] restarting hung monitor
IAAI AuctionNow Playwright session can silently hang. Watcher detects stale last_poll_at (>5 min), cancels the asyncio task, and restarts. Auto-recovers in ~6 min.
Per feedback_iaai_prices_lost_on_close.md, IAAI's "on the block" panel clears the hammer price the moment a lot sells. The bid scraper has only the time between when a lot becomes the block and when it gets cleared to read the price. Default poll_interval_s on IAAI was 120 seconds — IAAI auctions sell multiple lots per minute, so most hammer prices were gone by the next poll. The "messages" feed (last 2 sold per lane) backstopped some captures, recovering ~50% of prices. Fixed 2026-05-04: dropped poll_interval_s 120 → 8 seconds in iaai_session_service.py. Next IAAI live auction should capture near 100%.
[manheim] session X already scraped — skip
Bulk scraper stores ScrapeLog with source='manheim_industrial' but the per-session dedup guard was filtering on source=='manheim'. Mismatch made dedup miss every time. Fixed via _source_for_division helper.
manheim_simulcast.*upsert failed.*Object of type datetime is not JSON serializable manheim_timed.*upsert failed.*Object of type datetime is not JSON serializable TypeError: Object of type datetime is not JSON serializable manheim_simulcast.*0/[0-9]+ upserted manheim_timed.*0/[0-9]+ upserted
Both manheim_session_service.py upsert paths (simulcast around line 728, timed around line 1005) build `raw_data._all_fields = extra` where `extra` is the parse_detail() output. `extra` contains datetime fields (e.g. last_seen_at). raw_data is a JSON column → SQLAlchemy serializes via stdlib json.dumps which doesn't know about datetime → entire upsert fails. The exception is caught + logged at WARNING but the listing row never gets enriched, so the monitor sits there polling 0 active lots and reports "X listings_updated=0" forever. Why it didn't break on the Mac: the bug was probably present on Mac too but either (a) raw_data wasn't being serialized in the same code path, or (b) the lots that hit this path didn't include datetime fields in extra. On VPS the data shape changed slightly post-migration and the bug surfaced for every Manheim live + timed run.
Pickles listings carry their own auction_type field that can disagree with the session. SESSION IS SOURCE OF TRUTH. Drift causes wrong scrape path on sweeps.
[pickles-images] no missing photos
Script downloads photos for listings WITHOUT photos. If main scraper already pulled them inline, this is a no-op — which is correct. Not a bug.
sync_upcoming_sessions reads the sale-info page (authoritative). Concluded sales sometimes show stale start times. MUST keep past-date guard or scheduler tries to monitor sessions that already happened.
Unsold Pickles lots roll to a follow-up sale with NEW sale_number. If our monitor doesn't follow rollovers, lots vanish. Bid Live hook on ended sale routes to rollover event while it's live.
Two different systems use the SAME R-<6-hex> format: • Vehicle refs (project_vehicle_refs.md): R = Caravan/RV/Motorhome. Hash of VIN|colour|make|model|year. Stable per vehicle. • Scrape-run refs (app/models/scrape_log.py): R = scrape Run. Hash of run|log_id|source|started_at. Different on every scrape. When you see R-XXXXXX in self_healer log lines, it's the scrape-run ref. When you see it on /vehicle/<id> pages or in vehicle taxonomy, it's the vehicle ref.
permanent error [parse] This Session's transaction has been rolled back due to a previous exception during flush InvalidRequestError
Self-healer iterates all failed scrape_log rows and tries to write a heal-status mark on each. If ONE row triggers a flush exception (e.g. attempts to write a value violating a constraint), the AsyncSession enters rollback-only state. Every subsequent db.commit() in that cycle fails to persist — but the loop keeps going and emits 'first-time permanent error' log lines that imply the mark was set. Next healer tick, same rows look unmarked, same logs print again.
jinja2.exceptions.TemplateAssertionError: No filter named 'source_label' No filter named 'source_label' Exception in ASGI application
Jinja filters are registered per-Templates instance. Each router that creates its own `Jinja2Templates(directory=...)` must call `register_source_filter(templates)` if any of the templates it renders uses `| source_label`. Auctions router was missing this call; the filter only got picked up the first time another router (rules, runbook) was hit, then cached — race-condition'd 500s for /auctions/.
[velocicast] 0 lots returned
When a Pickles Bid Live sale closes, Velocicast purges the tile. Data is GONE FOREVER. One 30-min retry max, then permanent skip. Don't add long retry cooldowns — see feedback_velocicast_gone_means_gone.md
ValueError: invalid literal for int() with base 10: '115A'
Some Pickles lot numbers carry alphabetic suffixes (e.g. '115A'). save_velocicast_results used int(lotNumber) which throws on those. Fixed with regex digit extract.
[self_healer] R-XXXXXX (<source>): permanent error [watchdog] Watchdog: marked as failed after 125min with no completion Likely app crash or unhandled exception
A scrape subprocess (or in-process job) started, never wrote a completion record to scrape_log, and the watchdog's 125-min ceiling kicked in. Common causes: process killed by 23:00 daily restart while mid-scrape, OOM, network stall on long Algolia/HTTP fetch, deadlock on Postgres pool, or unhandled exception that didn't propagate to the cleanup handler.
permission denied for table bot_settings
Bot + every Next.js route use service_role (which bypasses RLS). RLS being enabled is fine — don't disable. Original 2026-04-21 disable was misdiagnosis; real bug was uncommitted route file. Re-enabled 2026-04-29.
asyncpg.exceptions.TooManyConnectionsError TimeoutError waiting for connection from pool
Some FastAPI handlers don't close their AsyncSession properly. Mitigated by 4-hourly auto-restart. Real fix is per-handler audit.
asyncpg.exceptions.InvalidCachedStatementError cached statement plan is invalid due to a database schema or configuration change SQLAlchemy asyncpg dialect will now invalidate all prepared caches asyncpg/prepared_stmt.py _prepare_and_execute dialects/postgresql/asyncpg.py.*greenlet_spawn
After we ALTER TABLE on a column, asyncpg's prepared-statement cache holds the old plan. Next query fails once with InvalidCachedStatementError. SQLAlchemy's asyncpg dialect catches this and invalidates ALL caches — second attempt succeeds. So it's self-recovering but emits one ugly traceback per affected connection. Stack frames in asyncpg/prepared_stmt.py or sqlalchemy/dialects/postgresql/asyncpg.py with no clear user-code root usually mean this family.
asyncpg.exceptions._base.InterfaceError connection is closed Session's transaction has been rolled back due to a previous exception during flush asyncpg.dialects.postgresql.asyncpg.InterfaceError
The scraper holds one AsyncSession open for the entire scrape run. Between DB writes it makes HTTP calls (search pages, lot details, bid history). If ANY gap between two DB ops exceeds Postgres's idle_in_transaction_session_timeout, PG kills the backend connection. Next db.commit() throws InterfaceError → SQLAlchemy session enters rollback-only state → every subsequent commit silently fails. Default PG cluster setting on this VPS was 10 min — Manheim NS scrape takes 11+ min, so it hit this every run. Bumped to 0 (disabled) on 2026-05-04 because our scraper workloads can legitimately hold long transactions and we have other safety nets (4-hourly auto-restart, per-row session in self_healer).
value too long for type character varying(N)
Original schema had drive_type VARCHAR(10), transmission VARCHAR(30), body_type VARCHAR(50). New scrapers send longer values that get truncated (or rejected, depending on PG strictness).
duplicate key value violates unique constraint "vehicles_vin_key"
_safe_set_vin tries to write a VIN that already exists on a different vehicle row. Original code threw on conflict. Fix: merge — repoint AuctionListing to the canonical vehicle row.
ModuleNotFoundError ImportError AttributeError at import time
Usually a Python import error in newly-deployed code. Service exits before binding port. Logs in journalctl reveal the traceback.
Reference for navigating the deployment.
Reference for safe restart order.
OSError: [Errno 28] No space left on device
Photos pile up on hot tier (/var/lib/auction-intel/static-hot) when tier mover is paused or behind. Logs also accumulate.
Manual scrape subprocess can outlive its parent if killed mid-flight. self_healer reaps zombies every cycle.
Reference procedure.
Could be: email filter blocking [email protected], user typed wrong domain, or Access policy doesn't include them. We hit this with Faraz initially — incognito worked, eventually resolved.
Reference.
cloudflared: connection refused
cloudflared process crashed or systemd unit failed.
Open bug. RLS fixed 2026-04-21, but Next.js API path still not writing. Likely Vercel env-var issue (SUPABASE_SERVICE_ROLE_KEY missing or stale).
avibm-bot.service: Failed
Bot crashed (often chromedriver drift) or was manually stopped.
Reference.
Open issue (~30-50% missing). IAAI photo download path failing silently. Older queue item — see project_storage_anomalies_todo.md.
iaai photo missing iaai photos failing to download
Photos live across two tiers — hot (/var/lib/auction-intel/static-hot/uploads) and cold (/home/auction/gdrive/AI Projects/auction-intel/static/uploads). They're unioned via mergerfs at /opt/auction-intel/static. Photos newer than ~14 days sit on the hot tier; photo_tier_mover.sh drains them to cold over time. An audit script that only walks ONE tier will report missing dirs that actually exist on the other tier. Always audit via the union mount. Also note: IAAI photo dir names use raw_data.StockId (IAAI internal numeric ID), NOT external_id (StockNum). Pattern: /static/uploads/iaai_<StockId>/photos/<n>.jpg
2,382-listing photo deletion incident on 2026-04-15. Never delete photos / listings / DB rows / folders without explicit per-scope approval. Carsales is the ONLY exception (minimal shape: 1 thumb only).
As of 2026-05-04: ~50 GB total Pickles photos (39 GB hot tier + 11 GB cold tier). ~7,134 active + ~9,500 historical listings. Average 24-26 photos per listing, 844x633 JPEG, ~80 KB each → ~2 MB per listing total. Growth rate is ~5 GB/week sustained. With 2 TB Drive and storage_failover auto-pause at 95%, we have 6+ months runway before any action needed. Per feedback_non_salvage_market_shape.md: NEVER trim or delete Pickles photos (or any auction-source photos). They're sellable archive data. Carsales is the only minimal-shape exception.
Only /thumbs/<source>_<id>.jpg is canonical. Legacy fix_hero_thumbnails.py is DISABLED. auction_service post-scrape spawn handles new thumbs; self_healer maps existing ones. Don't revive the old script.
Reference.
pg_dump: error rclone: failed to upload
Either pg_dump errored, or rclone Drive mount lost during write.
PUSH FAILED — auth needed Permission denied (publickey)
SSH deploy key missing / expired, or GitHub repo deploy-key list out of date.
Pre-VPS pattern: LaunchAgent on user's Mac auto-synced to plugged-in SSD. Now data is VPS-only; SSD sync is mostly historical.
Reference for restoring.
Either cron unit not loaded, file syntax error, or another job overlapped/killed it. Daily app-restart at 23:00 ACST will kill anything running through that minute.
[scheduler] scrapers disabled — skip
settings.json has scrapers_enabled=False. Set during a RED storage event or manual toggle. Storage failover should auto-flip back to True on recovery, but a stuck flag is possible.
All scheduled work uses Australia/Adelaide timezone (ACST = UTC+9:30, ACDT = UTC+10:30 during DST). Pages render local times via register_adelaide_filters Jinja filter.
Reference for the _DAILY scheduler list.
NameError: name 'grays' is not defined NameError: name 'state' is not defined NameError: name 'drive_pct' is not defined scheduler.py.*NameError
f-string log lines in scheduler.py used bare identifiers as dict keys (e.g. g.get(grays, 0) or result.get(state)) when they should have been string literals (g.get('grays', 0)). Python evaluates them as variables, throws NameError, the whole job aborts. Fixed 2026-05-03.
scraper was cancelled CancelledError Daily scrape.*raised an exception Daily scrape —
When systemd restarts auction-intel, in-flight asyncio tasks get cancelled. If a scrape was running, asyncio.gather propagates the CancelledError as an exception. Cosmetic — the scrape will fire again at the next scheduled slot. This used to fire frequently because the daily restart was at 03:00 ACST, colliding with the 03:00 Grays NS slot. Restart was moved to 23:00 on 2026-05-03 to clear the conflict.
Reference for handover state.
Reference procedure.
sudo: a password is required
Self-healer runs as 'auction' user, restarts avibm-bot.service via passwordless sudo. One specific command is granted via sudoers.d.
Where your disk space is going