Error Logging Policy

This page documents the convention adopted across background-tasks after SUP-00178: coding errors must be loud. The convention exists because two AttributeErrors went unnoticed for months — they were caught by except Exception blocks that logged at DEBUG / WARNING, which CloudWatch ERROR scans never surfaced.

The rule

Exception class	Log level	exc_info=True?	Behaviour
AttributeError, TypeError, NameError, ImportError	ERROR	Yes	Coding bug. Surface immediately so it shows up in CloudWatch ERROR alerts and CI log scans.
asyncpg.PostgresError, asyncio.TimeoutError, OSError, aiohttp.ClientError	WARNING	No	Transient infrastructure failure. Continue gracefully but record.
Domain BillingError, PipelineError, RateLimitError, etc.	WARNING / re-raise	No	Expected business outcome. Re-raise if the caller needs to react.
Anything else falling through except Exception	ERROR	Yes	Treat unknown exceptions as coding bugs by default; we'd rather see noise than silence.

What good looks like

try:
    await BillingGate.pre_check(company_id, "AGENT_CHAT", user_id=user_id)
except Exception as billing_err:
    if isinstance(billing_err, BillingError):
        await BillingGate.log_rejected(company_id, "AGENT_CHAT", type(billing_err).__name__, user_id=user_id)
        raise
    if isinstance(billing_err, (AttributeError, TypeError, NameError, ImportError)):
        logger.error(
            f"Billing pre-check coding error (non-blocking, please fix): {billing_err}",
            exc_info=True,
        )
    else:
        logger.warning(f"Billing pre-check failed (non-blocking): {billing_err}")

The first isinstance re-raises legitimate billing rejections. The second logs coding mistakes loudly without breaking the user-facing flow. Everything else is treated as a transient warning.

What bad looks like

try:
    cv_data = await get_cv_data_by_user_id(self.pool, user_id)  # AttributeError on self.pool
    ...
except Exception as e:
    logger.debug(f"Structured CV skills lookup failed for {user_id}: {e}")  # ❌ swallows the bug

The DEBUG line never showed up in CloudWatch ERROR scans. The matching engine silently returned "no skills" for affected users for months. Don't do this.

The fix: narrow the catch to expected exception types only, so coding bugs propagate out of the function:

try:
    pool = await get_pool()
    cv_data = await get_cv_data_by_user_id(pool, user_id)
    ...
except (asyncpg.PostgresError, ValueError, KeyError) as e:  # ✅ narrow
    logger.debug(f"Structured CV skills lookup failed for {user_id}: {e}")

Where this convention is enforced today

• app/services/agent/conversation_service.py — agentChat billing pre-check
• app/services/agent/document_worker.py — agent-document embedding billing pre-check
• app/services/bulk_cv/worker.py — bulk CV import billing pre-check
• app/services/matching/database/user_queries.py — tertiary skills lookup

When you write or review new code that wraps an async call in try/except Exception, ask:

1. What expected exceptions does the inner code actually raise? Catch only those.
2. If a coding bug slips through, would I want to see it in CloudWatch ERROR? If yes, follow the split-log pattern above.
3. Does the surrounding function need to fail, or does it have a defensible fallback? If it falls through, log at ERROR with exc_info=True so the stack trace is recoverable.

Related

• Monitoring — CloudWatch dashboards and where ERROR-level lines show up
• CV Skills Backfill — runbook for one of the bugs this convention was introduced to catch