Bulk CV Import

The Bulk CV Import feature allows companies to import multiple CVs at once from a ZIP file. The system extracts each CV using the existing CV extraction pipeline, creates User/Talent/CV records, and handles deduplication by email. All processing happens in the background with real-time progress streaming via GraphQL subscriptions.

Architecture

Per-CV Processing Flow

Supported File Types

Extension	Format	Notes
`.pdf`	PDF documents	Primary format, processed via OCR
`.doc`	Legacy Word	Processed via python-docx
`.docx`	Modern Word	Processed via python-docx
`.txt`	Plain text	Direct text extraction

Skipped Files

The system automatically skips:

__MACOSX/ folders and ._ prefixed files (macOS ZIP artifacts)
Hidden files (starting with .) in any directory level
Files with unsupported extensions (images, spreadsheets, etc.)
Directory entries

GraphQL Mutation

The mutation returns BulkCvImportProcessing directly (not a union type). Validation errors are returned as standard GraphQL errors.

mutation CreateBulkCvImport($input: BulkCvImportInput!) {
  createBulkCvImport(input: $input) {
    id
    status
    message
    progress
    totalFiles
    processedFiles
    successCount
    failedCount
    skippedCount
  }
}

Input

{
  "input": {
    "fileUrl": "bulk-cv-test/my-upload.zip",
    "companyId": "company-uuid",
    "userId": "user-uuid"
  }
}

Field	Type	Description
`fileUrl`	String	S3 key or S3 URL of the uploaded ZIP file
`companyId`	String	Company UUID — talents are added to this company's pool
`userId`	String	User UUID — the user who initiated the import

Validation

The mutation performs synchronous validation before returning:

URL validation — fileUrl must be an S3 path (not an arbitrary HTTP URL)
ZIP validation — Downloads the file and checks for valid ZIP magic bytes (PK\x03\x04)
Size validation — Compressed size must be < 500 MB (configurable)
ZIP bomb detection — Total uncompressed size must be < 2 GB
File count — Counts supported CV files; rejects if 0 or > 500
Creates DB record — Inserts BulkCvImport row with status=PENDING

Error Responses

Condition	Error Message
Non-S3 URL	`file_url must be an S3 path or storage URL`
Not a ZIP file	`File is not a valid ZIP archive`
ZIP too large	`ZIP file too large (X bytes, max Y bytes)`
ZIP bomb detected	`ZIP uncompressed size too large`
No CV files	`No supported CV files found in ZIP (supported: .pdf, .doc, .docx, .txt)`
Too many files	`Too many CV files (X, max 500)`

GraphQL Subscription

subscription BulkCvImport($importId: String!) {
  bulkCvImport(importId: $importId) {
    ... on BulkCvImportProcessing {
      id
      status
      message
      progress
      totalFiles
      processedFiles
      successCount
      failedCount
      skippedCount
    }
    ... on BulkCvImportSuccess {
      id
      status
      totalFiles
      successCount
      failedCount
      skippedCount
      processingTimeMs
      results {
        filename
        status
        userId
        talentId
        cvId
        email
        errorMessage
      }
    }
    ... on BulkCvImportFailure {
      id
      status
      error {
        code
        message
        recoveryHint
      }
    }
  }
}

Update Types

Type	When	Contains
`BulkCvImportProcessing`	Every 2s while processing	Progress percentage, file counts, success/fail/skip counters
`BulkCvImportSuccess`	Terminal: all or some succeeded	Per-file results with userId, talentId, cvId, email
`BulkCvImportFailure`	Terminal: all failed or not found	Error code, message, recovery hint

Progress Streaming

The subscription polls the database every 2 seconds. Updates are only emitted when processedFiles changes or a terminal status is reached.

Background Worker

File: app/services/bulk_cv/worker.py

The BulkCvImportWorker follows the same pattern as other background workers (Document Worker, Inbound Email Worker):

Setting	Default	Env Variable	Description
Poll interval	30s	`BULK_CV_POLL_INTERVAL`	How often to check for pending jobs
Max concurrent CVs	3	`BULK_CV_MAX_CONCURRENT`	Semaphore limit for parallel CV processing
Max retries	3	`BULK_CV_MAX_RETRIES`	Max attempts before giving up on a job
Max files per ZIP	500	`BULK_CV_MAX_FILES`	Maximum CV files allowed in a single ZIP
Max ZIP size	500 MB	`BULK_CV_MAX_ZIP_SIZE_BYTES`	Maximum compressed ZIP file size
Enabled	true	`BULK_CV_IMPORT_ENABLED`	Feature toggle

Job Claiming

Uses SELECT FOR UPDATE SKIP LOCKED to safely claim jobs in multi-worker scenarios:

Only claims jobs with status IN ('PENDING', 'EXTRACTING') and attemptCount < maxRetries
Stale locks (older than 10 minutes) are automatically reclaimed
Each claim increments attemptCount

Status Lifecycle

Circuit Breaker

The worker uses a circuit breaker to prevent cascading failures:

Failure threshold: 5 consecutive failures
Reset timeout: 300 seconds (5 minutes)
Call timeout: 600 seconds (10 minutes)
When OPEN, the worker backs off for 60 seconds before retrying

Deduplication Logic

When processing each CV, the system deduplicates by email address:

Created Records

For each successfully processed CV:

Record	Details
User	`isOnboarding=false`, `isActive=false`, `isVerified=false`. No invitation email sent.
Talent	Added to company's default TalentPool with `source=BULK_IMPORT`
CV	Linked to both User and Talent, set as `activeCvId`
Structured CV tables	Structured extraction data stored for matching
Skills	Extracted and normalized skills saved to CVSkill table — proficiency synonyms (`BEGINNER`, `MASTER`, …) are mapped to the DB enum before insert; per-row failures are isolated. See Proficiency normalization on the CV-extraction page.

note

If a bulk import lands with cv.text populated but its CVSkill / structured tables empty, that's the exact pattern the CV skills backfill script remediates. The script re-runs the LLM extraction on the cached text without re-fetching from S3.

Database Schema

-- Enum
CREATE TYPE "BulkCvImportStatus" AS ENUM (
  'PENDING', 'EXTRACTING', 'PROCESSING',
  'COMPLETED', 'PARTIALLY_COMPLETED', 'FAILED'
);

-- Table
CREATE TABLE "BulkCvImport" (
  "id"              UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  "companyId"       UUID NOT NULL REFERENCES "Company"("id"),
  "userId"          UUID NOT NULL REFERENCES "User"("id"),
  "status"          "BulkCvImportStatus" DEFAULT 'PENDING',
  "zipFileUrl"      TEXT NOT NULL,
  "totalFiles"      INT DEFAULT 0,
  "processedFiles"  INT DEFAULT 0,
  "successCount"    INT DEFAULT 0,
  "failedCount"     INT DEFAULT 0,
  "skippedCount"    INT DEFAULT 0,
  "results"         JSONB,
  "errorMessage"    TEXT,
  "processingTimeMs" INT,
  "attemptCount"    INT DEFAULT 0,
  "lockedAt"        TIMESTAMP,
  "lockedBy"        TEXT,
  "createdAt"       TIMESTAMP DEFAULT NOW(),
  "updatedAt"       TIMESTAMP DEFAULT NOW()
);

The results JSONB column stores per-file results:

[
  {
    "filename": "resume_john.pdf",
    "status": "SUCCESS",
    "user_id": "uuid",
    "talent_id": "uuid",
    "cv_id": "uuid",
    "email": "john@example.com",
    "error_message": null
  },
  {
    "filename": "broken_cv.pdf",
    "status": "FAILED",
    "user_id": null,
    "talent_id": null,
    "cv_id": null,
    "email": null,
    "error_message": "No email found in CV"
  }
]

File Structure

File	Purpose
`app/graphql/types/bulk_cv_import.py`	GraphQL types (Input, Processing, Success, Failure, FileResult)
`app/graphql/mutations/bulk_cv_import.py`	Mutation: validates ZIP, creates DB record
`app/graphql/subscriptions/bulk_cv_import.py`	Subscription: polls DB, streams progress
`app/services/bulk_cv/__init__.py`	Module init, exports worker singleton
`app/services/bulk_cv/worker.py`	Background worker (polling, processing loop)
`app/services/bulk_cv/repository.py`	AsyncPG database queries
`app/services/bulk_cv/zip_utils.py`	ZIP validation and extraction utilities
`app/services/bulk_cv/talent_creator.py`	Shared user/talent/CV creation logic
`tests/unit/services/test_bulk_cv_worker.py`	145 unit tests
`tests/e2e/test_bulk_cv_import_e2e.py`	11 E2E tests against dev environment

Billing Integration

The worker calls BillingGate.pre_check(company_id, "CV_EXTRACTION", user_id=user_id) before each CV extraction. If billing blocks the operation (insufficient credits, plan limit exceeded), the CV is marked as SKIPPED (not FAILED) and processing continues with the remaining files.

Logging

Every step of the import pipeline is logged with structured extra fields for monitoring:

Event Type	Description
`bulk_cv_record_created`	DB record created by mutation
`bulk_cv_job_claimed`	Worker claimed a pending job
`bulk_cv_downloading_zip`	Starting ZIP download
`bulk_cv_zip_downloaded`	ZIP downloaded, extracting files
`bulk_cv_processing_start`	File extraction complete, starting CV processing
`bulk_cv_file_start`	Starting individual CV processing
`bulk_cv_file_done`	Individual CV processing complete
`bulk_cv_file_success`	CV processed successfully (includes user/talent/cv IDs)
`bulk_cv_billing_blocked`	Billing pre-check blocked CV extraction
`bulk_cv_extraction_failed`	CV extraction pipeline failed
`bulk_cv_no_email`	No email found in parsed CV data
`bulk_cv_job_completed`	Job finished with final status and counts
`bulk_cv_job_failed`	Job failed with error

Testing

Unit Tests (145 tests)

pytest tests/unit/services/test_bulk_cv_worker.py -v -m unit

Covers: ZIP validation, file filtering (extensions, __MACOSX, hidden files, nested folders, unicode filenames, special characters), skills normalization, qualifications builder, dedup logic, progress tracking, error handling (S3 errors, billing blocks, extraction failures, missing emails), repository operations, subscription behavior, result dict key compatibility.

E2E Tests (11 tests)

pytest tests/e2e/test_bulk_cv_import_e2e.py -v -m e2e

Requires AWS SSO login and runs against ai-dev.aiqlick.com:

Test	Scenario
`test_full_import_flow_with_pdfs`	Full pipeline: upload 2 PDFs → mutation → subscription → COMPLETED
`test_dedup_reuses_existing_users`	Re-import same ZIP, verify user IDs match
`test_non_zip_file_rejected`	PDF file (not ZIP) rejected with error
`test_empty_zip_rejected`	ZIP with no CV files rejected
`test_subscription_not_found`	Non-existent import ID returns FAILED
`test_single_pdf_import`	ZIP with single PDF processes correctly
`test_zip_with_nested_folders`	CVs in deeply nested folders are found
`test_zip_with_macosx_artifacts`	macOS ZIP metadata is filtered out
`test_zip_with_only_unsupported_files`	ZIP with only images/spreadsheets rejected
`test_subscription_streams_incremental_progress`	Progress updates are monotonically increasing
`test_mutation_returns_correct_total_files`	File count excludes non-CV files

Limitations

Text files (.txt) may fail CV extraction if the LLM cannot parse structured data from plain text
No password-protected ZIPs — encrypted ZIP files will fail extraction
Email required — CVs without an email address in the parsed data are marked as FAILED
No invitation emails — Users created via bulk import are inactive and unverified; no emails are sent

Architecture​

Per-CV Processing Flow​

Supported File Types​

Skipped Files​

GraphQL Mutation​

Input​

Validation​

Error Responses​

GraphQL Subscription​

Update Types​

Progress Streaming​

Background Worker​

Job Claiming​

Status Lifecycle​

Circuit Breaker​

Deduplication Logic​

Created Records​

Database Schema​

File Structure​

Billing Integration​

Logging​

Testing​

Unit Tests (145 tests)​

E2E Tests (11 tests)​

Limitations​