Skip to main content

Correct File Upload and MinIO Flow for CV Extraction

Overview​

The correct flow for CV extraction uses your existing GraphQL infrastructure and MinIO storage, not a custom upload API. Here's how it works:

🔄 Complete Flow​

1. Generate Upload URL (GraphQL)​

const { data } = await generateUploadUrl({
  variables: {
    filename: file.name,
    contentType: file.type
  }
})

const { uploadUrl, objectPath } = data.upload.generateUploadUrl

What happens:

  • Your GraphQL server generates a signed MinIO upload URL
  • Returns both the uploadUrl (for uploading) and objectPath (for referencing)

2. Upload File to MinIO Directly​

const xhr = new XMLHttpRequest()
xhr.open('PUT', uploadUrl)
xhr.setRequestHeader('Content-Type', file.type)
xhr.send(file) // Upload directly to MinIO

What happens:

  • File is uploaded directly to MinIO using the signed URL
  • No file goes through your API server
  • Progress tracking available via XMLHttpRequest

3. Send Object Path via WebSocket​

const message = {
  type: 'cv_extraction_request',
  pipeline_type: 'cv_extraction',
  payload: {
    id: requestId,
    file_url: objectPath  // ← This is the MinIO object path, not a URL
  }
}

What happens:

  • Send the objectPath (like "cv-files/1234567890-resume.pdf") to the pipeline
  • Pipeline worker can access the file directly from MinIO using this path

🏗️ Architecture Benefits​

Why This Approach is Better​

  1. No API Server Load: Files don't go through your API server
  2. Direct MinIO Access: Pipeline workers read directly from MinIO
  3. Signed URLs: Secure, time-limited upload permissions
  4. Progress Tracking: Real-time upload progress
  5. Existing Infrastructure: Uses your current GraphQL setup

Data Flow Diagram​

Client ----[1. Generate URL]----> GraphQL Server
       <---[uploadUrl, objectPath]--

Client ----[2. Upload File]------> MinIO Storage
       <---[Upload Complete]------

Client ----[3. WebSocket Message]--> Pipeline Server
       |    (with objectPath)
       |
       <---[4. Processing Results]---

🔧 Implementation Details​

GraphQL Mutation Used​

mutation GenerateUploadUrl($filename: String!, $contentType: String) {
  upload {
    generateUploadUrl(filename: $filename, contentType: $contentType) {
      uploadUrl    # For uploading to MinIO
      objectPath   # For referencing in pipeline
    }
  }
}

WebSocket Message Format​

{
  "type": "cv_extraction_request",
  "pipeline_type": "cv_extraction", 
  "payload": {
    "id": "extract-1703123456-abc123",
    "file_url": "cv-files/1703123456-john_doe_resume.pdf"
  }
}

Response Processing​

{
  "type": "pipeline_result",
  "data": {
    "data": {
      "first_name": "John",
      "last_name": "Doe",
      "skills": [...],
      "experiences": [...]
    }
  }
}

🚫 What We Removed​

Mock Upload API​

  • Before: Custom /api/upload endpoint that returned fake MinIO URLs
  • After: Direct GraphQL + MinIO integration

Why the Mock Was Wrong​

  1. Created fake MinIO URLs that don't exist
  2. Files weren't actually stored in MinIO
  3. Pipeline couldn't access the files
  4. Added unnecessary API layer

✅ Current Implementation​

The updated useCvExtractionWebSocket hook now:

  1. ✅ Uses GENERATE_UPLOAD_URL GraphQL mutation
  2. ✅ Uploads directly to MinIO with signed URLs
  3. ✅ Sends real object paths via WebSocket
  4. ✅ Tracks upload progress properly
  5. ✅ Integrates with existing infrastructure

🔍 Key Files Updated​

lib/hooks/useCvExtractionWebSocket.ts​

  • Added GraphQL mutation import
  • Updated uploadFile function to use proper MinIO flow
  • Sends objectPath instead of fake URLs

Removed Files​

  • app/api/upload/route.ts - No longer needed

🧪 Testing​

To test the implementation:

  1. Upload a CV file - Should see real MinIO upload progress
  2. Check browser network tab - Should see PUT request to MinIO
  3. WebSocket message - Should contain real object path like "cv-files/filename.pdf"
  4. Pipeline processing - Should access file from MinIO successfully

📝 Production Notes​

Environment Setup​

  • Ensure MinIO credentials are configured
  • GraphQL server must have MinIO access
  • Pipeline workers need MinIO read permissions

Security​

  • Upload URLs are signed and time-limited
  • Object paths are scoped to appropriate buckets
  • No direct file access through web endpoints

This approach follows best practices for file handling in distributed systems and leverages your existing infrastructure properly.