Status: [UNTESTED] Endpoints found: http://localhost:8080/v1/documents http://localhost:8080/v1/jobs/11111111-1111-4111-8111-111111111111 http://localhost:8081/v1/query Source: https://github.com/felipe-medeiros-alves/async-rag-pipeline # async-rag-pipeline An event-driven retrieval-augmented generation (RAG) pipeline. Upload a PDF, receive a `job_id` immediately, wait while workers embed its chunks, then run a vector search optionally restricted to that upload. This repository is a local demonstration of asynchronous ingestion, backpressure, horizontal consumers, and filtered vector retrieval. It is not production-ready; Kafka delivery remains at-least-once, while chunk progress is deduplicated durably in Postgres. ## Architecture ```text ┌──────────────┐ PDF ──POST /v1/documents──► Go ingest ─► Postgres jobs│ │ └──────────────┘ ├──► upload volume │ └──► Kafka: document.chunks │ ▼ Python worker(s) │ │ deterministic │ └──► Kafka: document.events chunk upsert ▼ │ Qdrant chunks ▼ ▲ Go status consumer │ │ query ──POST /v1/query──► Python query API │ └──► Postgres ``` Kafka separates fast HTTP acceptance from CPU-heavy embedding. That gives the pipeline durable buffering, consumer-group load distribution, and independent worker scaling. An in-process queue would be simpler, but queued work would be tied to one ingest process and would provide weaker backpressure and recovery. ## Prerequisites and setup In