Internal tools · Document intelligence
DossieClean.
Dossie’s internal document-preparation tool, responsible for much of the processing behind DossieScholar’s collection of 15,000 research papers. It turns PDFs into reviewed metadata, text chunks, and embeddings for semantic retrieval.
Why it exists
A retrieval system is only as useful as the documents behind it. DossieClean gives Dossie an internal workflow for turning academic and research PDFs into reviewed metadata, configurable text chunks, and searchable embeddings.
15,000 research papers
DossieClean did most of the document-preparation work behind DossieScholar’s collection of 15,000 research papers from Addis Ababa Science and Technology University. The work connected PDF preparation and metadata review to the collection’s searchable knowledge foundation.
Review before storage
Single and batch uploads extract titles, authors, publication years, abstracts, and keywords with GPT-4o-mini. The metadata preview lets an operator correct those fields before saving, keeping a human review step in the ingestion process.
A retrieval-ready pipeline
PyPDF2 extracts PDF text. Configurable chunk size and overlap control how that text is divided; OpenAI text-embedding-3-small produces 1,536-dimension embeddings. Azure Blob Storage holds the PDF files while Qdrant holds the vector records.
Working with a collection
The dashboard shows document statistics and recent records. Natural-language search can target documents or chunks, while document details expose metadata and individual chunks. Batch processing includes real-time progress tracking, and settings expose chunking controls alongside dark and light themes.
The wider research workflow
The broader DossieScholar workflow included Python and FastAPI pipelines to clean, process, index, and embed the research collection, OCR digitization, and vector retrieval with RAG. API and concurrent-user tests with Locust helped identify bottlenecks and improve reliability. DossieClean supplied the internal preparation workflow within that larger system.
Built as an internal utility
The application uses Python 3.8+, Flask, Jinja2 templates, and JavaScript. Its API supports metadata extraction, reviewed saves, direct processing, batch execution, and batch-status checks. This project is presented as Dossie’s internal tooling rather than a separate public product.
How it comes together
- Upload
Add one PDF or a batch of research documents.
- Extract & review
Extract text and AI-generated metadata, then review or edit the fields.
- Chunk & embed
Apply configurable chunking and generate 1,536-dimension vectors.
- Store & retrieve
Save PDFs to Azure, index vectors in Qdrant, and search the collection.