ProjectProduction system
Procurement Intelligence at National Scale
Public procurement data is public, but it is not usable.
Documented
Why I made it
I worked on this because access to public information should not mean someone has to manually chase thousands of incompatible websites and PDFs to use it.
Government tender data is public but not usable: it's spread across 20,000+ agency websites with no shared schema, no shared identifiers for who's bidding on what, and no reliable way to tell a reissued tender from a genuinely new one. This is the pipeline I built at Prospecto to fix that: distributed crawlers for discovery, NLP/LLM-assisted extraction to turn unstructured notices into structured records, and entity resolution to unify the same supplier or buyer across sources that spell its name three different ways. The output feeds procurement teams a monitored, deduplicated view of tenders instead of a pile of PDFs.
Tools: Python · NLP · LLM Extraction · Entity Resolution · Distributed Pipelines
Public procurement data is, by law, public. In practice it's unusable. Every contracting authority publishes to its own site, in its own format, on its own schedule — there is no central index, no shared identifier scheme, and no guarantee that the same supplier is spelled the same way twice. Finding every tender relevant to your business means either paying for one of a handful of expensive aggregators, or checking thousands of sites by hand.
The problem
At Prospecto, I work on a platform that treats this as a data-engineering problem rather than a search problem. The scale is the constraint: 20,000+ government websites, each a distinct source with its own HTML structure, refresh cadence, and document formats (PDF notices, scanned attachments, structured XML feeds where they exist). A tender discovered on Monday and reissued with a minor amendment on Thursday needs to be recognized as the same tender, not two.And the failure is asymmetric. Splitting one tender into two is noise a user scrolls past; merging two distinct tenders hides one of them completely, which is the failure that loses somebody a bid.
How it works
The pipeline has three stages, and each one exists because the stage before it produces output that isn't directly usable.
| Stage | Input | Output |
|---|---|---|
| Discovery | 20,000+ agency websites | Raw notices, PDFs, attachments |
| NLP/LLM extraction | Unstructured notice text | Structured fields — CPV codes, deadlines, values |
| Entity resolution | Structured records, inconsistent naming | Unified supplier/buyer/tender identities |
Discovery is a distributed crawling layer that treats each site as its own schedule and failure domain — a site going down or changing its markup shouldn't stall discovery on the other 19,999. Extraction applies NLP and LLM-assisted parsing to turn free-text notices into structured records: what's being procured, under what code, by which deadline, at what estimated value. Entity resolution is the hardest of the three: the same municipal buyer might appear as "City of X," "Stadt X," and "X Municipality — Procurement Office" across three different notices, and the pipeline has to collapse those into one entity without merging genuinely distinct suppliers who happen to share a common word in their name.
Why this matters
None of these three stages is optional, and skipping any one of them is why most "procurement search" tools are really just search engines over unstructured PDFs. Discovery without extraction gives you a pile of documents. Extraction without entity resolution gives you structured records that still can't answer "show me everything this supplier has bid on." The pipeline's value is in the composition, not any single stage — it's the same reason RAG quality is a data problem before it's a retrieval problem: the hard part is almost never the model call, it's getting the underlying data into a shape where a model call can be trusted.
Limitations
- This writeup is intentionally an architecture summary, not a walkthrough. The platform, its clients, and its internals are Prospecto's, not mine to publish in detail — no repo, no screen recording, no benchmark numbers here.
- Entity resolution is never fully solved, only continuously improved. Ambiguous cases (genuinely similar but distinct supplier names, buyers that reorganize mid-year) get flagged for review rather than resolved silently.