HistoryLab AI

Most declassified government documents are technically public but practically hard to use. Collections like the CIA’s CREST database, the Foreign Relations of the United States series, and State Department cable archives sit in scattered formats with rough OCR and inconsistent metadata. Finding what’s relevant can take weeks.

HistoryLab AI is a conversational search engine over the five million-plus declassified documents collected and cleaned by Professor Matthew Connelly’s History Lab at Columbia. You ask a question in plain English — “What was the U.S. position on the Formosa Question between 1947 and 1952?” — and it retrieves relevant passages by semantic search, answers with citations, and links back to the source documents.

I built the retrieval and serving stack on top of the Lab’s corpus: embedded text chunks in a vector database, a two-stage vector search (a coarse pass over the corpus, then exact reranking), and an LLM that composes cited answers. The same corpus is also exposed to LLM agents through an MCP server.

The site launched publicly in spring 2025 and has been adopted by 100+ historians, academics, and government researchers. We presented it with the History Lab team at the Federal Depository Library Conference. It’s live at history-lab.ramus.network.