Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Question Answering (QA) has made progress in recent years due to large pre-trained language models, benchmark datasets and algorithms
- PRIMEQA is an open-source QA repository to facilitate replication of state-of-the-art QA methods
- PRIMEQA supports core QA functionalities and auxiliary capabilities such as question generation
- PRIMEQA is designed for building front-end applications, replicating SOTA methods and expanding pre-existing methods
Paper Content
Introduction
Entry points
- Provides different entry points for QA community
- Top-level scripts, Jupyter notebooks, Inference APIs, Service Layer, UI
Pipelines for openqa
- PRIMEQA core components and entry points make it easy to build an OpenQA pipeline.
- A lightweight wrapper is built around each core component, which implements a training and an inference API.
- PRIMEQA pipelines are easy to construct using pre-trained models and inference APIs.
- An example of a pipeline is connecting a ColBERT retriever to an FiD reader to construct a LFQA system.
Services and deployment
- Use Docker to package components into micro-services
- Easy and scalable to add, replace or delete services
- All components available via REST and/or gRPC micro-services
- Docker containers available on DockerHub
- Three sibling repositories for application deployment
- Demo application provides mechanism to collect user feedback
Community contributions
- PRIMEQA is a relatively new tool that has been positively received by the QA community
- Stanford NLP contributed ColBERT, which provides SOTA performance on OpenQA benchmark datasets
- Ohio State University contributed ReasonBERT, which outperforms a RoBERTa baseline on the extractive QA task
- LTI CMU’s NeuLab integrated OmniTab, which produces the best few-shot performance on Wiki-TableQuestions
- NASA and University of Alabama in Huntsville created a custom search application over Earth Science abstracts and papers using PRIMEQA
Conclusion
- PRIMEQA is an open-source library designed to facilitate reproducibility and reusability of QA research.
- PRIMEQA provides a ‘service layer’ to deploy pre-trained models for custom search applications.
- PRIMEQA is built on top of NLP open-source libraries and tools and has garnered significant positive traction.