Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Question Answering (QA) has made progress in recent years due to large pre-trained language models, benchmark datasets and algorithms
  • PRIMEQA is an open-source QA repository to facilitate replication of state-of-the-art QA methods
  • PRIMEQA supports core QA functionalities and auxiliary capabilities such as question generation
  • PRIMEQA is designed for building front-end applications, replicating SOTA methods and expanding pre-existing methods

Paper Content

Introduction

Entry points

  • Provides different entry points for QA community
  • Top-level scripts, Jupyter notebooks, Inference APIs, Service Layer, UI

Pipelines for openqa

  • PRIMEQA core components and entry points make it easy to build an OpenQA pipeline.
  • A lightweight wrapper is built around each core component, which implements a training and an inference API.
  • PRIMEQA pipelines are easy to construct using pre-trained models and inference APIs.
  • An example of a pipeline is connecting a ColBERT retriever to an FiD reader to construct a LFQA system.

Services and deployment

  • Use Docker to package components into micro-services
  • Easy and scalable to add, replace or delete services
  • All components available via REST and/or gRPC micro-services
  • Docker containers available on DockerHub
  • Three sibling repositories for application deployment
  • Demo application provides mechanism to collect user feedback

Community contributions

  • PRIMEQA is a relatively new tool that has been positively received by the QA community
  • Stanford NLP contributed ColBERT, which provides SOTA performance on OpenQA benchmark datasets
  • Ohio State University contributed ReasonBERT, which outperforms a RoBERTa baseline on the extractive QA task
  • LTI CMU’s NeuLab integrated OmniTab, which produces the best few-shot performance on Wiki-TableQuestions
  • NASA and University of Alabama in Huntsville created a custom search application over Earth Science abstracts and papers using PRIMEQA

Conclusion

  • PRIMEQA is an open-source library designed to facilitate reproducibility and reusability of QA research.
  • PRIMEQA provides a ‘service layer’ to deploy pre-trained models for custom search applications.
  • PRIMEQA is built on top of NLP open-source libraries and tools and has garnered significant positive traction.