Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Widespread calls for transparent AI systems lack precision.
  • Stakeholders often talk past each other.
  • A clear ideal of AI transparency is needed.
  • Literature survey identifies clusters of similar conceptions of transparency.
  • Common threads across all clusters provide clearer common language.

Paper Content

Introduction

  • Calls for greater transparency in AI systems
  • Term “transparency” overloaded with distinct meanings
  • AI’s suitcase words
  • Concrete aims and advances must be expressed in more precise language
  • Transparency invoked in connection with data collection, data processing, interpretable systems, fairness issues
  • EU regulations (GDPR, Ethics Guidelines for Trustworthy AI) vague demands for “meaningful information” and “comprehensible language”
  • Ideal AI transparency gives users and stakeholders tools to decide if AI system and decisions are trustworthy
  • Three overarching factors with which transparency is invoked: data, systems, outputs
  • Focus on inputs required to produce AI system
  • Explore ways to balance transparency and user concerns about data privacy and security
  • Distinguish between works focused on information about training data and active use of user data

Transparency on model training data

  • Machine learning systems are influenced by their training data
  • Policymakers are mandating disclosures about training data
  • Record transparency is achieved by describing datasets
  • Use transparency is communicating the specific purposes for which a dataset is appropriate
  • Disambiguation of terminology, visualization, and logging systems are useful for disclosure/data-provisioning transparency
  • Rules and norms are needed to ensure AI transparency for users

Transparency on the handling of user data

  • AI systems need user data to function
  • Demand for transparency around user data use is natural
  • US data policy is largely unregulated
  • EU has taken a more active approach with GDPR
  • Classic security research findings are applicable
  • Companies address consumer privacy and security to win favor
  • Tension between user desires and business realities
  • Need strong societal norms and regulation to resolve tension
  • Transparency tools needed to enable users to exercise rights

System-centered transparency factors

  • Practitioners need to debug models or reproduce results
  • Users need a basic overview of a system’s function
  • ML systems are black boxes, hindering explainability
  • Neural networks are dominant in AI research
  • Automated rationale generation is popular

System function disclosure

  • System function disclosure includes communications about system capabilities and limitations
  • Target audiences include external developers, lay users, and regulatory bodies
  • ACM considers disclosure required in its Code of Ethics
  • Frameworks for concise communication of model strengths and limitations are important
  • Qualitatively evaluate disclosure sufficiency with rubrics or automatically assess layperson-comprehensibility
  • Experts don’t always know how black box systems work
  • Clarity from data providers needed to communicate problems of systems
  • Solicit direct user feedback to contextualize failure cases
  • Different groups require different explanations and have different levels of expertise

Explainable ai and causality

  • Transparency is connected to explainable AI techniques
  • Simple ML models are explainable, but lack flexibility
  • Neural networks can be made more interpretable by converting their internal weight matrices or distillation
  • Attention mechanisms are often presented as interpretable
  • Input influence methods are often used as explainability techniques
  • Counterfactual reasoning can be used to explain decisions without altering black-box models
  • Explanations should establish a causal interpretation to be honest and user-centric

Generated rationales

  • Introduce automated rationale generation as an alternative form of explainability
  • Rationales provide insight into language models’ reasoning abilities
  • Rationales can help users fact-check outputs to mitigate potential for misinformation
  • Multiple failure modes exist for these rationales, including hallucination
  • Evaluating NLG explanations is a challenge, users must be educated on degree of trust

Output-oriented transparency factors

  • Output-oriented transparency is focused on providing good system performance for stakeholders.
  • Different concepts in system demonstration are distinguished, such as repeatability, replicability, and reproducibility.
  • Problems arise around how much transparency to provide without overloading information.
  • Motivation for system demonstration is often for fairness, accountability, or trust.
  • Explainable models can promote fairness and accountability, but can also harm privacy and security.

System demonstrability

  • System demonstration is necessary to support claims of function, performance, consistency, privacy, and security
  • Repeatability is the easiest to articulate and generally assumed
  • Replicability and reproducibility receive more attention in the AI transparency literature
  • Technologies can enable more replicable research, but have limitations
  • Non-technical factors can also affect reproducibility
  • Terminology fosters an atmosphere of precise communication to encourage robust AI systems

Fairness and accountability

  • Transparency is important for fairness
  • System fragility and systematic bias can lead to inaccurate results
  • Lack of transparency can reduce consumer and public trust
  • Explainable models increase accountability for fairness
  • Proactive stance on fairness encourages transparency and explainability
  • Fairness involves social context to mitigate systemic bias

Discussion

  • AI transparency research has multiple clusters with sometimes conflicting goals
  • Commonalities and conflicts between AI transparency research clusters can be found in desired ends, associated stakeholders, and utilized means

Desired ends

  • Desired ends of stakeholders often conflict
  • Power dynamic between stakeholders who choose means of transparency and lay users who cannot
  • Developers seek explainability and trust
  • Legislators invoke transparency requirements for visibility and equitable outcomes
  • Transparency of AI systems is necessary to give comprehensive transparency around sociotechnical systems
  • Information on overall goals of system is necessary to give users an honest understanding

Conflicting means

  • Human disclosure and system disclosure can achieve transparency goals.
  • Deception and information overload can be used to mislead less empowered stakeholders.
  • Disclosures can conflict with intellectual property protection needs.

Selection of study attributes

  • Fundamental tension between financial results and responsible AI requirements in corporate settings
  • Researchers positioned within entities that maintain large AI systems are often uniquely positioned to assess their real-world impacts
  • Societal impacts of AI systems may lead researchers to act as interested stakeholders
  • OpenAI researchers cite dangers of large language model abuse as motivation for their tiered-release strategy
  • Balancing act between researcher interest and desirable research goals
  • Measures of effectiveness in transparency vary depending on target measure
  • Goal is to empower users, not just assuage their concerns
  • Explanations often do not provide causal justification
  • Incentive is to persuade users to believe in systems, not demonstrate trustworthiness
  • Claims of researchers may differ from their actions
  • Precision in terminology and alignment of claims with reality must be prioritized in work

Conclusion

  • AI transparency-related works can be clustered into narrower categories
  • Terminological imprecision of system transparency needs to be resolved
  • Reproducibility and ethical considerations should be asked of authors
  • Research community should speak honestly, clearly and unambiguously about transparency
  • Attributes of transparency studies should be characterized
  • Means for transparency should be discussed