Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Widespread calls for transparent AI systems lack precision.
- Stakeholders often talk past each other.
- A clear ideal of AI transparency is needed.
- Literature survey identifies clusters of similar conceptions of transparency.
- Common threads across all clusters provide clearer common language.
Paper Content
Introduction
- Calls for greater transparency in AI systems
- Term “transparency” overloaded with distinct meanings
- AI’s suitcase words
- Concrete aims and advances must be expressed in more precise language
- Transparency invoked in connection with data collection, data processing, interpretable systems, fairness issues
- EU regulations (GDPR, Ethics Guidelines for Trustworthy AI) vague demands for “meaningful information” and “comprehensible language”
- Ideal AI transparency gives users and stakeholders tools to decide if AI system and decisions are trustworthy
- Three overarching factors with which transparency is invoked: data, systems, outputs
Data-related transparency factors
- Focus on inputs required to produce AI system
- Explore ways to balance transparency and user concerns about data privacy and security
- Distinguish between works focused on information about training data and active use of user data
Transparency on model training data
- Machine learning systems are influenced by their training data
- Policymakers are mandating disclosures about training data
- Record transparency is achieved by describing datasets
- Use transparency is communicating the specific purposes for which a dataset is appropriate
- Disambiguation of terminology, visualization, and logging systems are useful for disclosure/data-provisioning transparency
- Rules and norms are needed to ensure AI transparency for users
Transparency on the handling of user data
- AI systems need user data to function
- Demand for transparency around user data use is natural
- US data policy is largely unregulated
- EU has taken a more active approach with GDPR
- Classic security research findings are applicable
- Companies address consumer privacy and security to win favor
- Tension between user desires and business realities
- Need strong societal norms and regulation to resolve tension
- Transparency tools needed to enable users to exercise rights
System-centered transparency factors
- Practitioners need to debug models or reproduce results
- Users need a basic overview of a system’s function
- ML systems are black boxes, hindering explainability
- Neural networks are dominant in AI research
- Automated rationale generation is popular
System function disclosure
- System function disclosure includes communications about system capabilities and limitations
- Target audiences include external developers, lay users, and regulatory bodies
- ACM considers disclosure required in its Code of Ethics
- Frameworks for concise communication of model strengths and limitations are important
- Qualitatively evaluate disclosure sufficiency with rubrics or automatically assess layperson-comprehensibility
- Experts don’t always know how black box systems work
- Clarity from data providers needed to communicate problems of systems
- Solicit direct user feedback to contextualize failure cases
- Different groups require different explanations and have different levels of expertise
Explainable ai and causality
- Transparency is connected to explainable AI techniques
- Simple ML models are explainable, but lack flexibility
- Neural networks can be made more interpretable by converting their internal weight matrices or distillation
- Attention mechanisms are often presented as interpretable
- Input influence methods are often used as explainability techniques
- Counterfactual reasoning can be used to explain decisions without altering black-box models
- Explanations should establish a causal interpretation to be honest and user-centric
Generated rationales
- Introduce automated rationale generation as an alternative form of explainability
- Rationales provide insight into language models’ reasoning abilities
- Rationales can help users fact-check outputs to mitigate potential for misinformation
- Multiple failure modes exist for these rationales, including hallucination
- Evaluating NLG explanations is a challenge, users must be educated on degree of trust
Output-oriented transparency factors
- Output-oriented transparency is focused on providing good system performance for stakeholders.
- Different concepts in system demonstration are distinguished, such as repeatability, replicability, and reproducibility.
- Problems arise around how much transparency to provide without overloading information.
- Motivation for system demonstration is often for fairness, accountability, or trust.
- Explainable models can promote fairness and accountability, but can also harm privacy and security.
System demonstrability
- System demonstration is necessary to support claims of function, performance, consistency, privacy, and security
- Repeatability is the easiest to articulate and generally assumed
- Replicability and reproducibility receive more attention in the AI transparency literature
- Technologies can enable more replicable research, but have limitations
- Non-technical factors can also affect reproducibility
- Terminology fosters an atmosphere of precise communication to encourage robust AI systems
Fairness and accountability
- Transparency is important for fairness
- System fragility and systematic bias can lead to inaccurate results
- Lack of transparency can reduce consumer and public trust
- Explainable models increase accountability for fairness
- Proactive stance on fairness encourages transparency and explainability
- Fairness involves social context to mitigate systemic bias
Discussion
- AI transparency research has multiple clusters with sometimes conflicting goals
- Commonalities and conflicts between AI transparency research clusters can be found in desired ends, associated stakeholders, and utilized means
Desired ends
- Desired ends of stakeholders often conflict
- Power dynamic between stakeholders who choose means of transparency and lay users who cannot
- Developers seek explainability and trust
- Legislators invoke transparency requirements for visibility and equitable outcomes
- Transparency of AI systems is necessary to give comprehensive transparency around sociotechnical systems
- Information on overall goals of system is necessary to give users an honest understanding
Conflicting means
- Human disclosure and system disclosure can achieve transparency goals.
- Deception and information overload can be used to mislead less empowered stakeholders.
- Disclosures can conflict with intellectual property protection needs.
Selection of study attributes
- Fundamental tension between financial results and responsible AI requirements in corporate settings
- Researchers positioned within entities that maintain large AI systems are often uniquely positioned to assess their real-world impacts
- Societal impacts of AI systems may lead researchers to act as interested stakeholders
- OpenAI researchers cite dangers of large language model abuse as motivation for their tiered-release strategy
- Balancing act between researcher interest and desirable research goals
- Measures of effectiveness in transparency vary depending on target measure
- Goal is to empower users, not just assuage their concerns
- Explanations often do not provide causal justification
- Incentive is to persuade users to believe in systems, not demonstrate trustworthiness
- Claims of researchers may differ from their actions
- Precision in terminology and alignment of claims with reality must be prioritized in work
Conclusion
- AI transparency-related works can be clustered into narrower categories
- Terminological imprecision of system transparency needs to be resolved
- Reproducibility and ethical considerations should be asked of authors
- Research community should speak honestly, clearly and unambiguously about transparency
- Attributes of transparency studies should be characterized
- Means for transparency should be discussed