Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Tasks in information retrieval have been discussed for a long time.
  • Understanding and addressing users’ tasks is more important than ever.
  • This paper provides perspectives on tasks in IR, bottlenecks, and how to move forward.
  • A tree-like structure is presented to ground ongoing and future research.

Paper Content

Introduction

  • Scholars have argued for the importance of considering task information in information retrieval
  • Several attempts have been made to understand search tasks and use task knowledge to better provide support
  • Search services have incorporated spatial and temporal information to understand or expand a query
  • This paper provides a new foundation for task-related information in IR applications
  • It outlines paths and perspectives for capturing task-related information and using it in IR applications

A brief history of tasks in ir

  • Task is a set of physical, cognitive, and affective actions to achieve goals.
  • Task has been understood in IR-related research in different ways.
  • Specific aspects of task have been investigated in IR-related research.
  • Knowledge of task has been applied in IR.
  • Recent workshops have discussed task in IR.

Overview of approaches to task in ir

  • Early research in IR related to task was focused on understanding what motivated people to search for information
  • Vakkari developed a framework of task-based information searching with 3 stages
  • Tasks are often considered multi-level information seeking processes
  • Different characteristics or facets of tasks influence people’s interaction with intelligent systems
  • Broder proposed 3 types of search engine intent
  • Studies have indicated a close association between searchers’ performance of a task and the information need
  • Tasks involve different activities, have different constraints, and take different amounts of time to complete
  • Recent progress in task intelligence includes discovering digital assistant capabilities, estimating task durations, and automatically tracking task status

Task levels

  • Task contexts in information practices can be represented by a nested model of three levels.
  • Work tasks are separable parts of a person’s duties in their workplace.
  • Everyday life tasks can lead to active information seeking and searching practices.
  • Xie explored a four-level hierarchical framework of goals which covers a wide range of user goals and tasks.

Task facets

  • Previous research has examined the impacts of task types and facets on search interactions.
  • Manipulating task a priori determinability affects users’ perceived task difficulty and choices of search strategies.
  • Task difficulty affects whole-session and within-session search behaviors.
  • Li and Belkin developed a faceted approach to conceptualizing tasks in IR.

Task stages

  • Task process is an aspect of task that is different from static task properties
  • To understand a task, we need to explore objective task features and users’ responses to the task environment
  • Many search process models focus on behavioral aspects and examine transitions of information seeking and search actions
  • Ellis’ model identifies features of information seeking patterns and has been modified and tested empirically

Applying task knowledge in ir

  • Task knowledge can be used to provide better query suggestions
  • Task knowledge can be used to build user models for personalized search and recommendation
  • Task knowledge can be used to predict user satisfaction
  • Task knowledge can be used to improve relevance feedback
  • Task knowledge can be used to automate tasks and reduce user burden
  • Task knowledge can be used to lead users through their search process

Recent research, development, activities

  • Significant interest and activity in task-based IR from research community
  • Several workshops held on task-based IR, focusing on search interactions, searcher intents, and tasks in information search
  • Challenges of task-based and aggregated search discussed, such as mismatch between search interface and specialized task-based functionalities
  • Workshops discussed open research questions, including evaluation and role of context

Task composition and support

  • Explicitly represented tasks are presented as hierarchies, trees, or lists of aspects.
  • Implicit representations use a probability distribution or encoded vectors.
  • Combining explicit and implicit representations offers the best of both worlds.

Tasks as trees

  • A task is composed of subtasks, sub-sub-tasks, and so on.
  • Hierarchical representations are key to task modeling.
  • Search systems can offer support at each level of the hierarchy.
  • Low-level tasks are supported by techniques such as query auto-completion and query suggestion.

Moves

  • Moving left to right in the tree is to predict or suggest the next thing in a sequence
  • Moving up the tree is to recognize a more complex task
  • Moving down the tree is to decompose a task

Challenges

  • Some moves around the tree are easier than others
  • Some actions are unobserved or unobservable
  • Observed actions are sparse signals
  • Challenges include representing tasks, observing more context, and developing task-oriented interfaces

Task modeling

  • Extracting, representing, and applying task information
  • Reviewing possible approaches to modeling and extracting complex task structures

Task representation

  • Tasks can be defined at different granularity levels
  • Represent tasks from different theoretical and methodological perspectives
  • Represent tasks from high level of abstraction to concrete, granular action-oriented level
  • Multiple sub-tasks associated with a user’s information need
  • Bare tree extraction algorithm to extract hierarchical representation of tasks/sub-tasks
  • Vector representation of tasks implicit in search behaviors
  • Triangulate observable search events with other situational and contextual information
  • Visualize user-system interactions as complex graph network structure
  • Sequential heterogeneous graph embedding-based task model to capture structural features of interactive search sessions
  • Model search sessions as Markov Decision Processes, Hidden Markov Models, or Partially Observed Markov Models
  • Apply reinforcement learning approaches to predict or suggest the next action/task

Inferring tasks from observable events

  • Studies have used lexical and content-based features to determine topical and task change in query formulations.
  • Examples of features used include tagging, TF-IDF scoring, term filtering, category terms, query term cosine similarity, URL domain clicked, Jaccard coefficient between clicked URL sets, same session and the number of sessions in between, query reformulations, click entropy, query length, post-click actions, and session lengths, and dwell time for action events.
  • Studies have shown how such signals can indicate the nature of the task being performed, even when there is no explicit statement.
  • Task information applications can help simulate, develop, and evaluate task-aware support.
  • Existing search systems are not good enough for complex and exploratory search tasks.
  • Complex tasks need to be broken down into smaller sub-tasks.
  • Task trees can help reduce cognitive load and focus assistance on the specific task.
  • Searches are performed within a situational context.
  • Task models can be used to improve search engine performance.
  • Task models can take many forms.
  • Task models must evolve over time.
  • Task models can be used to personalize search results and generate query suggestions.
  • Complex tasks can span both time and space.
  • Cross-device searching involves initiating a task on one device and resuming it on another.
  • Task representation needs to be transferable between devices.
  • Multi-device experiences combine the strengths of multiple devices to support complex tasks.

Conversational agents

  • Understanding user intention behind seeking information
  • Addressing unknown unknowns
  • Making zero-query recommendations
  • Being proactive with recommendations
  • Abstracting out from query/question/observation to task/context
  • Leveraging world knowledge
  • Generating recommendations from task/context
  • Learning how to perform a task

Proactive search and recommender systems

  • Ability to identify and extract tasks has implications for search and recommender systems
  • Utilizing knowledge about tasks can improve system offerings
  • Modeling user tasks from observable actions unlocks new directions for solving problems and improving user engagement and satisfaction

Evaluating task-based applications

  • Evaluation is important in task-based search and recommendation systems.
  • The same methodologies used in IR can be used to evaluate task-based systems.
  • Task-based systems consider tasks holistically, spanning multiple queries and/or sessions.
  • Special attention should be given to metrics used to determine task-based system performance.

Methodologies

  • Standard evaluation methods can be used to evaluate task-based systems.
  • Cranfield experiments and TREC have been used to drive progress in tasks research.
  • Evaluation in IR should take a broader view on tasks, users, and context.
  • Living laboratories and mixed methods studies can provide a more complete picture of task performance.
  • Additional context should factor into task-based evaluation.

Metrics

  • Evaluating systems on the basis of search task performance has been explored for decades
  • Conceptualizing tasks and creating task models are important in determining appropriate task-based evaluation metrics
  • Evaluate task-based systems holistically to reach actionable conclusions and understand system performance
  • Popular metrics include task completion time, effort expended, engagement and progress through the task

Task

  • Outcome metrics focus on the product of tasks
  • Examples include task utility, satisfaction, and task success
  • Task utility is affected by task stage and relevance is personal and situational
  • Satisfaction is non-binary and impacted by task and user effects
  • Task success can be modeled based on behavioral signals

Additional considerations

  • Metrics for task-based systems include robustness, privacy, adaptivity, and scalability
  • Task performance is affected by intrinsic and extrinsic properties
  • Metrics must be contextualized and can interact and trade off
  • Task support systems contain multiple components
  • Sets of metrics are used to evaluate task-oriented dialog systems
  • Meta-analysis frameworks can be used to analyze metrics

Task futures

  • IR students and scholars often encounter user tasks in a new light
  • This can lead to advancements or redoing/adding to previous work
  • Time is ripe for a new generation of scholars to support users in accomplishing tasks
  • Future directions and ethical considerations should be discussed

Research threads and directions

  • Provide mechanisms to capture tasks for machine learning models and to give users control over what information is shared.
  • Find ways to uncover more events related to the task process.

Ethical considerations

  • Low information literacy can lead to users not being able to fully utilize available information or tools
  • Users often don’t know what they don’t know
  • Task-based IR systems can help users, but ethical considerations must be taken into account
  • Rapid development of AI techniques can lead to systems being less and less trustworthy
  • Feedback loops between systems and users can lead to bias in datasets
  • System designers must prioritize which tasks they support, raising questions about fairness
  • Task modeling must be done with ethical issues in mind