IJCAI-09 Workshop on
Open Advancement of Question Answering (OAQA-09)
July 11,12, or 13 (one day TBD), 2009
NOTE: THE OAQA WORKSHOP AT IJCAI-09 HAS BEEN CANCELLED
Workshop URL: http://www.lti.cs.cmu.edu/Workshops/OAQA-09
IJCAI 09 URL: http://ijcai-09.org/index.html
Contact: oaqa@isi.edu
Submission Deadline:
Author Notification:
Final Versions:
Question Answering (QA) is an application area of Computer Science which attempts to build software systems that can provide accurate, useful answers to questions posed by human users in natural language (e.g., English).
Although QA is not new, the language processing community has yet to develop a clearly articulated and commonly accepted guiding framework and research methodology, parallel to that of IR, MT, or text summarization. As a result, despite ten years of system evaluations in the TREC QA track for specific kinds of questions and answers, the community does not have a clear idea how much progress was made during that period for QA in general. Part of the problem is an ongoing absence of common evaluation metrics tailored to drive the technology forward along dimensions like breadth, accuracy, speed, confidence estimation, and domain adaptability. Another part of the problem is a lack of transparency and repeatability in the experimental design of QA system evaluations and corresponding publications.
To address these problems, this workshop focuses on the methodology surrounding QA research, and not on specific technical results. The goal of Open Advancement in Question Answering (OAQA) is to accelerate the pace of progress in QA, by establishing methodologies and an environment that will facilitate the development and evaluation of effective core algorithms and the deployment of practical solutions in a sharable and maintainable way. The workshop will seek to form a broad collaboration, involving group consensus on best practices regarding key elements of QA system development, including:
a) a common set of performance metrics;
b) a shared logical architecture for integration and evaluation;
c) a shared deployment architecture for run-time systems;
d) a set of common challenge problems to drive innovation along important dimensions;
e) a collaborative, distributed process for integration, testing and evaluation; and
f) an open collaboration approach to software development, to support transparent, controlled evaluation and straightforward replication of results.
The OAQA-09 Workshop is expected to foster teaming of stakeholders from academia, industry, and government. We plan to work together to establish a broader collaboration that will help to share innovation across the field as a whole, and provide deeper insight into the generality and domain adaptability of emerging QA technologies. We hope to do this by defining clear and useful metrics, a set of challenge problems designed to drive key performance metrics and an open architecture and code-base that can be easily reused to enable interoperability and to measure end-to-end performance as well as individual component impact on task performance.
Submissions
As with all IJCAI workshops, size is limited and attendees must receive an invitation from the organizers. Invited participants will be chosen from the authors of paper submissions. We will accept two kinds of papers: full length papers of no more than 6 pages in the AAAI format, and position papers of 1–3 pages.
Since the workshop will not be a forum for presentation of technical results in QA per se, full length papers must present evaluation metrics, architectural ideas, challenge problems, testing and instrumentation of QA systems or components, as well as high-level visions for QA systems. Technical results may appear in papers if they help to motivate the value of a metric, problem, etc., but should not be the main focus.
Position papers should present ideas, opinions, or criticisms that the authors would like to discuss at the workshop, and should be succinctly presented and well argued.
We
expect many submissions, thus attendees will be selected based on the relevance
of their submission to the workshop goals, and the cogency of the position
taken. In addition, a diversity of
opinions is sought and consideration will be given to new ideas within the
workshop scope.
Organizing Committee
David Ferrucci, IBM Research
Eduard Hovy, USC/ISI
Eric
Nyberg, CMU
Chris Welty, IBM Research
Program Committee (invited)
James Allan, UMass
Amherst
Boris Katz, MIT
Eric Brown, IBM Research
Chris Manning,
Teruko Mitamura, CMU
Bruce Porter, UT
Tomek Strzalkowski, University
at
Ken Barker, UT
Dan Roth, UIUC
Jerry Hobbs, USC/ISI
Dan Oblinger,
DARPA
Danny Bobrow, PARC
Noriko Kando, National Institute of Informatics (NII),
Günter Neumann, DFKI,
Robert Gaizauskas,
Chin-Yew Lin, Microsoft Research
Hwee Tou Ng,
Bernardo Magnini,
Bill Teahan,
Lide Wu, Fudan University, China