Test Suites in Grammar Engineering: Evaluation and Usage, Study notes of Linguistics

Download Test Suites in Grammar Engineering: Evaluation and Usage and more Study notes Linguistics in PDF only on Docsity! Grammar Engineering April 11, 2005 Test suites Overview • General announcements • Evaluation in computational linguistics • Uses of test suites • Evaluating precision grammars • Levels of adequacy • Test suite software ([incr tsdb()] demo) • More matrix tour Also by Wednesday • Vocabulary • Case: overt case or not, ergative or accusative, cases on core arguments, where case is marked morphologically, any funnyness (case changes with definiteness, etc) • Agreement: person, number, gender, definiteness; subject-verb, object-verb, determiner-noun, adjective-noun • If your language has neither case nor agreement, talk to me about something else to code ... So: testsuites feasible or no? Evaluation and computational linguistics • Why is evaluation so prominent in computational linguistics? • Why is it not so prominent in other subfields of linguistics? • What about CS? Uses of testsuites • How far do I have left to go? • Internal metric • Objective comparison of different systems → How do you evaluate precision grammars? • Where have I been? • Regression testing • Documentation Levels of adequacy (1/2) • grammaticality • “right” structure • “right” dependencies • exact match semantics Levels of adequacy (2/2) • Only legitimate parses (how do you tell?) • Some set of parses including the preferred one • Preferred parse only/first • Preferred parse within first N Test suite software • What does the LKB batch parse utility do? • What else would you like it to do? • → [incr tsdb()] demo