Overview of Database Management Systems: Information Retrieval Systems and Differences, Slides of Database Management Systems (DBMS)

Download Overview of Database Management Systems: Information Retrieval Systems and Differences and more Slides Database Management Systems (DBMS) in PDF only on Docsity! Database Management Systems, R. Ramakrishnan 1 Introduction to IR Systems: Supporting Boolean Text Search Chapter 27, Part A Database Management Systems, R. Ramakrishnan 2 Information Retrieval A research field traditionally separate from Databases • Goes back to IBM, Rand and Lockheed in the 50’s • G. Salton at Cornell in the 60’s • Lots of research since then Products traditionally separate • Originally, document management systems for libraries, government, law, etc. • Gained prominence in recent years due to web search Database Management Systems, R. Ramakrishnan 3 IR vs. DBMS Seem like very different beasts: Both support queries over large datasets, use indexing. • In practice, you currently have to choose between the two. Expect reasonable number of updates Read-Mostly. Add docs occasionally SQLKeyword search Generate full answerPage through top k results Structured dataUnstructured data format Precise SemanticsImprecise Semantics DBMSIR Database Management Systems, R. Ramakrishnan 4 IR’s “Bag of Words” Model Typical IR data model: • Each document is just a bag (multiset) of words (“terms”) Detail 1: “Stop Words” • Certain words are considered irrelevant and not placed in the bag • e.g., “the” • e.g., HTML tags like <H1> Detail 2: “Stemming” and other content analysis • Using English-specific rules, convert words to their basic form • e.g., “surfing”, “surfed” --> “surf” Database Management Systems, R. Ramakrishnan 5 Boolean Text Search Find all documents that match a Boolean containment expression: “Windows” AND (“Glass” OR “Door”) AND NOT “Microsoft” Note: Query terms are also filtered via stemming and stop words. When web search engines say “10,000 documents found”, that’s the Boolean search result size (subject to a common “max # returned’ cutoff). Database Management Systems, R. Ramakrishnan 6 Text “Indexes” When IR folks say “text index”… • Usually mean more than what DB people mean In our terms, both “tables” and indexes • Really a logical schema (i.e., tables) • With a physical schema (i.e., indexes) • Usually not stored in a DBMS • Tables implemented as files in a file system • We’ll talk more about this decision soon Database Management Systems, R. Ramakrishnan 13 Fancier: Phrases and “Near” Suppose you want a phrase • E.g., “Happy Days” Different schema: • InvertedFile (term string, count int, position int, DocURL string) • Alternative 3 index on term Post-process the results • Find “Happy” AND “Days” • Keep results where positions are 1 off • Doing this well is like join processing Can do a similar thing for “term1” NEAR “term2” • Position < k off Database Management Systems, R. Ramakrishnan 14 Updates and Text Search Text search engines are designed to be query-mostly: • Deletes and modifications are rare • Can postpone updates (nobody notices, no transactions!) • Updates done in batch (rebuild the index) • Can’t afford to go off-line for an update? • Create a 2nd index on a separate machine • Replace the 1st index with the 2nd! • So no concurrency control problems • Can compress to search-friendly, update-unfriendly format Main reason why text search engines and DBMSs are usually separate products. • Also, text-search engines tune that one SQL query to death! Database Management Systems, R. Ramakrishnan 15 { DBMS vs. Search Engine Architecture The Access Method Buffer Management Disk Space Management OS “The Query” Search String Modifier Simple DBMS} Ranking Algorithm Query Optimization and Execution Relational Operators Files and Access Methods Buffer Management Disk Space Management Concurrency and Recovery Needed DBMS Search Engine Database Management Systems, R. Ramakrishnan 16 IR vs. DBMS Revisited Semantic Guarantees • DBMS guarantees transactional semantics • If inserting Xact commits, a later query will see the update • Handles multiple concurrent updates correctly • IR systems do not do this; nobody notices! • Postpone insertions until convenient • No model of correct concurrency Data Modeling & Query Complexity • DBMS supports any schema & queries • Requires you to define schema • Complex query language hard to learn • IR supports only one schema & query • No schema design required (unstructured text) • Trivial to learn query language Database Management Systems, R. Ramakrishnan 17 IR vs. DBMS, Contd. Performance goals • DBMS supports general SELECT • Plus mix of INSERT, UPDATE, DELETE • General purpose engine must always perform “well” • IR systems expect only one stylized SELECT • Plus delayed INSERT, unusual DELETE, no UPDATE. • Special purpose, must run super-fast on “The Query” • Users rarely look at the full answer in Boolean Search Database Management Systems, R. Ramakrishnan 18 Lots More in IR … How to “rank” the output? I.e., how to compute relevance of each result item w.r.t. the query? • Doing this well / efficiently is hard! Other ways to help users paw through the output? • Document “clustering”, document visualization How to take advantage of hyperlinks? • Really cute tricks here! How to use compression for better I/O performance? • E.g., making RID lists smaller • Try to make things fit in RAM! How to deal with synonyms, misspelling, abbreviations? How to write a good web crawler?