Projects / Computer Science / Computer Science project 45

Search Engine from Scratch: Crawler, Index and Ranking

Approx. component budget: Rs. 0 - 1,500

Objective

Build the core pieces of a search engine and judge result quality with real queries.

What you will build

A search engine for a chosen site or topic that crawls pages, builds an inverted index, ranks results with TF-IDF and BM25, and shows snippets in a web interface.

Results you can expect

• Index size and build time for 10
• 000 pages
• Query time in milliseconds
• Precision@10 on 20 test queries
• And an improvement from adding BM25 over simple counting

What you will learn

Crawling politely, text processing, inverted indexes, ranking functions, and evaluating search.

Tools and notes

Python, requests, BeautifulSoup, SQLite or a simple index file, Flask. Obey robots.txt and crawl only sites you are allowed to.

Resume points from this project

• Built and tested Search Engine from Scratch: Crawler, Index and Ranking; objective: build the core pieces of a search engine and judge result quality with real queries.
• Measured and analysed index size and build time for 10 and 000 pages; wrote the report and presented the results in a viva. Add your own numbers: [x].

Replace [x] with your own measured results. Write only what you really did.

More computer science projects

Back to all projects