Improving search engine efficiency with emerging memory and storage
People
Lead
Collaborators
Search engines are ubiquitous. Although information retrieval via web search engines is mature, a new more critical venue for text search is emerging. In particular, social media platforms, such as Facebbok and Twitter, are driving new avenues for real-time search. In this project, I am actively looking for an honors or a master's student to investigate the use of emerging memories and storage devices, such as Intel Optane memory, and NVMe-based solid state drives, to improve the performance and efficiency of full-text search.
The student needs to have excellent system building skils. The student should be extremely comfortable and proficient in the use of Linux OS. We will tease apart and significantly modify an industrial strength search library namely Lucene. Lucene is written in the Java programming language. Proficiency in Java is therefore essential. The main task for this project would be replacing the backend of Lucene (filesystem-backed database) with an in-memory key-value store. If all this seems exciting to you, and you are up for a challenging but highly rewarding project, please contact me at Shoaib.Akram@anu.edu.au.
Funding
ANU College of Engineering and Computer Science (Funding is available for motivated students)
Partners
Google and Twitter
Software
Lucene Search Library
References
Earlybird: Real-Time Search at Twitter