Improving search engine efficiency with emerging memory and storage

People

Collaborators

Search engines are ubiquitous. Although information retrieval via web search engines is mature, a new more critical venue for text search is emerging. In particular, social media platforms, such as Facebbok and Twitter, are driving new avenues for real-time search. In this project, I am actively looking for an honors or a master's student to investigate the use of emerging memories and storage devices, such as Intel Optane memory, and NVMe-based solid state drives, to improve the performance and efficiency of full-text search.  

The student needs to have excellent system building skils.  The student should be extremely comfortable and proficient in the use of Linux OS.  We will tease apart and significantly modify an industrial strength search library namely Lucene.  Lucene is written in the Java programming language.  Proficiency in Java is therefore essential.  The main task for this project would be replacing the backend of Lucene (filesystem-backed database) with an in-memory key-value store.  If all this seems exciting to you, and you are up for a challenging but highly rewarding project, please contact me at Shoaib.Akram@anu.edu.au.  

 

 

 

Funding

ANU College of Engineering and Computer Science (Funding is available for motivated students)

Partners

Google and Twitter

Software

Lucene Search Library

References

https://lucene.apache.org

Earlybird: Real-Time Search at Twitter

Updated:  10 August 2021/Responsible Officer:  Dean, CECS/Page Contact:  CECS Marketing