Screen Agent: A Voice-First Browser Extension for Accessible Web Navigation Using Local AI

Author:

Shan Htet San

Co-author:

Mentor:

Hao Tang
Jiawei Liu

Abstract:

For most, web browsing has opened a world of information at their
fingertips. However, for a blind and low-vision (BLV) user, the experience
of web browsing feels more like searching for a needle in a haystack while
a robotic voice reads the entire stack out loud. Web readers can be slow
and cognitively exhausting. How can AI help BLV users navigate the web
more efficiently, without depending on how accessible a website was built
to be?
Closely working with a visually impaired individual, we introduce
Screen Agent, an accessible Chrome extension that leverages AI to transform
web browsing into conversational search. By extracting page structure,
Screen Agent enables conversational navigation, allowing users to explore
the web pages simply by chatting with an AI assistant. It includes an
always open assistant panel, the ability to switch between web pages
without losing your conversation, adjustable reading speed, and works
even on older computers without advanced graphics hardware. Moreover,
it supports local AI that ensures data privacy and prevents data leakage.
As a pilot study, this work explores the shift to AI-driven browsing,
providing a more convenient and efficient way for BLV users to interact
with the web by removing the typical hurdles of manual screen reading.