This month, the Microsoft AI for Accessibility program is calling for project proposals that advance AI-powered innovations in education that will empower people with disabilities. Through a two-part series, we are highlighting projects we are supporting.
Imagine you are performing exceedingly well in high school, only to find out that you will have to take one of the most prestigious university entrance exams with a reader who does not fully understand the technical content. This is the story of thousands of students in Kenya and India who are not provided an equal opportunity to pursue the academic courses of their choice.
The majority of students who are blind or low vision in countries in Africa, Latin America, and parts of Asia and the Middle East rely on in-person readers and writers to take exams and complete homework. However, this leads to several detrimental impacts on performance, including lack of availability of test readers and writers with domain-specific knowledge, frustrations of test readers and writers during long exams or with unfamiliar material, poor working relationships due to confusion, and lack of communication, among others. Furthermore, local universities have been forced to focus more on technology and reduce face-to-face accommodations due to COVID-19 protocols.
inABLE, a nonprofit organization in Kenya that empowers the blind and low vision through computer assistive technology, partnered with I-STEM, a not-for-profit company in India that develops STEM technical solutions, received a Microsoft AI for Accessibility grant to revolutionize how blind students take exams.
“The I-Assistant is an artificial intelligence (AI) solution that gives blind and low vision people an alternative to an in-person test reader and writer with an automated tech-powered conversational experience,”, explains Irene Mbari-Kirika, Founder and Executive Director of inABLE. I-Assistant leverages a custom model built on top of Azure Computer Vision, particularly Text Extraction (OCR), that detects various document structures and layouts (e.g., tables, headings) and processes text, including math expressions. Then a readable and accessible document is created – thereby converting any exam into an accessible, digital format. Using Azure Custom Speech Recognition, Text-to-Speech, and Language Understanding (LUIS), users can intelligently navigate the exam or document and perform context-aware actions, like checking exam status by asking questions such as “return to the question I skipped”, “how many questions have I answered?”, or “how much time do I have left?”
Kartik Sawhney, I-Stem cofounder, emphasizes the strengths of the AI through this work: “We are very pleased that we were able to build a fully functioning prototype with multiple AI-powered components coming together to provide a natural conversational experience to the users. What is even more interesting is that many of the individual components such as OCR and custom speech by themselves have great promise.”
To further promote research into content accessibility that was a core component of the I-Assistant project, the project team is sharing an open-source OCR and document analysis dataset so that other teams can build new innovations on top of it.
I-Assistant’s natural conversational experience considers the specific needs of the blind community while interacting with the computer, such as navigating fast and effectively and accurate speech-to-text. The system provides the blind or low vision student with the same experience as a sighted person, with the added advantages of consistency, accuracy, and comfort. I-Assistant helps blind and low vision students take tests more independently – a true testament to empowering people with disabilities.
