Japan has the highest percentage of population deemed elderly in the world (i.e., people over the age of 65). The portion is predicted to reach 30% by 2025. An aging society poses various societal challenges including societal isolation. Mobile devices, such as smartphones and tablets, can play an important role in connecting elderly persons with family members and can provide cost-effective daily services and opportunities to join community activities. However, lack of experience with mobile devices often prevents elderly persons from adopting such services. It is thus necessary to assist the elderly by giving them answers to questions they may have about their devices and services, to enable large-scale deployment of mobile devices to the aging population. In this study, we evaluated a speech-based question-and-answer system that we designed for elderly novice users of mobile devices through a pilot study on an eldercare service platform with 1,011 elderly participants who were encouraged to use health-check services, video telephony, and other services on a tablet device. The participants could ask questions about the services, the application, and device by using speech whenever they wanted. The results suggest the feasibility of using speech-based interfaces as the main interaction medium for the elderly.