H. Safety and control issues for AI
(RFI question 3)

Last updated July 28, 2016

To reap the societal benefits of artificial intelligence, we will first need to trust it. That trust will be earned through experience, of course, in the same way we learn to trust that an ATM will register a deposit, or that an automobile will stop when the brake is applied. Put simply, we trust things that behave as we expect them to. But trust will also require a system of best practices that can guide the safe and ethical management of AI; a system that includes alignment with social norms and values; algorithmic accountability; compliance with existing legislation and policy; and protection of privacy and personal information. IBM is in the process of developing this system in collaboration with our partners, university researchers, and competitors.

Fears of the success of AI are greatly exaggerated. Notwithstanding the real and dramatic recent successes demonstrated by industry and academia, today’s systems are easy to turn off. Still computer science, offers several types of architectural and algorithmic safe guards for the future as systems become more truly autonomous and perhaps even someday algorithmic self-awareness.

Safe guards: With various colleagues inside and outside IBM we have identified a long-term focus for AI research: developing cognitive assistants with professional facility in deep domains. Of great interest are agents whose behaviors can, by construction, be confined within certain envelopes that are consistent with correctness criteria. We have identified four key attributes of such agents -- knowing deeply, reasoning with purpose, learning continuously and interacting naturally. At the core of a reference architecture for this approach is a reactive executive controller (REC), a component that accepts input from the environment, directs an ensemble of computational agents, and synthesizes from their output a response to the environment. We argue that techniques from synchronous programming should be used to program the REC. Synchronous programming offers a very rich theory of program behavior and correctness, developed over several decades, established in the arena of embedded computing. They support a notion of multiform time, by which programs can be written parameterized on a notion of time that is in fact controllable compositionally by another program. We believe such a framework (as realized in languages such as TCC) offers the right setting in which to realize ideas of interruptibility proposed recently as a way of safely interrupting a learning agent without permitting it to learn around the interruption.

Trust: AI systems will function and work together with humans, often making or suggesting decisions to them, and understanding their decisions. To follow the system’s suggestions, humans must trust the system. This requires developing capabilities for explaining the system’s behavior in ways that can be understood by other humans. Just as for humans, this may well require developing explicit rationalization capabilities that can abstract away from details of internal neural implementations to communicate (in natural language, exploiting rigorous reasoning, logic and mathematics) at the right level of generality.

Ethical Norms: Deep, profound questions about what ethics and morality (should) mean for agents need to be investigated. (Agents do not intrinsically have notions of pain, pleasure and perpetuation of the species –forces that ultimately drive the need for biological systems to interact, cooperate and develop norms.) Cultural influences significantly affect the behavior of humans: how should these be taken into account when designing/developing AI systems that are intended to work across the globe, with people from very different cultures and value systems?

A specific sub-issue under this heading is the political role that AI might play. As noted above, AI has huge potential in helping with crime fighting. However, the definition of crime in some countries veers way from theft, fraud, personal harm, anti-social behavior and so on, into political dissidence. There is a risk, in other words, that AI may be used by some regimes to further repressive purposes as well as reduce “regular” crimes. One approach to addressing this issue might be some form of international treaty that would outlaw such uses; it should also be the responsibility of AI suppliers to understand exactly how their customers want to use their technologies and respond appropriately.

Social Norms: In the future we expect AI systems to present as individuals with personality, history and character, interacting with other humans, building relationships, trusting and capable of being trusted, persuading and capable of being persuaded, getting things done together. Humans expect certain level of social norms, behaviors from other individuals they are interacting with. Developing AI systems that are skilled socially represents a deep and serious challenge.

Algorithmic Transparency: This is about knowing why a system behaves as it does and therefore what it is likely to do in the future. One issue with neural networks, another form of AI, is that you cannot figure out "why" it comes up with a specific output. This is a differentiator for knowledge-based cognitive, as it can tell you why it is saying something - what evidence it used in reasoning and inference chains.Also, data provenance transparency is needed as well since addressing emergent privacy intrusions from combining otherwise non-problematic data sets will be a challenge. Bias is data comes in many forms, and freedom from bias since it could be high qualitybut inherently biased.

Unexpected Interactions: It is not just AI "running amok". Cognitive systems will produce conclusions and initiate actions that are beyond humans - that raises the possibility that two interacting systems, individually working as designed, could collectively behave in unanticipated or dangerous ways.Also an issue about applications and who gets access for what kinds of mischief. For example, the crime-detection capabilities referenced above could just as easily be used to detect activity by political dissidents, by some unpleasant regime. Do we want to be seen to be supporting that?

Back to summary.