Ethics risk toolkit

The AI programmer’s dilemma
Justin Moore, AI, Autonomous Cars and Moral Dilemmas, Techcrunch.com, 19 October 2016

“Take, for example, an autonomous car self-driving along the road when another car comes flying through an intersection. The imminent t-bone crash has a 90 percent chance of killing the self-driving car’s passenger, as well as the other driver. If it swerves to the left, it’ll hit a child crossing the street with a ball. If it swerves to the right, it’ll hit an old woman crossing the street in a wheelchair.”

Here the programming of the AI within the car determines the outcome even before the driver has turned the key to start the engine. That programming does not involve a split-second reaction as the outcome follows from the programming, determined at the time of coding. The programming itself therefore involves an ethical judgment.

Shift in ethical responsibility

Because the introduction of AI has the potential to shift responsibility up the supply chain to the manufacturer or to another AI Supply Chain Participant, in the case of, say, a driverless car, it will be the manufacturer (or other responsible AI Supply Chain Participant), rather than the driver, who will be forced to make new ethical judgments.

Same facts, same result

In State v Loomis¹ the Wisconsin Supreme Court  recently approved a trial court’s use of an algorithmic risk assessment in relation to the assessment of risk of criminal recidivism for sentencing purposes.  Historically there has been a wide disparity in sentencing terms for the same offences, in part because of subjective and sometimes wildly varying human prejudices when applying sentencing guidelines. An argument in favour of using such AI is that, based on the same set of facts, it produces the same sentencing term, and so achieves consistency of treatment.

What does the Process stage involve?

A process imposed in relation to decision-making undertaken by AI involves:

• Ethics Board: with supervisory power over all ethical decisions. It includes external members;
• Ethics Policy: created by the Ethics Board, it sets out rules and principles for ethical decision-making in the creation of an AI system. Such a policy can govern “how the AI should be used, who is qualified to use it, what training is required for operators, and what operators and other people can expect from the AI”;⁴
• Compliance by design: amending the interface between ethics and the rest of the AI system to ensure compliance with the Ethics Policy, such as by creating “privacy by default” or by introducing “kill switches”. Businesses should mandate consideration of ethics at the pre-product design stage.⁵ For example, it is a realistic goal to embed explicit norms into such systems, because norms can be considered instructions to act in defined ways in defined contexts;⁶
• Industry standards, codes of conduct and certification: technical industry standards (such as ISO 10218-1:2006 promulgated by the International Standards Organization, which covers safety-associated design, protective measures, and industrial robot applications); codes of conduct (such as those put forward by the European Parliament for robotics engineers and research ethics committees, along with model licences for designers and users);⁷ and external certification of compliance with standards and code. The Ethics Policy can incorporate such requirements by reference; 
• Ethics Compliance Log: an audit trail designed to deliver accountability (see What Steps are Required to Deliver Accountability?, above). The log records compliance with the Ethics Policy. Logs may cover computer code, algorithms, behaviour, or tests. In each case, they will show how that functionality was evaluated for ethical compliance, which may then be tested in the Validation stage (see Validation);
• ethics-bounded optimisation: assuming that ethical principles are embedded in any AI system, it is then necessary to ensure that the AI itself does not learn to work around them - particularly where the system is an ANN. That is, if ethical principles are treated simply as additional constraints on behaviour, the AI may learn to circumvent those constraints. Any learning process must give ethical principles additional power to control the optimisation carried out by the AI over and above other constraints or rewards;⁸
• Ethics training programme: to disseminate the Ethics Policy and associated guidelines. “Ethical training for AI is a necessary part of the solution.”⁹

How is Validation implemented?

There are two broad approaches to Validation:

• intrinsic validation: examining the rules governing ethical behaviour to confirm that they comply with the Ethics Policy. This is appropriate for an expert-system architecture and could be carried out by lawyers and associated professionals. Generally speaking, if there is transparency in relation to the algorithm, intrinsic validation may be possible; on the other hand, if there is no such transparency, then extrinsic validation may instead need to be used;
• extrinsic validation: analysing the behaviour of the AI system in a real-world environment or simulation to measure its compliance with the outcomes expected from ethical judgments in accordance with the Ethics Policy. This effectively treats the system as a “black box” and tries to infer compliance from observation of its behaviour. It requires an interdisciplinary team with particular emphasis on statistical knowledge. This may be the only option for ANN systems (where interrogation of the underlying structure in a meaningful way is not possible).

Validation of an AI system operating within a wider IT eco-system which includes other AI systems can give rise to particular problems. An AI system might work well on its own, but may have unexpected interaction effects in relation to other AI systems. Peter McBurney, Professor of Computer Science at King’s College London, gives this example:

“Suppose that driverless cars communicate with CCTV cameras to detect the presence of pedestrians, and both sets of devices communicate with traffic lights to optimise traffic flow. Something goes wrong and a driverless car hits a pedestrian. Each intelligent device was working perfectly. Some system-level interaction property instead caused a problem.”

AI that is designed for “situatedness” may give rise to additional challenges in terms of Validation (and attribution of liability). “Situatedness” connotes the idea that successful operation of an AI system may require a sophisticated awareness of the environment within which the system operates, including predictions (and models) about how other entities in the environment are likely to act. Testing other than in a live environment would produce a less than complete picture of the performance of the AI system.

The above example illustrates that intrinsic Validation of an AI system will not typically be sufficient. Extrinsic Validation will also be required.

So-called “automation bias” will need to be measured and corrected as part of the Validation stage where humans are inserted into a decision-making process involving AI (see Inserting Humans into the Loop).

Automation bias is the name given for the problem that humans tend to: (1) give undue weight to the conclusions presented by automated decision-makers; and (2) ignore evidence that suggests a different decision should be made.

For example, an automated diagnostician may assist a GP reach a diagnosis, but it may also have the effect of reducing the GP’s own independent competence as he or she may be unduly influenced by the automated diagnostician. The result may be that the overall augmented system may not be as much of an improvement as had been expected.