On the Moral Hazard of Autonomy

This paper describes the concept of moral hazard as applied to technologies that incorporate automation and autonomy. Moral hazard is said to exist when a party to a transaction feels more comfortable taking undue risks because another party will bear the costs if things go badly. As opposed to regular physical hazards, a moral hazard comes from within a person. In this paper, we reveal two categories of moral hazards related to autonomy. The first category of moral hazard occurs when the owner of the autonomy introduces an autonomous system without accepting the full responsibility for improper operation thereby shifting the risks from one party to another party. This category of moral hazard is similar to moral hazards experienced in other industries and can often be addressed through appropriate policy and establishing liability for irresponsible behavior. The issue becomes more complicated in cases where the operator of the autonomy may not have a full understanding of the system behavior. In the second category of moral hazard, risks are shifted from people to autonomy. In this category, the humans in proximity to the autonomous system begin to trust its behavior. Their behavior may change in that they may believe they are more insulated from harm and subsequently exhibit more risky behavior towards increasingly autonomous technologies. Mitigating this type of moral hazard may require the autonomy to possess certain design features to discourage this type of harmful human behavior so that humans do not suffer needlessly in their interactions with autonomous systems by placing inappropriate trust where that trust is neither warranted nor deserved.