Computers Say ‘No’ But AI’s Decisions Must Be Fair & Transparent

American teachers have prevailed in a lawsuit with their school district over a computer program that assessed their performance.

The system rated teachers in Houston by comparing their students’ test scores against state averages. Those with high ratings won praise and even bonuses. Those who fared poorly faced the sack.

The program did not please everyone. Some teachers felt that the system marked them down without good reason. But they had no way of checking if the program was fair or faulty: the company that built the software, the SAS Institute, regards its algorithm a trade secret and would not disclose its workings.

The teachers took their case to court and a federal judge ruled that use of the EVAAS (Educational Value Added Assessment System) program may violate their civil rights. In settling the case, the school district paid the teachers’ fees and agreed to stop using the software.

The law has treated others differently. When Wisconsin police arrested Eric Loomis in 2013 for driving a car used in a shooting, he was handed a hefty prison term in part because a computer algorithm known as Compas judged him at high risk of re-offending. Loomis challenged the sentence because he was unable to check the program. His argument was rejected by the Wisconsin supreme court.

The arrival of artificial intelligence has raised concerns over computerised decisions to a new high. Powerful AIs are proliferating in society, through banks, legal firms and businesses, into the National Health Service and government. It is not their popularity that is problematic; it is whether they are fair and can be held to account.

Researchers have documented a long list of AIs that make bad decisions either because of coding mistakes or biases ingrained in the data they trained on.

Bad AIs have flagged the innocent as terrorists, sent sick patients home from hospital, lost people their jobs and car licences, had people kicked off the electoral register, and chased the wrong men for child support bills. They have discriminated on the basis of names, addresses, gender and skin colour. 

Bad intentions are not needed to make bad AI. A company might use an AI to search CVs for good job applicants after training it on information about people who rose to the top of the firm. If the culture at the business is healthy, the AI might well spot promising candidates, but if not, it might suggest people for interview who think nothing of trampling on their colleagues for a promotion. 

Opening the black box
How to make AIs fair, accountable and transparent is now one of the most crucial areas of AI research. Most AIs are made by private companies who do not let outsiders see how they work. Moreover, many AIs employ such complex neural networks that even their designers cannot explain how they arrive at answers. The decisions are delivered from a “black box” and must essentially be taken on trust. That may not matter if the AI is recommending the next series of Game of Thrones. But the stakes are higher if the AI is driving a car, diagnosing illness, or holding sway over a person’s job or prison sentence.

Recently, the AI Now Institute at New York University, which researches the social impact of AI, urged public agencies responsible for criminal justice, healthcare, welfare and education, to ban black box AIs because their decisions cannot be explained. 
“We can’t accept systems in high stakes domains that aren’t accountable to the public,” said Kate Crawford, a co-founder of the institute. The report said AIs should pass pre-release trials and be monitored “in the wild” so that biases and other faults are swiftly corrected.

Tech firms know that coming regulations and public pressure may demand AIs that can explain their decisions, but developers want to understand them too. 

Klaus-Robert Müller, professor of machine learning at the Technical University of Berlin, has trained an AI to diagnose breast cancer using variety of medical data. It is not good enough for the AI to simply spit out a diagnosis, he says. “It’s absolutely mandatory for the individual patient to know what the heck is going on.”

To understand how their AI reached decisions, Müller and his team developed an inspection program known as Layerwise Relevance Propagation, or LRP. It can take an AI’s decision and work backwards through the program’s neural network to reveal how a decision was made. 

Instead of exposing the full inner workings of an AI, it figures out what it would take to change the AI’s decision. Suppose an AI turns down a mortgage applicant. A researcher at Oxord Univerity, Dr Sandra Wachter, has formulated Wachter’s method which when used correctly might reveal that the loan was denied because the person’s income was £30,000, but would have been approved if it was £45,000. It allows the decision to be challenged and informs the person what needs to change to get the loan. 

For some researchers, the time to start regulating AI has arrived. “We have seen too many slip-ups, and AI is too powerful not to have government be part of the solution,” said Craig Fagan, policy director at Tim Berners-Lee’s Web Foundation. “It’s asking companies to take on a lot of responsibility to manage such rapid economic, political and social transformation and not have some government oversight.” 

Along with Luciano Floridi and Brent Mittelstadt at the Oxford Internet Institute, Wachter has called for a European AI watchdog to police the technology. The body would need powers to send independent investigators into organisations to scrutinise their AIs and extract meaningful explanations. 

To keep people safe, AIs could be certified for use in critical arenas such as medicine, criminal justice and driverless cars. “If we’re deploying them in critical infrastructure, we need to be sure they meet safety standards,” Wachter said.
“We need transparency as far as it is achievable, but above all we need to have a mechanism to redress whatever goes wrong, some kind of ombudsman,” said Floridi. “It’s only the government that can do that.”

Guardian

You Might Also Read: 

Artificial Intelligence Needs Regulation:

The AI Apocalypse:

Computer Says No:
 

« N. Korean Hackers Plan to Devastate UK
Management Coverup At Uber After 57m Customers Hacked »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

Perimeter 81 / How to Select the Right ZTNA Solution

Perimeter 81 / How to Select the Right ZTNA Solution

Gartner insights into How to Select the Right ZTNA offering. Download this FREE report for a limited time only.

Jooble

Jooble

Jooble is a job search aggregator operating in 71 countries worldwide. We simplify the job search process by displaying active job ads from major job boards and career sites across the internet.

LockLizard

LockLizard

Locklizard provides PDF DRM software that protects PDF documents from unauthorized access and misuse. Share and sell documents securely - prevent document leakage, sharing and piracy.

Clayden Law

Clayden Law

Clayden Law advise global businesses that buy and sell technology products and services. We are experts in information technology, data privacy and cybersecurity law.

CSI Consulting Services

CSI Consulting Services

Get Advice From The Experts: * Training * Penetration Testing * Data Governance * GDPR Compliance. Connecting you to the best in the business.

Efecte

Efecte

Efecte is a Nordic SaaS company specialized in IT Service Management, Self-Service, Identity Management and Access Governance solutions.

Guardian360

Guardian360

The Guardian360 platform offers unrivalled insight into the security of your applications and IT infrastructure.

Secucloud

Secucloud

Secucloud GmbH is a provider of high-availability cyber-security solutions, offering a cloud-based security-as-a-service platform, particularly for providers.

Vesta

Vesta

Vesta Corporation is a global provider of a scalable suite of fraud and payment solutions for online commerce.

CIRISK

CIRISK

CIRISK offers a wide range of services from consulting to audit or project management to help you develop your cyber security or information security strategy.

Ashley Page

Ashley Page

Ashley Page offer a unique cyber insurance and risk management solution - Cyber+Insure.

Cyber Security Advisor

Cyber Security Advisor

Notice how sophisticated the cybersecurity market is. Think how would you pick the security provider, assess your company, and be sure of your security decisions? Cyber Security Advisor is the answer!

FAIR Institute

FAIR Institute

The FAIR Institute is a non-profit professional organization dedicated to advancing the discipline of measuring and managing information risk.

National Cyber Security Center (NCSC) - Vietnam

National Cyber Security Center (NCSC) - Vietnam

National Cyber Security Center of Vietnam has a central monitoring function and is a technical focal point for monitoring and supporting information security for people, businesses and systems.

Cyberwatch Finland

Cyberwatch Finland

Cyberwatch Finland's services improve decision-makers’ strategic situational picture and enable successful holistic cyber risk management.

Searchlight Cyber

Searchlight Cyber

Searchlight Cyber is a leading darknet intelligence company. Working with law enforcement, industry, and end users to help protect society against the threats of the darknet.

SecOps Group

SecOps Group

SecOps Group is a boutique cybersecurity consultancy helping enterprises identify & eliminate security risks on a continuous basis.

Obsidian Security

Obsidian Security

Protect your business-critical applications by mitigating threats and reducing risk with Obsidian, the first truly comprehensive security solution for SaaS.

Sri Lanka CERT

Sri Lanka CERT

Sri Lanka CERT is the National Centre for Cyber Security, which has the national responsibility of protecting the nation’s cyberspace from cyber threats.

Acclaim Technical Services (ATS)

Acclaim Technical Services (ATS)

ATS provide operational products, services and solutions to the defense and intelligence communities for all types of critical mission needs.

Incode

Incode

Incode is the leading provider of world-class identity solutions that is reinventing the way humans authenticate and verify their identities online.