US school's AI system mistakes an empty Doritos bag for a gun, here’s what happened next

An artificial intelligence (AI) based detection system at a US school mistakenly identified a 16-year-old student's empty Doritos bag as a gun. This led to a confrontation between the student and officers.

The student, Taki Allen of Kenwood High School in Baltimore, Maryland, had crumpled an empty Doritos bag and put it in his pocket after eating it. Nearly 20 minutes later, while he was waiting to be picked up after football practice, officers approached him. After that, officers searched Allen and located the empty chip bag. An officer then showed Allen an image from the school's AI detection system, explaining that the crumpled bag appeared to be a firearm.In a statement to WMAR2 News, Allen said: “Police showed up, like eight cop cars, and then they all came out with guns pointed at me, talking about getting on the ground. I was putting my hands up like, 'What's going on?' He told me to get on my knees, arrested me and put me in cuffs.”

What the school said about the AI system incident

The school’s superintendent Dr Myriam Rogers, said the system doesn’t have any problems and functions as intended."The program is based on human verification, and in this case, the program did what it was supposed to do, which was to signal an alert and for humans to take a look to find out if there was cause for concern in that moment," Rogers said.In a letter sent to parents, the school's principal Kate Smith wrote: "Please know that ensuring the safety of our students and school community is one of our highest priorities."

However, Allen claimed that Smith did not speak to him until three days after the event.“I was expecting them to at least come up to me after the situation or the day after, but three days later that just shows like, do you really care or are you just doing it because the superintendent called me," he said.He told WMAR-2 News that when Smith finally approached him, she asked how he was and told him to reach out if he needed anything.

Allen said the experience has left him uneasy at school."Now, I feel like sometimes after practice I don't go outside anymore. Cause if I go outside, I don't want - don't think I'm safe enough to go outside, especially eating a bag of chips or drinking something. I just stay inside until my ride comes," he noted.He believes the school should use more reliable technology to prevent similar situations. Superintendent Rogers stated she plans to review the alert process and consider possible improvements.“I don't think no chip bag should be mistaken for a gun at all," Allen added.