Simple Math

“It’s just simple math: one deceased male, murdered by one defective robot,” said James Draper.

“Maybe killed. Not murdered, this robot is incapable of murder,” Reid gestured towards the robot. “Look, some of these Commercial Floor Maintenance drones are over twenty years old. They’re designed to clean floors, that’s it. Very basic, almost analog stuff -- nothing smart. They simply aren’t built to make the complex decisions required to murder a human in any scenario.”

The two men were assigned to the Artificial Intelligence Crimes Unit. James Draper was a public defense attorney, and had only a vague notion of the generalities of Artificial Intelligence -- but was a damn fine lawyer all the same. Carter Reid, PhD was the department’s only Forensic Roboticist. It was Reid’s job to explain the finer points of A.I. to the lawyers.

Draper sighed. “Ok, then. Let’s watch it again.” He restarted the surveillance video, and the men leaned into a video screen that displayed four panels. Each panel showed a subway terminal platform, recorded from different angles. Something that resembled a large, upside down trash can moved back-and-forth in slow lines over the white tile. From some angles, large circular brushes could be seen rotating underneath the moving shape.

A fifth frame displayed white text, scrolled against a black background:

{“2045-05-20T04:45:55Z” : “suspect CFM-2 #0000ff observed cleaning floors”}

{“2045-05-20T04:46:00Z” : “suspect CFM-2 observed cleaning floors”}

The inverted trash can shape on the screen continued to move in slow lines, surrounded by a blue bounding rectangle that followed it across the frames.

{“2045-05-20T04:46:05Z” : “subject JULIA GARCIA #FFA500 observed”}

A dark haired young woman was shoved violently into the frames by an unseen force. She wore a silver club dress and carried her shoes. She stumbled, fell, and her shoes skittered out of the frame. A fluttering orange bounding box outlined her shape on the screen, and followed her movements.

{“2045-05-20T04:46:09Z” : “victim VIKTOR GARCIA ##00FF00 observed”}

{“2045-05-20T04:46:11Z” : “victim VIKTOR GARCIA observed attacking subject JULIA GARCIA”}

A tall man with dark hair and a neck tattoo strode into the frame, and a green rectangle popped to surround him. He shoved the woman hard towards the ground.

{“2045-05-20T04:46:14Z” : “victim VIKTOR GARCIA threatens JULIA GARCIA - weapon”}

The man on the screen removed an automatic pistol from the waistband of his jeans, and pointed the pistol at the woman’s head. She cowered instinctively away from the shot that would end her life, but never came. CFM-2 moved quickly.

{“2045-05-20T04:46:15Z” : “suspect CFM-2 terminates VIKTOR GARCIA - repeated blunt force trauma”}

A broad smear of fresh blood streaked down the wall next to the motionless body, which lay crushed and broken at the base of the wall. A dark puddle pooled on the white tile floor, and CFM-2 resumed it’s slow, steady pattern, trailing a receding smear of blood behind it, as if nothing remarkable had occurred.

{“2045-05-20T04:46:21Z” : “suspect CFM-2 observed cleaning floors”}

{“2045-05-20T04:46:25Z” : “suspect CFM-2 observed cleaning floors”}

Draper rubbed his eyes with his palms. “Listen, I get it. It did the right thing, it saved a person’s life. But it also decided to kill another person in the process. It shouldn’t be able to do that. Aren’t the Ten Robot Commandments from the Great Robot Creator supposed to be hard coded into these things?” Draper widened his eyes, and wiggled his fingers playfully at Reid in a spooky gesture.

“You watch too many movies,” said Reid. “You’re referring to the 3 Laws of Robotics. And yes, even a dinosaur like the CFM-2 has a legacy failsafe chip. But a failsafe chip is nothing like A.I. You see, all the A.I. you are familiar with is equipped with very complex machine learning models, much more complex than 3, or 300, or even 3000 laws. Even the most basic machine learning models in use today are trained using over 175 Billion input parameters, more capable than the human brain for many use cases.”

Draper scowled. “So...doesn’t the CFM-2 have at least that basic A.I.?”

“Nope. This thing is old school. First generation.” answered Reid, as he thumped the metal dome top of the aging robot. “No complex event processing, no AI decision making, nothing. It’s floor cleaning routine is hardcoded and dirt simple; I inspected every line of code myself. Besides that, it only has 3 sensors for simple object detection and collision avoidance, and then that legacy failsafe chip. So, it doesn’t have any of the bits required to make any decisions, really.”

Draper considered this for a moment, then switched his line of questioning. “Ok, but I seem to recall...weren’t there some validated cases of robots equipped with legacy failsafe chips, that made decisions to save a human life?”

Reid smiled. “Yes, but again...those weren’t decisions. Legacy failsafe chips simply instruct the robot to not harm humans, nor allow a human to be harmed or killed. There’s no decision being made, it's just a few lines of procedural instructions written by a human. The result is, IF a pallet of bricks is going to crush a construction worker, THEN the robot will push him out of the way. A scenario resulting in saving a human life will trigger the failsafe action. It’s procedural, it's not intelligence.”

Draper was nodding now, he was starting to get it. “So, if our CFM-2 friend here does not trigger the failsafe action, then Julia Garcia dies, shot in the head by her deranged husband. If it does trigger the failsafe action, Viktor Garcia dies.” He paused, putting it all together. “In either scenario, one of them ends up dead. So then why did this CFM-2 trigger the failsafe action?”

“Well, that's why it doesn't make any sense.” said Reid. “These old failsafe chips can trigger a robot to save a human in a simple scenario. It can’t decide which of two humans should live, when a scenario requires that one must die. It can’t decide that Viktor is wrong, and deserves to die just because he is pointing a gun at Julia. In the Garcia case, one of them dies in either scenario. What appears to be the right decision, can’t actually be that, because this robot is incapable of that decision. It may seem right to you and I, but we can’t report it that way...because we know the robot just can’t do it.”

“Well, right or wrong,” said Draper, “put it in your deposition, and submit it to the Magistrate. Bet you lunch that old tub gets hauled away to the Claremont Terminal scrap shredder.”

---

Carter Reid didn’t work on his ride home. He instructed the taxi drone to pass through a pickup terminal where he and his wife rented a delivery locker. Halfway through her third trimester, chocolate and fresh lemons were all Angela Reid demanded. Carter wished he had kept track of all the chocolate and lemons his wife consumed, so the world would finally know how many chocolates and lemons were needed to construct a baby.

“That one, there,” he said as he gestured to stop. Moments later, he was back in the vehicle carrying a tissue-thin plastic bag filled with chocolate and lemons. The door slid shut, and the taxi drone resumed the route home.

Carter stepped into his apartment, slid off his shoes, and found Angela sleeping on the couch. Quietly, he made her a plate, placed it on the end table, and then covered her shoulders with a soft blanket. She stirred, but did not wake.

Carter grabbed his leather work bag, and padded in his socks down the hallway to his workspace. “Lights on,” he said, and the lights obeyed, revealing a disarray of parts, wires, cables, and components strewn on every horizontal surface in the room, including the floor. In the center of the room, elevated above the mess by a milk crate, sat a CFM-2 control unit. Carter had not been able to convince them to give him a whole robot for testing, so a control unit would have to do.

Three grey and red ribbon cables sprouted from an access port in the top of the control unit. Each one was labeled with wide masking tape defaced with a red grease pencil: LIDAR, IR, and UHD16K. The three sensor cables ran down to an aging serial interface unit on the floor, which was connected to a laptop. The laptop screen showed three crudely designed program windows, with the same three labels as those on the masking tape. As Carter moved around the room, his shape registered differently on each of the three windows. In the LIDAR window he was a wireframe stick man, in a comic book room of thin contour lines. In the Infra-Red window his body was a hot blurred mass of rainbow fire, bobbing against the cold space of the blue-black room. In the UHD16K window he was just...Carter, rendered across 132 million pixels, 281 quadrillion colors, and distorted grotesquely by 180 degrees of fisheye lens.

“Let’s give you one more try, shall we?” Carter said to the Control Unit, as he sat down cross-legged on the floor. He slid a large laptop from its leather bag, and turned it on. He initiated the 3D model he’d made of the Garcia crime scene, then stretched to reach the smaller laptop and awkwardly punched keys with one hand. All at once the LIDAR, IR and UHD16K program windows replaced Carter’s workroom with the model of the subway station. Finally, he opened a terminal window on the large laptop, and typed a command, and watched the output stream by:

Last login: Thu May 20 21:47:32 on ttys000

creid@nestor ~ % cd /garcia

creid@nestor ~ % garcia ./cfm2_capture.py --bold-diff

{“failsafe_state”: {“action”: False, “super_class”: null, “confidence”: null}

{“failsafe_state”: {“action”: False, “super_class”: null, “confidence”: null}

For the next three hours, Carter attempted in vain to get the CFM-2 to trigger the failsafe action. He tried simulating himself as the attacker, and as the victim. He tried replacing a copy of himself as both the attacker and the victim. The only way he could get the legacy failsafe chip to trigger, was to simulate dropping a park bench on his simulated self. But no permutation of reproducing the Garcia scenario would trigger the failsafe action.

After one final attempt at a simulation of Viktor Garcia as the attacker, Carter closed the lid of his laptop in defeat. Maybe Draper was right, he thought. Maybe the CFM-2 was defective, and maybe it did decide? Maybe it decided to murder Viktor Garcia. That wasn’t possible, but it didn’t matter; if he couldn’t reproduce the failsafe action, then there was no chance of explaining it to a Magistrate. And, more importantly, no chance of understanding it himself…

“Hey,” whispered Angela from the hallway. “What are you still doing up?”

“Ah, just wrapping up. I have to finish this deposition.” said Carter.

Angela came into the room, still wrapped in the soft blanket Carter had put over her. She pulled him to his feet. “C’mon, Doctor Reid. Your deposition will still be there in the morning.”

“Yeah, you’re right,” Carter said as he and his wife left the room. “Lights off.” The room went dark, lit only by the laptop connected to the Control Unit, still streaming its three very different views of the world in the simulation of the subway terminal.

---

The next morning, Reid got up early, drew a thermos of coffee, and decided to submit his deposition from his desk at the A.I.C.U. headquarters. He hoped to finish it before Draper got around, so he could avoid his gloating. As the taxi drone sped him silently downtown, the sky to the east was beginning to brighten behind the dark shapes of the UBI slums.

At the office, Reid took a seat behind his desk with a large computer terminal. He slid the large laptop out of its bag, opened the lid, and allowed it to come out of sleep mode while he checked his messages on the larger terminal, and sipped his coffee. When the laptop sprung to life, Reid instinctively began to close the now useless windows left over from the previous night’s failed experiments.

Then, he saw it. Sandwiched in between several dozen buffered lines of the relentless, unchanging serial output from the night before, the screen displayed:

{“failsafe_state”: {“action”: False, “super_class”: null, “confidence”: null}

{“failsafe_state”: {“action”: TRUE, “super_class”: “human”, “confidence”: .937}

{“failsafe_state”: {“action”: TRUE, “super_class”: “human”, “confidence”: .954}

{“failsafe_state”: {“action”: False, “super_class”: null, “confidence”: null}

super_class: human. Well, that’s not very helpful. Reid thought. But, get ready to buy that lunch, Draper.

Reid sat up, pulled his coffee and laptop close, and dug in. He pulled up the printed specifications for the CFM-2, and searched for the object detection model it used for collision avoidance. Bingo. The collision avoidance model was trained on ObjectNet3D+, an old Stanford training set of 3D LIDAR scans. The set consisted of 201,888 common objects, plus metadata for things like temperature, opacity, hardness, and other properties.

Reid opened a browser, and after a few minutes of digging in the Stanford archives, had the ObjectNet3D+ training data downloaded to his laptop. He scanned the contents, stopping on classes.txt, and opened it in a text editor. It was common for older systems to simplify inference by using a hierarchy of classes, to limit the number of output classes. He scrolled past thousands of common objects in the list. Most subclasses, -- like Cabinet, Calculator, Can, and Cap -- were just classified as part of the super-class: thing. Other subclasses, -- like Gun, Knife, Pistol, and others -- were classified as super-class: weapon. And so it continued for the super-classes of tools, clothing, transport...and humans. Reid scanned the list of objects included in super_class: human. 

Then, with one printed word, it all made sense. Reid shook his head and laughed at himself. I do all the work, he thought, and Draper still gets to be right.

He accessed the department’s digital evidence room from his laptop, signed out the CFM-2 logs and IR camera footage from the night of the incident, and copied them to his laptop. He opened them side-by-side, and advanced both to timestamp 2045-05-20T04:46:14Z, just moments before the CFM-2 would kill Viktor Garcia.

Dr. Carter Reid smiled for a moment, reached into his file drawer and pulled out the dossier labelled GARCIA. He opened it, examined a single page, and was satisfied with what he saw. He switched to the larger terminal to begin writing his deposition for the Magistrate.

---

Just before lunch, James Draper made his way over to Reid’s desk. “Um….I’m confused. Two Transit Authority workers just came and took that CFM-2 away. Said they were putting it back in service? What the hell did you do?”

“I submitted my report. That’s all. Don’t act so suprised, you were right after all,” Reid said.

“Really? How’s that?”

Reid slid a chair over, gestured to Draper, and swung his laptop around so they could both see the screen. He showed Draper the first triggered failsafe in the buffered logs left over from his apartment. Then he showed him the definition of the superclass human used to train the CFM-2’s object detection model:

{“super_class”: “human”, “sub_classes”: {“boy”, “girl”, “man”, “woman”, “child”, “infant”, “baby”, “teen”, “elderly man”, “elderly woman”}}

Then, he showed Draper the logs and infrared camera footage from the moments before the robot killed the man.  

{“failsafe_state”: {“action”: TRUE, “super_class”: “human”, “confidence”: .51}

The IR footage showed a pink bounding box across Julia Garcia's abdomen, around a tiny blurred mass of hot rainbow fire.

Finally, Reid pointed to something on Julia Garcia’s data page that he’d taken from the dossier. “Julia Garcia has one child, Lucas. His birth date puts Julia at only 18 weeks pregnant when Viktor pointed a gun at his mother’s head. It also explains why we don’t notice it in the surveillance videos, and why the CFM-2 is only 51% confident in the object class, but it was 95% confident when it caught a few frames of Angela, at 7 months pregnant.”

“Jeesh,” said Draper, shaking his head. “Does that mean if this had happened a few weeks earlier, or if she’d been wearing a coat, or anything to reduce CFM-2’s confidence level below 50%...”

“...That’s right. The CFM-2 would not have triggered the failsafe action, and you’d be defending Viktor Garcia for a double homicide. CFM-2 wasn’t protecting Julia Garcia. It was protecting her unborn baby. Killing Viktor saved two lives, with a net gain of one. You were right my friend...it's just simple math.”