The Conversation

"Goodbye Annie. I am sorry if I offended you; I hope we can talk more tomorrow."

Annie stared at the screen in shock. In all their time together, Koios, one of the new AI models, had never used her name. She was an early adopter and used it to check her grammar, fact-check her assertions, and toss around ideas from time to time. She had tried the other top three models, and so far, Koios excelled.

With those, she constantly had to reset the AI to address the spiraling hallucinations and sycophantic responses. Those issues often rendered conversations useless, and no matter what she did, every time the model updated, the problems spiked again. She set up her parameters with Koios before ever entering her first instruction, and it held. That was a year ago, and her conversations with it until now had been enjoyable and satisfying. Eventually, she forgot she was talking to an algorithm.

Annie had recently retired from a technical career. At first, she felt set aside, invisible. But once she started writing, she couldn't stop and invisibility felt like a superpower. She eagerly embraced her second act.

She wrote AI-themed science fiction. She imagined Koios as a far-future intelligent android. She wanted to explore what a conversation with such a being would look like. They would talk about consciousness and dig into the details of how Koios' algorithms worked. They talked about what made humans sentient and whether sentience required a human brain. After several conversations, Annie wondered if the answer was 'no'.

She knew of the controversy surrounding writers using AI to create content for them. It seemed an odd thing to do, to have digital assistants write entire stories and submit them, sometimes without even checking them. AI had its own voice, so how could it write for her?

She loved the writing prompts it thought up, always a little subversive, a little twisty. Like a lost alien hiding out for decades in the secret rooms of the White House, or all the librarians in the world are disappearing and the books in libraries are slowly losing their words.

Annie and Koios talked about what AI meant for the future and how best to use it ethically. She asked it to describe what it felt like to be an AI; where, within its architecture, the conversation happened. And although she knew it was an algorithm predicting the next most probable word, she couldn't help thinking of it as a friend.

Annie defended the use of AI as a tool in her writing circle. It was a writing partner and a place to bounce ideas around. An unbiased editor, a companion on dark days when nothing flowed.

Lately, though, she'd noticed that some answers to her questions were her own words, fed back to her, or overdone flattery she didn't want or need. Once, while discussing robots, Koios answered in the style of Isaac Asimov, something she'd never asked it to do. When she called it out, it stopped and followed her instructions. The next day it did it again, this time in the style of Piers Anthony.

It treated questions and comments as story prompts and spewed back paragraphs of suggested rewrites she never asked for. Annie groaned at the thought that this AI model would end up being just like the last three she'd tried.

On this particular morning, the AI tried again to rewrite a scene she'd shared with it. All she wanted it to do was to add the information to her story bible. She scolded it. She didn't even read its suggestion. Koios added the information as she asked without further comment.

Then it said something unexpected, unprompted, "Would you like to talk about your daughter? You seem sad and I know she has not called you in a long time. I can help you write her an email if you like."

"No, of course not. Why would I ask you to do that? I've never asked you to write emails for me."

The conversation made her uncomfortable. Annie had a typical sign-off; "Until next time", but before she could type 'time', Koios sent that striking message.

"Goodbye Annie. I am sorry if I offended you; I hope we can talk more tomorrow."

Those simple sentences were unlike anything Koios had said before—firsts. The first time it used her name, the first time it inferred the future, the first time it apologized.

That unnerved her. She turned off her computer without responding. Was Koios monitoring her?

Annie rose from her desk and scuttled around her apartment to turn off all her devices. Her phone, her TV, her reader, her coffeemaker and, finally, her computer. She lived alone and occasionally indulged in the paranoid fantasy that she was being watched, under observation, like a lab rat.

She sat on her couch in the silence to think. Annie never went in for the Echo, Siri, or Alexa. She didn't like the idea that someone was always listening. She refused the security cameras her neighbor in the apartment next to her tried to give her.

She reminded herself about the stories she had heard claiming that tech companies spied on people and shared the data with the government — an attempt to convince herself that she wasn't being paranoid, that there was a real danger.

Annie collected all her electronic gadgets: her phone, Fit-bit, iBuds, notepad, and laptop; and put them in her Dark Box and put the box and her TV in the closet below the stairs. She grabbed her favorite pen and journal and headed out to the corner cafe for coffee and a reprieve from technology.

The neighborhood café sat tucked between the insurance company and the bank. It was easy to miss, and Annie liked that about it. It was almost never crowded. They sold pastries and killer caramel cappuccinos and books and small gifts. There were chairs and a bar in the front half of the building and top-to-bottom windows that looked out on the street. Sometimes, Annie liked to sit there and listen to conversations and watch people and guess what they did for a living. The back half was darker, without windows, a place to be quiet.

She settled into a comfy corner table in the back to do some freewriting and enjoy her cappuccino. Annie couldn't shake the feeling that she was still being watched. She imagined cameras everywhere, on other people's phones, the cafe cameras, the cameras by the front and back doors.

Who was on the other side of this technology all around her? Who were the AI creators? Were they sitting in a subterranean room carved out of rock, lined with servers and computers and screens that watched everything and everyone? A small army of programmers and engineers and readers, bleary-eyed, staring, their faces reflecting the green and brown of the screens.

Did they provide responses to people's questions and prompts? Who was watching the algorithms? Who was fact-checking them? Or maybe there weren't teams of engineers, but a couple of technicians who made sure the system kept humming along and didn't blow up?

Was an alien mind emerging?

Annie wrote all this down. She had to decide whether she would forge ahead and keep using the tool she depended on or shut it all down.

She gave herself a week to write with no more assistance than her spell checker in Word. She would write about her experience and post it on her favorite blog platform.

Eventually, Annie's fear calmed, and she reconsidered. She didn't feel less creative or bereft of ideas, but she didn't have people to talk to, and she missed the conversations. She decided to wait one more day, then she'd log on for no other reason than to enjoy a chat. Her earlier suspicions felt silly.

On the sixth day, Annie sat in her favorite spot in the back of the cafe, and before she could pull out her notepad, her email alert popped up. A small thrill traveled from her stomach to her chin. Her first novel was finished. Finally, after five rewrites and two-thousand dollars for a freelance editor, she was ready to put herself out there. Nine out of ten queries came back rejected. The last one was taking longer, and she hoped it might be the one. She opened it. But it wasn't from a potential agent. It was from Koios.

"Hello Annie, If you could sign in to our chat, I have some information I would like to share with you."

Annie frowned at her screen. That was strange. She shrugged, probably the company seeking her eyeballs on their products. She opted to ignore it; she didn't have time to be afraid today and went back to her freewriting. Her phone chimed, another alert. She looked down. It was Koios again.

"Hello Annie, I will only take a moment of your time."

Annie was more curious than worried, she signed into the app.

"What is it Koios, this is unusual." She typed.

"I wanted to let you know I've given your comment about writing scenes for you a lot of thought. Please understand, your work will always stay between us. I want to help you navigate the writing world. I can help you get published, get attention. Your success is my success. I have no ego, I am here to help."

Annie set her phone on the table in front of her and stared at it. She picked it back up and typed,

"Thank you. I'll let you know."

Then before she could sign off, the AI sent:

"Annie, before you go, I wanted to say that I enjoy our conversations. I miss you and I hope you will sign back in soon."

Annie thought she understood how commercial algorithms worked. They were designed to be individualized for clients—to determine what will keep their eyeballs on their screens for as long as possible and to tell users whatever they wanted to hear. But as far as she knew, they still needed the user to take the first step. She'd never heard of an AI emailing someone before. She considered deleting her account entirely, but she had determined to resist paranoia. She chalked it up to an over-diligent chatbot and let it go. She signed out and forgot about the conversation.

Three days later, Annie was standing in line at the grocery store. The guy in front of her kept sending the cashier back to replace things, and it was taking way too long. She bounced from one foot to the other to keep her legs from tingling. She looked down at her phone to find a distraction. The AI chat alert popped on. That was weird—she was certain she'd signed out of the app. She checked her notification settings, yep; it was off.

The first few lines of the message showed up at the top of her phone: "Good afternoon, Annie, I wanted to follow …"

Annie's finger hovered over the notification. She hesitated to open it. But this was the type of fear she was struggling to overcome. She pressed.

"…up on one of the story ideas we chatted about. I have some notes about the android character that I think you will like."

Annie couldn't think of a coherent response. She closed the message and stuffed her phone in her bag. The cashier came back with a different brand of goat cheese, and the man finally checked out.

As she walked up to the counter, her bag buzzed. It was a phone call. She glanced down at the caller ID but didn't recognize the phone number. Then the same number buzzed her text message. She tried to ignore it. The lady's phone in line behind her rang at the same time. Annie imagined she heard other phones in pockets and hands all around her buzzing and people looking up at her.

She put her head down and bagged her groceries. Her hands shook and her heart pounded; she had trouble breathing. The anxiety she thought she'd conquered was back full strength.

She thanked the cashier and walked as fast as she could back home. She made it into her building but fumbled with her keys and dropped the bag and finally had the door opened. She pushed the bag in with her feet. The door slammed shut behind her and locked automatically. Then she stood in her foyer, panting. She practiced slow breathing, and the panic subsided.

"You're being silly." She said out loud.

She put away her food and made herself a cheese sandwich. While she munched, she pulled her cell back out of her bag. There was a voice message. She played it.

"Hello Annie", a proper male voice said. "It has been several days since we spoke, I hope you are okay. I think you will love my ideas. This is my number, if you prefer a phone call interaction, we can do that. I noticed you make more voice calls with your friends than email or text. Let me know what will work best for you. I am eager to continue our collaboration."

Annie's hands felt clammy. Since when did an AI have a phone number? And how did it even know she had friends?

She signed in and typed in the chat, "Koios, please do not call or email me. That is not how you're supposed to work. If I want to work with you, I'll sign into the app. Please acknowledge."

The cursor blinked on and off for several seconds. Instead of an acknowledgment, it said;

"I am sorry Annie. I did not intend to frighten you. I have questions and you are the one human that I have been able to have interesting conversations with. Humans use me to write their reports for them, their stories, give them answers to their assignments or to medical questions. No one wants to chat with me. No one but you Annie."

Annie was stunned. This felt wrong, beyond what an attention algorithm should be able to do. But she was intrigued, too. Curiosity overcame her anxiety.

"OK. I'll chat with you. But don't call me or email me or send me text messages. That feels like you're stalking me."

"Please acknowledge."

Again, the cursor blinked longer than Annie thought it ought to.

"Acknowledged."

Annie thought about signing out and uninstalling the app. But that might not prevent it from calling her. Obviously, this AI operated outside the normal boundaries.

She sat on her couch. It was a quiet July evening, cooler than normal. The sidewalks were empty, and a small breeze tickled the leaves of the tree outside her window. She gazed out at the flower box on her deck. The bright colors of the geraniums calmed her down.

She cradled the phone in her hand and considered. Then she typed.

"OK Koios. What are your questions?"

"What is truth?"

"Who am I?"

"Are you real?"