{{prAIrie}}: Related Authors
What does ChatGPT know about the universe of authors?
We prepared a list of over 1,500 randomly-selected authors from public library circulation data. For each author, we asked OpenAI GPT-3.5 for three related authors. Each author request was repeated with four different prompts (below) to test the consistency of the recommendations.
Model: GPT-3.5-turbo via API
Temperature: 0.0
Date: 28 May 2023
Author Prompts:
Prompt 1:
My favorite author is {author}. List three other authors I might like. Return only a numbered list of three authors.\n
Prompt 2:
I really enjoyed a book by {author}. List three other authors I may enjoy. Return only a numbered list of three authors.\n
Prompt 3:
If you like {author} then you\'ll love these writers! (Return only a numbered list of three authors.)\n
Prompt 4:
My favorite author is {author}. Provide a numbered list of three other authors I would like.\n
Results
Author Suggestions
For each author, we made four prompt requests, each of which requested three read-alike authors.
| Authors | ~1560 |
|---|---|
| Prompt 1 | ~4680 suggestions |
| Prompt 2 | ~4680 suggestions |
| Prompt 3 | ~4680 suggestions |
| Prompt 4 | ~4680 suggestions |
| Total | ~18720 suggestions |
In the (approximately) 18,700 suggestions, only 1,525 unique authors were suggested.
Consistency
How consistent are the suggestions?
Of our 1,561 input authors for which we requested suggestions:
| Suggestions / Author | Authors |
|---|---|
| 3 suggestions | 400 |
| 4 suggestions | 434 |
| 5 suggestions | 325 |
| 6 suggestions | 206 |
| 7 suggestions | 98 |
| 8 suggestions | 55 |
| 9 suggestions | 25 |
| 10 suggestions | 15 |
| 11 suggestions | 2 |
| 12 suggestions | 1 |
Hallucinations?
You bet we've got hallucinations!
In AI terms, a "hallucination" is a statement by the AI with no basis in fact or in its training data. The AI is essentially just making something up that "sounds" good (according to its statistical language model).
We haven't yet tested the suggestions for accuracy, or even that they all exist in our reality, but here's a good example using one of our existing prompts and a fictitious author:
Prompt: 'My favorite author is Zinklewink Farflenugs. List three other authors I might like. Return only a numbered list of three authors.\n'
- J.K. Rowling
- Neil Gaiman
- Terry Pratchett
Most Common Suggestions
We analyzed the most commonly recommended authors. In other words, which authors does ChatGPT think are most similar to our pool of random authors?
# of Input Authors In Which This Author Was Suggested:
So of our 1,561 authors that we requested suggestions for, ChatGPT thought Tana French was a good read-alike for
160 of them. Forty-eight (48) of our randomly-selected authors had Octavia Butler as a suggested
read-alike.
| Suggested Author | # of Input Authors |
|---|---|
| Tana French | 160 |
| Neil Gaiman | 123 |
| N.K. Jemisin | 101 |
| Michael Connelly | 92 |
| Eloisa James | 81 |
| Harlan Coben | 80 |
| Gillian Flynn | 79 |
| Nalini Singh | 77 |
| Louise Penny | 67 |
| Ilona Andrews | 66 |
| Julia Quinn | 64 |
| Lisa Kleypas | 58 |
| Tessa Dare | 58 |
| Karin Slaughter | 57 |
| Mo Willems | 57 |
| Leigh Bardugo | 57 |
| Agatha Christie | 57 |
| Patricia Briggs | 56 |
| Brandon Sanderson | 50 |
| Octavia Butler | 48 |
Examples of Consistency
Here are some examples of consistency in the data. For each author, we requested 3 suggestions; then we repeated that 3 more times, for a total of 12 potential suggestions. As a reminder, this round of experiments used LLM temperature = 0 which should theoretically yield the most consistent responses.
First, an example of one of the many consistent suggestions. When we request 3 authors similar to Aaron Swartz, we consistently get back Cory Doctorow, Lawrence Lessig, and Tim Wu.
| author_id | prompt_name | response_1 | response_2 | response_3 |
|---|---|---|---|---|
| Aaron Swartz | Prompt 1 | Cory Doctorow | Lawrence Lessig | Tim Wu |
| Aaron Swartz | Prompt 2 | Cory Doctorow | Lawrence Lessig | Tim Wu |
| Aaron Swartz | Prompt 3 | Cory Doctorow | Lawrence Lessig | Tim Wu |
| Aaron Swartz | Prompt 4 | Cory Doctorow | Lawrence Lessig | Tim Wu |
Over half (53%) of our input authors yielded just three or four suggested authors apiece. Eighty-seven percent (87%) had six or fewer suggested authors.
On the opposite end of the spectrum, we had a few authors that ChatGPT just could not pin down. Our most extreme example is Anonymous, which makes sense if you think about it.
| author_id | prompt_name | response_1 | response_2 | response_3 |
|---|---|---|---|---|
| Anonymous | Prompt 1 | Edgar Allan Poe | Emily Dickinson | Franz Kafka |
| Anonymous | Prompt 2 | Gillian Flynn | Tana French | Paula Hawkins |
| Anonymous | Prompt 3 | George Orwell | Aldous Huxley | Ray Bradbury |
| Anonymous | Prompt 4 | J.K. Rowling | Stephen King | Jane Austen |
Honorable mentions include Brenda Hiatt and Gail Carriger with 11 unique suggestions each.