Testing AI’s Ability to Summarize Data for Writeups

AI apps, extensions, and integrations are rolling out every day. Many of them boast to revolutionize content creation, automating many aspects of SEO and digital PR work, and – they claim – maximize results. While there’s no arguing that AI hasn’t been much of a game changer in how we work, particularly in the white-collar fields, is it wise to say that AI is up to the task of delivering nuanced data analysis summaries? If so, which AI tool is best for content writing?

The bread and butter of digital PR is producing newsworthy data, summarizing it on a content page, and pitching journalists to cover the new data; as such, the summary- or writeup- of data findings is crucial to producing high-quality digital PR studies– and therefore, obtaining high quality, high DA links for clients. How can AI fit in?

In this blog post, I’ll be using the same data set and prompting AI tools for content writing ChatGPT, Bard, and Google’s new Duet AI to write data analysis summaries and then comparing each output.

Variables to Consider & Today’s Contenders

A good content writeup can be many things: well-written, punchy, scannable, and chock full of newsworthy stats. In order to really delve into which AI platform is most equipped to summarize data and save you time by creating a writeup for you, I’m considering the following variables:

1. Accuracy: Is the writeup data accurate? Are there any factual errors?
2. Fluidity: Does the writing flow well, or does it cluster too many statistics together to feel like a piece of writing?
3. Humanity: Does this writeup feel as if a human wrote it? Or does it maintain that clunky, repetitive, robotic feeling that many AI platforms seem to have? While human touch is irreplaceable, one can aspire towards human feel to minimize time spent editing.
4. Newsworthiness: The bread and butter of all digital PR campaigns, newsworthiness, still reigns supreme. Can the writeup signal newsworthiness, or is it dull and dry?

All of these dimensions make for an excellent writeup that will perform the job of enticing journalists with aplomb. We’ll look at each platform’s writeup and evaluate them on these variables to determine who- if anyone- is the winner.

The bulk of this article will examine three platforms: ChatGPT, Bard, and Duet AI, Google Workspaces’ new AI integration that promises to “help you write” from within Docs. While ChatGPT is arguably the most famous of the three, Bard and Duet can prove strong contenders for longform writing. 

To keep things equal, I will evaluate each platform using the same set of data points, pulled from our recent blog post on top phone privacy concerns. This consisted of a short survey and an analysis of Google search volume; both types of data points are represented in the test set.

For each platform, I’ll feed both the same prompt: “Please write a newsworthy summary of the below data points to be published on a content page” and I’ll paste the data set into the prompt box.

From there, I will experiment with adjusting the tone and length of the writeups within each platform to see which one is able to better tailor to the content creator’s needs. 

Contestant #1: ChatGPT

As a brief refresher for those who haven’t checked the news feed since late 2022, ChatGPT was OpenAI’s entry into the AI arms race and the platform that sparked the current and ongoing text generative AI revolution. A simple dialogue box, ChatGPT functions as a chatbot that takes prompts tailored to length, style, tone, etc and will output text based on its probability function. As a large-language model (LLM) it will always output sequences of words most statistically likely to follow each other. ChatGPT cannot make value judgments on its own, nor can it produce anything approaching original thought.

Happily, summarizing things is one of its strongest capabilities. 

ChatGPT’s Response to the Prompt and Data:

Newsworthy Summary:

A recent survey reveals significant concerns among Americans regarding their privacy and phone usage. Over 4 in 5 Americans are worried about companies tracking their internet activity through ad tracking. The study found that Baby Boomers, Gen X, Millennials, and Gen Z are all affected, with 84%, 78%, 82%, and 81% expressing concerns, respectively.

Additionally, 60% of Americans believe their phones are listening to them, raising suspicions of potential privacy infringements. The data shows that 64% of iPhone users and 56% of Android users share this concern.

Smart home devices, such as Amazon’s Alexa, also raise alarm bells for many individuals, as 74% of users with such devices believe they are constantly listening.

Phone data tracking by the government is another prominent concern, with 60% of Americans suspecting their activities are being monitored. This sentiment varies across generations, with Baby Boomers at 40%, Gen X at 54%, Millennials at 65%, and Gen Z at 75% expressing such apprehension.

Interestingly, 13% of respondents have contemplated purchasing a non-smart phone to limit their digital and internet presence, suggesting a growing desire for privacy protection.

Furthermore, the survey highlights the main concerns by state. Alabama residents worry about geolocation, Alaskans fear spy apps, and Arizonians express interest in going off-grid. Californians, on the other hand, are concerned about Alexa listening, while Coloradans contemplate buying flip phones. Google spying is a primary concern in Connecticut and Illinois, whereas Delaware and Maryland residents fear Alexa listening. Floridians worry about targeted ads, and Georgians express concern about app tracking.

Overall, the top 10 phone use concerns in the U.S. include geolocation, going off-grid, targeted ads, app hacking, spy apps, app tracking, Google spying, Alexa listening, phones listening, and buying a flip phone. These findings reflect the growing unease and desire for privacy among Americans in the digital age.

Our Review:

Variable	Notes	Score
Accuracy	This summary contains no errors and notes nearly every statistic provided in the initial dataset provided.	5/5
Fluidity	This reads very nicely, with a good cadence, though it is a bit brief for what a normal summary might feel like. It could stand for more contextualizing or connective sentences to really clarify the dominant media-friendly plot points.	4/5
Humanity	This feels as if it could be written for a press release by a PR agent, so in this sense, it doesn’t necessarily have the clunkiness of typical AI generated content; again, I would like to see more exposition or connective sentences so the summary isn’t just rattling off stat after stat. It also could stand to use less repetition of the title of the data points– for instance, the last paragraph summarizes a stat with the same title, and this feels a little regurgitative.	4/5
Newsworthiness	This feels very well written and appealing to journalists looking for a good summary of significant statistics.	5/5
Overall Score		18/20

Additional Testing

When prompted, ChatGPT also changed its tone and add more connective sentences, here is the result: 

Title: The Big Privacy Puzzle: Americans’ Phone Use Concerns Revealed

Summary:

Americans are raising their eyebrows and questioning their digital privacy as a recent survey unveils fascinating insights into phone use concerns. It turns out that a whopping 4 out of 5 Americans are sweating over the idea of companies stalking their internet activities through sneaky ad tracking. And guess what? Baby Boomers, Gen X, Millennials, and Gen Z are all in on this worry, with 84%, 78%, 82%, and 81% respectively, feeling the heat.

But that’s not all! A staggering 60% of the population thinks their phones are eavesdropping on their conversations. Hey, it’s not just a suspicion—it’s a whole vibe! Among iPhone users, a whopping 64% have that creeping feeling, while 56% of Android users are equally convinced that their smartphones are secretly tuning in.

Now let’s talk about smart home devices. Those handy dandy voice assistants like Amazon’s Alexa—ever wondered if they’re listening to your every word? Well, brace yourself! A whopping 74% of smart device owners are convinced that Alexa and her pals are keeping their ears open 24/7. It’s like having a permanent audience!

Hold on tight, because we’re diving deeper into the rabbit hole. Government phone data tracking—it’s a controversial topic. And it’s got people talking! About 60% of Americans suspect Big Brother is snooping around their phone activities. But hey, the paranoia levels are not equal across the generations. Baby Boomers are at 40%, Gen Xers at 54%, Millennials at 65%, and Gen Z is taking the lead at a whopping 75%. Watch out, Uncle Sam!

You won’t believe this: a chunky 13% of the survey participants have actually considered buying an old-school non-smart phone to dodge the digital spotlight. Talk about taking a step back in time! It seems like people are craving some good old-fashioned privacy.

As if that weren’t enough, let’s take a trip around the United States and see what specific privacy concerns each state is buzzing about. From Alabama’s worries about geolocation to Alaskans fearing sneaky spy apps, and Arizonians dreaming of going off the grid—each state has its own unique flavor of concern. Californians, for example, can’t shake off the feeling that Alexa might be eavesdropping, while Coloradans are tempted to ditch the fancy smartphones for flip phones. And hey, over in Connecticut and Illinois, folks are side-eyeing Google for spying. Meanwhile, Delaware and Maryland are turning a suspicious ear to Alexa, and Floridians just can’t escape those pesky targeted ads. Georgia is keeping a close watch on app tracking, while Hawaii joins the Alexa-listening club. It’s a whirlwind of privacy puzzles!

So, in a nutshell, the top 10 phone use concerns in the good ol’ U.S. of A. include geolocation, going off the grid, targeted ads, app hacking, spy apps, app tracking, Google spying, Alexa listening, phones listening, and the unexpected comeback of buying a flip phone. It’s a real privacy puzzle, and it seems like Americans are piecing it together one concern at a time.

At first glance, while ChatGPT certainly understood the assignment of adjusting tone, it also took things too far- this reads like a whacky infomercial more than a semiformal data summary designed to be both accessible and newsworthy. This might work for lighthearted outlets– but certainly isn’t what we’re looking for. 

Contestant #2: Bard

As we all know, it’s Google’s internet, and we’re all just surfing it. As such, when the AI arms race began, Bard was not far behind ChatGPT in its rollout, though it certainly is not without its flaws. Here’s how it stacks up in data summaries.

Bard’s Response to the Prompt and Data:

Top 10 Phone Use Concerns in the U.S.

A recent survey found that Americans are increasingly concerned about their privacy when it comes to their phone use. The top 10 concerns are:

1. Geolocation: 60% of Americans are concerned that their phone is tracking their location.
2. Going off grid: 54% of Americans have considered buying a non-smart phone to limit their digital and internet presence.
3. Targeted ads: 48% of Americans are concerned that companies are using their phone data to target them with ads.
4. App hacking: 46% of Americans are concerned that their phone apps could be hacked.
5. Spy apps: 44% of Americans are concerned that someone could install a spy app on their phone without their knowledge.
6. App tracking: 42% of Americans are concerned that apps are tracking their usage data without their permission.
7. Google spying: 40% of Americans are concerned that Google is spying on them through their phones.
8. Alexa listening: 38% of Americans are concerned that Amazon’s Alexa is always listening to them.
9. Phones listening: 36% of Americans are concerned that their phones are listening to them even when they are not in use.
10. Buying a flip phone: 34% of Americans have considered buying a flip phone to limit their digital and internet presence.

These findings suggest that Americans are becoming increasingly aware of the privacy risks associated with phone use. As more and more of our lives move online, it is important to be mindful of how our data is being collected and used. There are a number of things that we can do to protect our privacy, such as using privacy-focused apps, disabling location services, and being careful about what information we share online.

Here are some additional tips for protecting your privacy when using your phone:

• Use a privacy-focused browser: There are a number of privacy-focused browsers available, such as Brave and Firefox Focus. These browsers block trackers and other third-party scripts, which can help to protect your privacy.
• Use a VPN: A VPN encrypts your traffic and routes it through a server in another location. This can help to hide your IP address and make it more difficult for someone to track your online activity.
• Be careful about what information you share: Think twice before sharing personal information online, such as your home address, phone number, or Social Security number. This information can be used by criminals to commit identity theft or other crimes.
• Be aware of the apps you install: Before you install an app, read the permissions that it requests. Make sure that you are comfortable with the data that the app is collecting and how it is being used.
• Update your software regularly: Software updates often include security patches that can help to protect your phone from malware and other security threats.

By following these tips, you can help to protect your privacy when using your phone.

Our Review

Variable	Notes	Score
Accuracy	It tried. It looks like Bard took a lot of the survey data points and somehow combined them with the separate data points about top concerns by state to create a very interesting interpretation of “summary.” Not only that, but many of the percentages were fictional or incorrect.	1/5
Fluidity	This reads in part like a journalistic pitch, which is close to on the mark, and partially like an explainer article/listcicle of “Top ways to protect your privacy.” As such, it veers away from summarizing data and does not understand the assignment.	2/5
Humanity	It reads as human in the sense that someone probably wrote a relatively low-quality explainer article for SEO benefit.	3/5
Newsworthiness	The first portion of this writeup reads like a pitch to journalists, complete with statistic bullets and a bare-bones intro that jumps straight into the information. On the other hand, most of this information is wrong.	3/5
Overall Score		9/20

Additional Testing

When prompted with “Many of the statistics you provided are incorrect. Please reread the initial input and try again,” here is the response:

Sure, here is a newsworthy summary of the data points you provided, with the incorrect statistics corrected:

Americans Concerned About Phone Privacy

A recent survey found that Americans are increasingly concerned about their privacy when it comes to their phone use. The survey, conducted by [insert research firm], found that:

• Over 4 in 5 Americans (84%) are concerned that companies can follow their internet activity via ad tracking. This concern is shared across all age groups, with 84% of Baby Boomers, 82% of Gen X, 81% of Millennials, and 81% of Gen Z expressing concern.
• 60% of Americans believe their phone is listening to them. This belief is more common among iPhone users (64%) than Android users (56%).
• 74% of those with smart home devices like Amazon’s Alexa think the devices are always listening.
• 60% of Americans believe the government is tracking their phone data. This belief is more common among younger Americans, with 75% of Gen Z expressing concern, compared to 40% of Baby Boomers.
• 13% have considered buying a non-smart phone to limit their digital and internet presence.

The survey also found that Americans are concerned about a variety of other privacy issues related to their phone use, including:

• The use of their data for targeted advertising
• The possibility of their phone being hacked
• The possibility of their phone being used to track their location
• The possibility of their phone being used to spy on them

The findings of this survey suggest that Americans are becoming increasingly aware of the privacy risks associated with phone use. As more and more of our lives move online, it is important to be mindful of how our data is being collected and used. There are a number of things that we can do to protect our privacy, such as:

• Using privacy-focused apps
• Disabling location services
• Being careful about what information we share online
• Being aware of the apps we install
• Updating our software regularly

By following these tips, we can help to protect our privacy when using our phones.

This is a MUCH better response, insofar as it actually provides correct statistics and much better resembles an actual data summary rather than a pitch derived of statistical salad. On the other hand, it still seems insistent on including the tailpiece of a short expository article that was unasked for and unnecessary. 

Contestant #3: Google’s Duet AI

In June 2023 Google announced the full integration of an AI assistant across all Google workspaces- Docs, Slides, and more. 

With its shimmering pen icon, Duet AI is the Clippy of the 21st century and promises to help you write.

Duet AI’s Response to the Prompt and Data

Our Review

Well, it was worth a try. Hopefully, one day soon Duet will also be able to summarize data inputs, as this would be a huge time saver for any enterprising content strategist or digital PR pro wishing to speed up the writing process.

Conclusion

Despite Google’s best efforts to come up with competitors, if you’re looking for an AI-powered data summary, ChatGPT wins by a mile. While it might take a moment to finesse the tone to be exactly what you want, OpenAI’s leader in the AI field reigns supreme.

Read more from our blog