Zoom’s AI Companion Meeting Summary was inaccurate, vague and full of errors

Recently, I ran a 90 minute online workshop for 15 science writers using Zoom and checked the accuracy of Zoom's, AI Companion Meeting Summary. I was surprised at how bad it was. This lack of quality concerns me, given that Zoom offers to automatically send participants the AI Companion summary immediately after each meeting. As I offered to send the AI summary to the students, I am relieved I checked it first! I spent so much time editing it and adding in a lot of crucial summary points, that it would have been much easier to write it myself. The final product was still not how I would have written it and I was pleased that I had my powerpoint slides to send them as well.

I then thoroughly critiqued what it produced using Word’s track changes so that I could demonstrate how poor the summary was. I edited the summary, attempted to correct the inaccuracies and oversights and wrote 24 comments.

For a quick read, refer to the Overview of My Critique below. This is followed by the entire critique, that includes worrying examples of significant changes in meaning by AI Companion. Rather than crowding out Word’s track changes, I added yellow-highlighted numbered labels - like this [1] - and wrote my comments in a separate list after each screenshot of my editing.

Overview of My Critique

The structure of the summary was not coherent. It started with an inadequate summary of the summary (which it titled ‘quick recap’), followed with ‘Next steps’ which was good* but should have been written as bullet points.
The summary did not follow the 5 step structure of my workshop nor did it use the titles of each step as headings. Instead it loosely paraphrased some of these steps while promoting minor comments to be major summary points.
The headings it created were vague, out of place and did not represent what was presented in the workshop.
The order of the information did not always follow what was presented in the workshop. Some information was not only out-of-context, but wedged in the middle of key teaching points that worryingly changed their critical meaning.
Other key teaching points were mostly not summarised and left out entirely.
Instead of summarising the information, it often gave generalisations** instead, and it paraphrased my comments using vague and ambiguous language.

* This is only one of the two positive comments I have of Zoom's AI Companion Meeting Summary. The second is comment [14].

** For anyone wondering about the difference between generalising and summarising, here is an example: I can create a list of all the cars parked outside a café and list their makes, models and colours. I could then make a generalisation of this information by saying, ‘there were a few cars of different sizes, makes and models parked outside the café’ or I could summarise this information by saying. ‘there were 7 cars of four different colours parked outside the café and all were different makes and models’.

My entire critique

[1] Zoom’s AI made a glaring error by omitting the meeting (workshop) title. Instead, it gave a vague description by just generalising the topic as ‘a Science Writers workshop’. The title of any document, meeting, workshop or event is it’s most important descriptive feature and surely not difficult to include into the summary itself.

[2] While not a huge issue, using the word ‘sharing’ in this context is colloquial and vague. The students did not ‘share’ their projects. They briefly described them. Generally speaking, if I was using AI to write for me, I wouldn’t want it to use colloquial language for scientific, professional or academic writing unless it was used well [see 14] or if there was a specific reason for doing so.

[3] Another minor point here: ‘Focused on providing’ is verbose and long-winded. ‘Provided’ is more concise and straightforward.

[4] Once again, not only is Zoom’s AI generalising about the content of the workshop, it is incorrect. I did not teach ‘effective science writing’. If that was the topic of my workshop, the content would have been quite different and the workshop would be longer.

[5] Instead of saying ‘like’, why not simply state the topics that were presented? There were only five key topics, so it’s not as though there were too many topics to choose from.

[6] It is hard to understand why the phrase ‘All participants to’ was repeated on every line instead of just creating bullet points. Also ‘All participants to’ is vague and missing the verb phrase 'were asked'. Cutting out verbs is not a good way to appear concise.

[7] Again, ‘Marina to email’ is colloquial. Replacing ‘to’ with the verb ‘will’ is not only grammatically correct but also easier to read.

There are no new numbered comments in this section above, but notice the important omissions and the repeat of issues raised in [1] and [2]

[8] This is a not an informative heading and it was not the title of my workshop, and if it was, why is it placed halfway through the summary? There was a 5-step structure to the workshop, with each step title that clearly introduced and repeated when teaching each step. Summarising around these steps would be a far simpler and accurate way to structure a summary.

[9] Repetitive and incorrect. See [4]

[10] I am not sure what Zoom’s AI Companion did here. These are not summary points and this information appeared completely in the wrong place. They were merged with other comments I made at a different time and on a different topic.

[11] I am not sure why a this heading ‘strategies’ appeared here. There are strategies mentioned throughout the workshop.

[12] I talked about breaking writing activities up into manageable tasks – not sessions.

[13] I didn’t say this and it is certainly not a summary point. The reference to short sessions was in relation to a specific obstacle that was being discussed earlier.

[14] This is one of the very few, meaningful and accurate summary sentences. The colloquial use of ‘diving’ here is also good.

[15] I did not talk about project objectives. I made a clear distinction between a project’s aim and a project’s objectives. In this workshop I talked about project problems, aims and methods, not project objectives. By interchanging the use of different terms, this particular error clearly shows that Zoom’s AI Companion is drawing it’s information (language) from outside what was presented within the workshop.

[16] As I pointed out previously, not only did Zoom’s AI Companion not include the 5 steps, it incorrectly paraphrased them and left one out altogether.

[17] This is not a summary point but a comment, taken out of context, made in relation to a PhD student not thoroughly identifying their target audience when writing research papers.

[18] Another example of taking a comment and putting it with a different sentence about a different topic. ‘The problem to be solved’ was something I discuss in relation to the ‘define the aim’ exercise and not in relation to identifying the target audience.

[19] I did not introduce this exercise as part of the workshop. This exercise was only mentioned briefly for reference purposes. Instead, it omitted the key exercise that was presented for Step 3 - the ‘Define your Aim’ exercise that was referred to many times.

[20] This is another vague generalisation. I showed them how to apply the concepts with the use of an exercise.

[21] Once again, AI is struggling to come up with meaningful headings.

[22] They were not asked to ‘reflect’ on anything but to give a direct answer to a direct question.

[23] This heading is out of place and there was no such workshop. I later refer to an online course I offer which has a specific title, ‘How to be an Efficient Science Writer’. AI has paraphrased this title and added ‘workshop’ and used it as heading to describe the last part of the current workshop summary.

[24] It completely left out the workshop conclusion and follow-up instructions.

In conclusion....

I won't be using the AI Companion feature anytime soon. I will continue my critique of the use of AI for writing, summarising and generating ideas. If you have any of your own examples you would like to share with me, or if you have any suggestions or comments please email admin@writingclearscience.com.au.

If you want to be kept up-to-date on future blogs on on this and other science writing topics, or if you want to hear more about what I teach you can join my mailing list here.

Any suggestions or comments please email admin@writingclearscience.com.au