I let AI write & submit a research paper… now we wait

Summary

In this video, a professor with extensive experience in academic publishing tests the capabilities of AI tools, specifically ChatGPT and Deep Research, to draft a research paper for submission to a peer-reviewed journal. The experiment explores how AI has evolved, highlighting the improved quality of AI-generated papers and its potential to pass as expert-level research. Despite concerns about ethical use and AI's ability to produce reliable citations, the professor submits the paper, observing how peer review might react to AI-assisted content. The video raises questions about AI's role in academic publishing and the future of research.

Takeaways

• 😀 The professor has published over 400 peer-reviewed papers and has been skeptical of AI's ability to generate quality academic work until recently.
• 😀 Previous AI models like GPT-3.5 were criticized for hallucinating references and exhibiting poor writing that could easily be detected as AI-generated.
• 😀 The new deep research model from ChatGPT (GPT-4) has significantly improved, producing high-quality, expert-level papers that could match the work of a PhD student or assistant professor.
• 😀 AI now shows a much stronger understanding of research methods, clarifying the need for specific parameters (e.g., time range, population, outcomes) in literature reviews.
• 😀 The professor ran an experiment by submitting an AI-generated manuscript to a journal, minimizing human intervention to test the AI's capabilities.
• 😀 Journals often have strict AI policies, so the professor had to submit to a semi-predatory journal to avoid ethical conflicts around AI-generated content.
• 😀 The professor emphasizes the importance of transparency when using AI in research, declaring its usage in the submission process as required by the journal's policies.
• 😀 One of the most notable improvements in AI is its ability to properly cite real references, reducing the occurrence of hallucinated references.
• 😀 Deep research's ability to break down complex relationships in research (such as social capital and health) was described as highly sophisticated and would typically take a researcher months to complete.
• 😀 Despite AI's advancements, the professor highlights the risks of unethical use and the need to balance AI with solid research fundamentals to avoid dependency and fraud in academic work.

Q & A

Why is the professor skeptical about using AI tools in academic research initially?

The professor was initially skeptical because AI tools like earlier versions of ChatGPT would hallucinate, create fake references, and produce poor writing that could easily be flagged as AI-generated. These issues made AI unreliable for academic research.

What changes in the professor’s perspective after using Deep Research and ChatGPT 4.0?

After using Deep Research and ChatGPT 4.0, the professor was impressed by the AI's ability to produce high-quality research papers at an expert level, which exceeded expectations for a first-year PhD student. The AI could now generate more sophisticated, mature content with accurate references.

How does the professor describe the difference between earlier versions of AI and the new Deep Research model?

Earlier AI models produced short, poorly structured, and sometimes inaccurate content. In contrast, the Deep Research model could produce comprehensive, well-organized papers, accurately citing references, and following solid research practices, making it comparable to expert-level work.

What was the professor’s goal with the experiment involving AI-generated research?

The professor’s goal was to test whether an AI-generated manuscript could pass through the peer review process of a journal, despite the AI being used extensively in the creation of the paper, including literature reviews, citations, and even reviewer suggestions.

Why did the professor choose to submit the paper to a 'semi-predatory' journal?

The professor chose a 'semi-predatory' journal because its policies did not have clear guidelines against AI usage in research submissions, allowing for the experiment to proceed without ethical concerns about the journal's AI policies.

What was the challenge the professor faced while trying to format references in the manuscript?

The professor struggled with getting the AI to properly format and cite references within the text. Despite multiple attempts to correct this, the AI still failed to include references correctly in the manuscript, which required manual adjustments.

How did the AI assist in the selection of reviewers for the manuscript?

The AI was used to suggest potential reviewers for the manuscript by providing real names, contact details, and justifications for why each reviewer was suitable, making the process of finding reviewers easier for the professor.

What was the professor's approach to submitting the manuscript and cover letter?

The professor adhered to a minimal-change approach, keeping the AI-generated content intact as much as possible. The AI helped generate the cover letter, though the professor did not heavily edit or refine it to maintain the authenticity of the experiment.

What ethical considerations were addressed during the manuscript submission process?

The professor followed the journal's ethical requirements by disclosing the use of generative AI tools in writing the manuscript, ensuring transparency and honesty about the role AI played in the creation of the paper.

What does the professor hope to learn from submitting the AI-generated manuscript?

The professor hopes to learn how the peer review system will handle AI-generated content, particularly whether it will be accepted or rejected, and to gather feedback on AI's role in academic research and publication.