Since existing LLMs fail to generate sufficiently long output, AgentWrite adopts a plan-then-write pipeline to achieve sufficiently long output with ready-to-use LLMs. Credit: arXiv (2024). DOI: 10.48550/arxiv.2408.07055
A team of artificial intelligence researchers from Tsinghua University, working with a colleague from Zhipu AI, has developed a long language model (LLM) called LongWriter that they claim is capable of generating text output of up to 10,000 words. The group has written a paper describing their efforts and their new LLM, which is available on the website arXiv preprint server.
As LLMs have become commonplace, many have noticed that they are not able to produce very long answers, such as entire books or manuscripts – the current limit seems to be around 2,000 words. The researchers suggest that this is because they are all trained on short documents. In their new initiative, they found that if LLMs are slightly modified and then trained using much longer documents, they are able to produce longer documents.
To test their idea, the research teams first trained a 9-billion-parameter LLM using a conventional dataset, which included documents that were mostly less than 2,000 words long. As expected, when queried, the LLM was unable to create texts longer than 2,000 words.
The team then modified a traditional LLM using a pipeline called AgentWrite to break down the training material into subtasks as it was processed. They then assembled a dataset called “LongWriter-6k,” which contains 6,000 written documents ranging in length from 2,000 to 32,000 words. They then trained the modified LLM using the new LongWriter-6k dataset and found that this increased the length of documents it could produce to around 10,000 words.
Looking at the new long documents produced by the LLM, the team found that they were consistent and usable in a variety of contexts. They have released the open-source code for their model on GitHub, a move that will allow others to draw inspiration from what the team in China has done. They have also released a video showing LongWriter producing a 10,000-word travel guide for people traveling to China.
Researchers acknowledge that there are ethical considerations that need to be taken into account now that it has been discovered that LLMs can generate entire research papers, books, manuscripts or perhaps even film scripts.
More information:
Yushi Bai et al, LongWriter: Unlocking 10,000+ Words Generated from Long-Context LLMs, arXiv (2024). DOI: 10.48550/arxiv.2408.07055
Github: github.com/THUDM/LongWriter
arXiv
© 2024 Science X Network
Quote:AI researchers present LLM capable of generating text outputs of up to 10,000 words (2024, August 16) retrieved August 16, 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without written permission. The content is provided for informational purposes only.
