A Roadmap for Big Model release_rev_23f5d650-36a4-4132-8e00-9a6f63d94567

by Sha Yuan, Hanyu Zhao, Shuai Zhao, Jiahong Leng, Yangxiao Liang, Xiaozhi Wang, Jifan Yu, Xin Lv, Zhou Shao, Jiaao He, Yankai Lin, Xu Han (+88 others)

Entity Metadata (schema)

abstracts[] {'sha1': '3d5b13db43c0667ebc3e82f6964aa9d066266945', 'content': 'With the rapid development of deep learning, training Big Models (BMs) for\nmultiple downstream tasks becomes a popular paradigm. Researchers have achieved\nvarious outcomes in the construction of BMs and the BM application in many\nfields. At present, there is a lack of research work that sorts out the overall\nprogress of BMs and guides the follow-up research. In this paper, we cover not\nonly the BM technologies themselves but also the prerequisites for BM training\nand applications with BMs, dividing the BM review into four parts: Resource,\nModels, Key Technologies and Application. We introduce 16 specific BM-related\ntopics in those four parts, they are Data, Knowledge, Computing System,\nParallel Training System, Language Model, Vision Model, Multi-modal Model,\nTheory&Interpretability, Commonsense Reasoning, Reliability&Security,\nGovernance, Evaluation, Machine Translation, Text Generation, Dialogue and\nProtein Research. In each topic, we summarize clearly the current studies and\npropose some future research directions. At the end of this paper, we conclude\nthe further development of BMs in a more general view.', 'mimetype': 'text/plain', 'lang': 'en'}
container
container_id
contribs[] {'index': 0, 'creator_id': None, 'creator': None, 'raw_name': 'Sha Yuan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 1, 'creator_id': None, 'creator': None, 'raw_name': 'Hanyu Zhao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 2, 'creator_id': None, 'creator': None, 'raw_name': 'Shuai Zhao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 3, 'creator_id': None, 'creator': None, 'raw_name': 'Jiahong Leng', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 4, 'creator_id': None, 'creator': None, 'raw_name': 'Yangxiao Liang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 5, 'creator_id': None, 'creator': None, 'raw_name': 'Xiaozhi Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 6, 'creator_id': None, 'creator': None, 'raw_name': 'Jifan Yu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 7, 'creator_id': None, 'creator': None, 'raw_name': 'Xin Lv', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 8, 'creator_id': None, 'creator': None, 'raw_name': 'Zhou Shao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 9, 'creator_id': None, 'creator': None, 'raw_name': 'Jiaao He', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 10, 'creator_id': None, 'creator': None, 'raw_name': 'Yankai Lin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 11, 'creator_id': None, 'creator': None, 'raw_name': 'Xu Han', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 12, 'creator_id': None, 'creator': None, 'raw_name': 'Zhenghao Liu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 13, 'creator_id': None, 'creator': None, 'raw_name': 'Ning Ding', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 14, 'creator_id': None, 'creator': None, 'raw_name': 'Yongming Rao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 15, 'creator_id': None, 'creator': None, 'raw_name': 'Yizhao Gao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 16, 'creator_id': None, 'creator': None, 'raw_name': 'Liang Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 17, 'creator_id': None, 'creator': None, 'raw_name': 'Ming Ding', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 18, 'creator_id': None, 'creator': None, 'raw_name': 'Cong Fang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 19, 'creator_id': None, 'creator': None, 'raw_name': 'Yisen Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 20, 'creator_id': None, 'creator': None, 'raw_name': 'Mingsheng Long', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 21, 'creator_id': None, 'creator': None, 'raw_name': 'Jing Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 22, 'creator_id': None, 'creator': None, 'raw_name': 'Yinpeng Dong', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 23, 'creator_id': None, 'creator': None, 'raw_name': 'Tianyu Pang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 24, 'creator_id': None, 'creator': None, 'raw_name': 'Peng Cui', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 25, 'creator_id': None, 'creator': None, 'raw_name': 'Lingxiao Huang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 26, 'creator_id': None, 'creator': None, 'raw_name': 'Zheng Liang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 27, 'creator_id': None, 'creator': None, 'raw_name': 'Huawei Shen', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 28, 'creator_id': None, 'creator': None, 'raw_name': 'Hui Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 29, 'creator_id': None, 'creator': None, 'raw_name': 'Quanshi Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 30, 'creator_id': None, 'creator': None, 'raw_name': 'Qingxiu Dong', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 31, 'creator_id': None, 'creator': None, 'raw_name': 'Zhixing Tan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 32, 'creator_id': None, 'creator': None, 'raw_name': 'Mingxuan Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 33, 'creator_id': None, 'creator': None, 'raw_name': 'Shuo Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 34, 'creator_id': None, 'creator': None, 'raw_name': 'Long Zhou', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 35, 'creator_id': None, 'creator': None, 'raw_name': 'Haoran Li', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 36, 'creator_id': None, 'creator': None, 'raw_name': 'Junwei Bao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 37, 'creator_id': None, 'creator': None, 'raw_name': 'Yingwei Pan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 38, 'creator_id': None, 'creator': None, 'raw_name': 'Weinan Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 39, 'creator_id': None, 'creator': None, 'raw_name': 'Zhou Yu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 40, 'creator_id': None, 'creator': None, 'raw_name': 'Rui Yan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 41, 'creator_id': None, 'creator': None, 'raw_name': 'Chence Shi', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 42, 'creator_id': None, 'creator': None, 'raw_name': 'Minghao Xu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 43, 'creator_id': None, 'creator': None, 'raw_name': 'Zuobai Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 44, 'creator_id': None, 'creator': None, 'raw_name': 'Guoqiang Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 45, 'creator_id': None, 'creator': None, 'raw_name': 'Xiang Pan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 46, 'creator_id': None, 'creator': None, 'raw_name': 'Mengjie Li', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 47, 'creator_id': None, 'creator': None, 'raw_name': 'Xiaoyu Chu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 48, 'creator_id': None, 'creator': None, 'raw_name': 'Zijun Yao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 49, 'creator_id': None, 'creator': None, 'raw_name': 'Fangwei Zhu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 50, 'creator_id': None, 'creator': None, 'raw_name': 'Shulin Cao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 51, 'creator_id': None, 'creator': None, 'raw_name': 'Weicheng Xue', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 52, 'creator_id': None, 'creator': None, 'raw_name': 'Zixuan Ma', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 53, 'creator_id': None, 'creator': None, 'raw_name': 'Zhengyan Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 54, 'creator_id': None, 'creator': None, 'raw_name': 'Shengding Hu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 55, 'creator_id': None, 'creator': None, 'raw_name': 'Yujia Qin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 56, 'creator_id': None, 'creator': None, 'raw_name': 'Chaojun Xiao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 57, 'creator_id': None, 'creator': None, 'raw_name': 'Zheni Zeng', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 58, 'creator_id': None, 'creator': None, 'raw_name': 'Ganqu Cui', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 59, 'creator_id': None, 'creator': None, 'raw_name': 'Weize Chen', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 60, 'creator_id': None, 'creator': None, 'raw_name': 'Weilin Zhao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 61, 'creator_id': None, 'creator': None, 'raw_name': 'Yuan Yao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 62, 'creator_id': None, 'creator': None, 'raw_name': 'Peng Li', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 63, 'creator_id': None, 'creator': None, 'raw_name': 'Wenzhao Zheng', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 64, 'creator_id': None, 'creator': None, 'raw_name': 'Wenliang Zhao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 65, 'creator_id': None, 'creator': None, 'raw_name': 'Ziyi Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 66, 'creator_id': None, 'creator': None, 'raw_name': 'Borui Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 67, 'creator_id': None, 'creator': None, 'raw_name': 'Nanyi Fei', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 68, 'creator_id': None, 'creator': None, 'raw_name': 'Anwen Hu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 69, 'creator_id': None, 'creator': None, 'raw_name': 'Zenan Ling', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 70, 'creator_id': None, 'creator': None, 'raw_name': 'Haoyang Li', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 71, 'creator_id': None, 'creator': None, 'raw_name': 'Boxi Cao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 72, 'creator_id': None, 'creator': None, 'raw_name': 'Xianpei Han', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 73, 'creator_id': None, 'creator': None, 'raw_name': 'Weidong Zhan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 74, 'creator_id': None, 'creator': None, 'raw_name': 'Baobao Chang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 75, 'creator_id': None, 'creator': None, 'raw_name': 'Hao Sun', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 76, 'creator_id': None, 'creator': None, 'raw_name': 'Jiawen Deng', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 77, 'creator_id': None, 'creator': None, 'raw_name': 'Chujie Zheng', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 78, 'creator_id': None, 'creator': None, 'raw_name': 'Juanzi Li', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 79, 'creator_id': None, 'creator': None, 'raw_name': 'Lei Hou', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 80, 'creator_id': None, 'creator': None, 'raw_name': 'Xigang Cao', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 81, 'creator_id': None, 'creator': None, 'raw_name': 'Jidong Zhai', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 82, 'creator_id': None, 'creator': None, 'raw_name': 'Zhiyuan Liu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 83, 'creator_id': None, 'creator': None, 'raw_name': 'Maosong Sun', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 84, 'creator_id': None, 'creator': None, 'raw_name': 'Jiwen Lu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 85, 'creator_id': None, 'creator': None, 'raw_name': 'Zhiwu Lu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 86, 'creator_id': None, 'creator': None, 'raw_name': 'Qin Jin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 87, 'creator_id': None, 'creator': None, 'raw_name': 'Ruihua Song', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 88, 'creator_id': None, 'creator': None, 'raw_name': 'Ji-Rong Wen', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 89, 'creator_id': None, 'creator': None, 'raw_name': 'Zhouchen Lin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 90, 'creator_id': None, 'creator': None, 'raw_name': 'Liwei Wang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 91, 'creator_id': None, 'creator': None, 'raw_name': 'Hang Su', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 92, 'creator_id': None, 'creator': None, 'raw_name': 'Jun Zhu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 93, 'creator_id': None, 'creator': None, 'raw_name': 'Zhifang Sui', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 94, 'creator_id': None, 'creator': None, 'raw_name': 'Jiajun Zhang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 95, 'creator_id': None, 'creator': None, 'raw_name': 'Yang Liu', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 96, 'creator_id': None, 'creator': None, 'raw_name': 'Xiaodong He', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 97, 'creator_id': None, 'creator': None, 'raw_name': 'Minlie Huang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 98, 'creator_id': None, 'creator': None, 'raw_name': 'Jian Tang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 99, 'creator_id': None, 'creator': None, 'raw_name': 'Jie Tang', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
ext_ids {'doi': None, 'wikidata_qid': None, 'isbn13': None, 'pmid': None, 'pmcid': None, 'core': None, 'arxiv': '2203.14101v3', 'jstor': None, 'ark': None, 'mag': None, 'doaj': None, 'dblp': None, 'oai': None, 'hdl': None}
files
filesets
issue
language en
license_slug ARXIV-1.0
number
original_title
pages
publisher
refs []
release_date 2022-04-02
release_stage submitted
release_type article
release_year 2022
subtitle
title A Roadmap for Big Model
version v3
volume
webcaptures
withdrawn_date
withdrawn_status
withdrawn_year
work_id w3hf3seoevbptfkufmvlzrw5bu
As JSON via API

Extra Metadata (raw JSON)

arxiv.base_id 2203.14101
arxiv.categories ['cs.LG', 'cs.AI', 'cs.CL']
arxiv.comments arXiv admin note: text overlap with arXiv:2107.06499 by other authors