by Adam Kolides, Alyna Nawaz, Anshu Rathor, Denzel Beeman, Muzammil Hashmi, Sana
Fatima, David Berdik, Mahmoud Al-Ayyoub, Yaser Jararweh
Simulation Modelling Practice and Theory Vol. 126, July 2023
Link To Paper
Abstract:
With the emergence of foundation models (FMs) that are trained on large amounts of data at
scale and adaptable to a wide range of downstream applications, AI is experiencing a paradigm
revolution. BERT, T5, ChatGPT, GPT-3, Codex, DALL-E, Whisper, and CLIP are now the
foundation for new applications ranging from computer vision to protein sequence study and from
to speech recognition to coding. Earlier models had a reputation of starting from scratch with
each new challenge. The capacity to experiment with, examine, and comprehend the capabilities
and potentials of next-generation FMs is critical to undertaking this research and guiding its
path. Nevertheless, these models are currently inaccessible as the resources required to train these
models are highly concentrated in industry, and even the assets (data, code) required to replicate
their training are frequently not released due to their demand in the real-time industry. At the
moment, only large tech companies such as OpenAI, Google, Facebook, and Baidu can afford to
construct FMs. We attempt to analyze and examine the main capabilities, key implementations,
technological fundamentals, and socially constructed possible consequences of these models inside
this research. Despite the expected widely publicized use of FMs, we still lack a comprehensive
knowledge of how they operate, why they underperform, and what they are even capable of because
of their emerging global qualities. To deal with these problems, we believe that much critical
research on FMs would necessitate extensive multidisciplinary collaboration, given their essentially
social and technical structure. Throughout the investigation, we will also have to deal with the
problem of misrepresentation created by these systems. If FMs live up to their promise, AI might
see far wider commercial use. As researchers studying the ramifications on society, we believe FMs
will lead the way in massive changes. They are closely managed for the time being, so we should
have time to comprehend their implications before they become a major concern.
