Foundation Models for Physics

AI has rapidly found its way into our daily lives. We use it for coding, for writing texts, for generating images and much more. But until recently, the development of AI for applications in the physical world was a bit lagging. Now, foundation models and world models promise to lower that barrier.

See also our page on Physics AI and our whitepaper on Physics AI.

What is a Foundation Model?

Foundation models are very large machine learning models that are trained on a wide variety of data. The idea is that a foundation model learns the generic aspects of an entire domain. With additional finetuning, it can then be targeted to specific use cases. This makes building applications much easier: there is no longer a need to train an entire model from scratch. This saves enormous amounts of time and money.

Physics Foundation Models

Most current foundation models are built on text data, images, video and sound. This is relatively easy as there are huge quantities of such data available on the internet. Also, this type of foundation model can be finetuned to applications that have a big market, like coding assistants and generators for images, video and music.

Foundation models for natural processes are harder to develop. This is partly because it is not easy to compile a large dataset to train an foundation model for a wide range of natural processes. Even within a single scientific domain, like physics, there is a wide variety of processes and associated data. When going to multi-physics, including for example also chemistry and biology, it’s even harder to compile a dataset that sufficiently covers the field.

But some large datasets are now available (like The Well) and researchers have started to use them for physics foundation models in much the same way as the large language models that now drive most AI-applications. Notable examples are Poseidon, Walrus and PhysiX. The research is progressing fast and new examples will for sure be emerging quickly.

World Models

At the same time, there is a separate but related development called world models. An original work here is the paper by Ha and Schmidhuber. Nvidia has strongly embraced this concept because of the potential use in robotics.

An important aspect of world models is that they build an internal representation of the world. This means that objects in their world do not suddenly vanish or appear. Even if they are temporarily hidden, they still exist in the internal representation. Obviously, this consistency is a very nice feature for AI to be applied in the physical world.

What about scientific machine learning?

The attention to foundation models and world models tends to overshadow the work that is going on in scientific machine learning, which is the combination of machine learning with more traditional scientific computing approaches. These methods are aimed at making models comply with the laws of nature, at least to a certain degree that is somewhat controllable.

At the moment, this principle does not seem to be used in physics foundation models, at least not in all of them: the foundation models that were mentioned earlier are entirely data driven. In that case, there is no guarantee that they will adhere to the laws of physics. As many use cases of physics AI require a high degree of reliability, the applications of such purely data-driven foundation models seem to be limited.

A similar argument can be made for world models. Again, these do not inherently comply with the laws of nature, even if they are less unpredictable because of their internal world representation.

Now what?

Even though they learn the physics implicitly and not explicitly, world models are already widely used for applications with a physical component.

Physics foundation models, on the other hand, are still under development as we write this. If they prove to be reliable enough, it will mean a serious breakthrough in the use of AI for applications in physical reality. Environmental models are an obvious example, but applications in the design or optimization of physical systems are also conceivable, and there are many other use cases.

But using physics foundation models I not necessarily easy.  For finetuning a physics foundation model, a large dataset about the specific application is still needed and not all industries will have such datasets. It would be great if companies and institutions would share their data to create sets of sufficient scale, but that typically runs into issues of trade secrets and confidentiality.

Using the laws of nature in the training process would help, because this would add extra information to the training and could give some guarantees in terms of reliability. Such developments are already emerging, for example in the vinci project.

In any case, it’s good to keep an eye on these developments. As stated in the beginning of this blog post, there is a fair chance that the barriers to using AI for the physical world will soon be seriously lowered, but we are not entirely there yet.