Issue

The application crashes due to a metaspace out of memory issue and the following logs will be visible in the application logs:

2023-08-31T10:05:38.286105 [APP/PROC/WEB/0] #
2023-08-31T10:05:38.286121 [APP/PROC/WEB/0] # java.lang.OutOfMemoryError: Metaspace
2023-08-31T10:05:38.286124 [APP/PROC/WEB/0] # -XX:OnOutOfMemoryError="kill -s USR2 7"
2023-08-31T10:05:38.286127 [APP/PROC/WEB/0] #   Executing /bin/sh -c "kill -s USR2 7"...
2023-08-31T10:05:38.287591 [APP/PROC/WEB/0] INFO: Waiting for the application to shutdown...

After a problem that crashes the Java Virtual Machine (JVM), the application will recover automatically and be rebooted to minimize the downtime. If multiple instances of the application are running, all user traffic will be sent to the instances that are up and running while the crashed instance is recovering.

Environment

Applications in the Mendix Cloud

Cause

The issue that the application is encountering is caused by the Metaspace running out of memory. In the Metaspace, the Java artifacts are loaded, which, together with other loads of the application, can make the Metaspace run out of memory. Currently, when the error occurs, the affected instance of the application restarts automatically to minimize the business impact.
 
The main cause of the java.lang.OutOfMemoryError: Metaspace is either too many classes or too big classes being loaded to the Metaspace. Since classes are only loaded in the Metaspace when needed, it is being used, so removing classes will always impact the structure of the application (i.e. it is not about removing unused libraries) and will require a refactoring of the application.
 
Solution/Workaround

Below short- and long-term solutions are outlined to address this problem:

• The long-term solution would be to refactor the application such that it uses fewer Java classes. The easiest way of doing this is by reducing the number of external modules since these come with more Java libraries, and thus Java classes that they utilize.
• For the short-term solution or in cases where it is not possible to refactor the application, an alternative solution would be to, increase the Metaspace limit. It is not a bad thing that the application is using too many classes, but it could also be that this only serves as a temporary solution because it does not tackle the potential underlying problem. If the application leaks memory or just loads an unreasonable number of Java classes into Metaspace the above solution will not actually improve anything; it will just postpone the problem. However, it is not possible to determine this without increasing the metaspace limit.
  
  We offer a solution to increase the Metaspace limit with an unsupported environment variable.

Please be aware that Mendix support cannot be of assistance since analyzing and troubleshooting the health and performance of customers' applications or infrastructure is not in the scope of Mendix support, except when we suspect an underlying issue exists in the Mendix Platform. See our documentation on what we expect from you for more information.

Additional information 

For more information, see Mendix support - what we expect from you, JVM object heap & Unsupported environment variable