VMware vCenter Server Appliance /storage/log full

After experiencing stability and performance issues on a vCenter Server Appliance at one of the customer environments, I went on and did some analysis. It appeared that the /storage/log directory was full.

This post describes how to verify if you are (at risk of) running into the same issue, how to solve it and how to prevent it from happening in the future.

Version details

This particular customer was running vCenter Server Appliance 6.0 build 5318203.

Analysis

After having a look at the Administration > System Configuration screen in vSphere Web Client, I noticed several warnings and criticals about disk space usage:
  • Inventory Service
    • Disk space usage is within acceptable limits for 7.0
    • Disk space usage exceeded threshold for 96.0
  • VMware Postgres
    • Filsystem holding VMware Postgres service error logs is running low on available space. Available: 407 MB, Total: 10071 MB
  • Syslog Service
    • Available storage for logs /storage/log reach critical threshold – less than 15% available

Resolution

Using the following VMware KB article, I was able to clean up some of the space and reconfigure the log rotation to prevent the directory from filling up in the future: https://kb.vmware.com/kb/2143565

Additionally, I decided to increase the VMDK where /storage/log is stored with 5GB by following this article: https://kb.vmware.com/kb/2126276

Conclusion

After applying both changes, the environment now looks healthy and performs as expected:

ScreenClip

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s