-
Type:
Task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Continuous Integration
-
Tags:
-
Sprint:nxplatform #121
-
Story Points:2
Recently the jobs were failing at the Docker build step with:
[2024-09-12T20:02:34.032Z] #34 exporting to image [2024-09-12T20:02:34.032Z] #34 102.8 error: failed commit on ref "intoto-sha256:47e43d535c38675c56445e442062ed210eac8fead3835a1975bf08316fdf38c1": unexpected status from PUT request to https://docker.platform.dev.nuxeo.com/v2/nuxeo/nuxeo/blobs/uploads/eb9206d4-8a77-4181-a49a-1518e6e8f43d?digest=sha256%3A47e43d535c38675c56445e442062ed210eac8fead3835a1975bf08316fdf38c1: 500 Server Error
due to low disk space on the CI internal Nexus:
2024-09-13 08:06:18,954+0000 ERROR [nexus_QuartzSchedulerThread] *SYSTEM com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPooled - $ANSI{green {db=config}} Error on transaction commit `11185EE6`
com.orientechnologies.orient.core.exception.OLowDiskSpaceException: Error occurred while executing a write operation to database 'config' due to limited free space on the disk (141 MB). The database is now working in read-only mode. Please close the database (or stop OrientDB name="config" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5073)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1729)
It wasnt' enough to delete the tmp files in the Nexus storage volume and lowering the minimum free disk space required to allow running daily cleanups.
We've managed to manually increase the size of the Nexus PeristentVolumeClaim (PVC) to 1 TB and run the cleanup tasks.
Let's definitively increase the size of the Nexus PVC to 1 TB.
Let's also increase the size of the Jenkins PVC, recently observed as almost full:
