-
Type:
Bug
-
Status: Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Jenkins X
-
Team:PLATFORM
-
Sprint:nxplatform 11.1.21
-
Story Points:1
Currently the platform team's internal Nexus has a 100Gi volume to store blobs.
This is not enough regarding the lack of cleaning policy (that will have to be set up).
Since the partition is full, the pipelines are failing with 500 errors such as:
17:54:03 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.1:deploy (default-deploy) on project nuxeo-nxr-jsf-ui: Failed to deploy artifacts: Could not transfer artifact org.nuxeo.ecm.distribution:nuxeo-nxr-jsf-ui:zip:PR-3510-11.1-20191030.165400-1 from/to nexus (http://nexus/repository/maven-public/): Failed to transfer file http://nexus/repository/maven-public/org/nuxeo/ecm/distribution/nuxeo-nxr-jsf-ui/PR-3510-11.1-SNAPSHOT/nuxeo-nxr-jsf-ui-PR-3510-11.1-20191030.165400-1.zip with status code 500 -> [Help 1]
Indeed, when connecting to the pod, we can see that:
$ jx rsh jenkins-x-nexus-74d978cc79-4wx44 [root@jenkins-x-nexus-74d978cc79-4wx44 /]# df Filesystem 1K-blocks Used Available Use% Mounted on overlay 98868448 81740560 17111504 83% / /dev/sdj 103146424 98933320 4196720 96% /nexus-data
and in the logs:
2019-10-31 00:00:00,597+0000 ERROR [nexus_QuartzSchedulerThread] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Exception `739519D1` in storage `plocal:/nexus-data/db/config`: 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
com.orientechnologies.orient.core.exception.OLowDiskSpaceException: Error occurred while executing a write operation to database 'config' due to limited free space on the disk (4086 MB). The database is now working in read-only mode. Please close the database (or stop OrientDB), make room on your hard drive and then reopen the database. The minimal required space is 4096 MB. Required space is now set to 4096MB (you can change it by setting parameter storage.diskCache.diskFreeSpaceLimit) .^M
DB name="config"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5073)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1729)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
at org.sonatype.nexus.orient.transaction.OrientTransaction.commit(OrientTransaction.java:83)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at org.sonatype.nexus.orient.transaction.OrientOperations.call(OrientOperations.java:56)
at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.acquireNextTriggers(JobStoreImpl.java:923)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:287)
Let's increase the volume size to 300Gi.
