-
Type: Bug
-
Status: Resolved
-
Priority: Critical
-
Resolution: Fixed
-
Affects Version/s: 10.10
-
Fix Version/s: 10.10-HF36, 11.4, 2021.0
-
Component/s: S3
-
Release Notes Summary:HTTP 500 errors during S3 upload have been fixed.
-
Tags:
-
Backlog priority:800
-
Sprint:nxplatform #20, nxplatform #21, nxplatform #22
-
Story Points:1
S3 upload returns 500 errors on complete.
Steps to reproduce following HF33 in https://nxdev-uat-10-10.beta.nuxeocloud.com/
1) Create a long-running import. The longer the content upload, the easier it seems to reproduce. I reproduced multiple times by trying to upload 200, 50 MB pictures which takes over 4 hours with 5 Mbps upload speed.
2) Wait for one file/batch to encounter a 500 error on complete. Observe in the UI that the upload did not finish and is now "stuck" in an incomplete state.
Nuxeo server error behind the 500.
com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FC55E3AD4FD9E454; S3 Extended Request ID: kmWPmGKDlhw2il2nwuPHTMappM6iOyznA2GU5rqdmz32wWI3VaKsJcna+rQH3BUSR8dmAEvRQkk=) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[aws-java-sdk-core-1.11.738.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) ~[aws-java-sdk-s3-1.11.738.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) ~[aws-java-sdk-s3-1.11.738.jar:?] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1335) ~[aws-java-sdk-s3-1.11.738.jar:?] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1309) ~[aws-java-sdk-s3-1.11.738.jar:?] at org.nuxeo.ecm.core.storage.sql.S3DirectBatchHandler.completeUpload(S3DirectBatchHandler.java:298) ~[nuxeo-core-binarymanager-s3-10.10-HF31.jar:?] at org.nuxeo.ecm.restapi.server.jaxrs.BatchUploadObject.uploadCompleted(BatchUploadObject.java:490) ~[nuxeo-rest-api-server-10.10-HF32.jar:?] at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.4.jar:1.19.4] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.4.jar:1.19.4] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.4.jar:1.19.4] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.4.jar:1.19.4] at org.nuxeo.ecm.webengine.app.jersey.WebEngineServlet.containerService(WebEngineServlet.java:72) [nuxeo-webengine-core-10.10-HF30.jar:?] at org.nuxeo.ecm.webengine.app.jersey.WebEngineServlet.service(WebEngineServlet.java:56) [nuxeo-webengine-core-10.10-HF30.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [servlet-api-9.0.38.jar:4.0.FR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-websocket-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.elasticsearch.ElasticSearchFilter.doFilter(ElasticSearchFilter.java:55) [nuxeo-elasticsearch-core-10.10-HF32.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.webengine.jaxrs.session.SessionCleanupFilter.run(SessionCleanupFilter.java:50) [nuxeo-webengine-jaxrs-10.10-HF08.jar:?] at org.nuxeo.ecm.webengine.jaxrs.HttpFilter.doFilter(HttpFilter.java:49) [nuxeo-webengine-jaxrs-10.10-HF08.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.webengine.app.WebContextFilter.doFilter(WebContextFilter.java:56) [nuxeo-webengine-core-10.10-HF30.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.webengine.jaxrs.context.RequestContextFilter.run(RequestContextFilter.java:48) [nuxeo-webengine-jaxrs-10.10-HF08.jar:?] at org.nuxeo.ecm.webengine.jaxrs.HttpFilter.doFilter(HttpFilter.java:49) [nuxeo-webengine-jaxrs-10.10-HF08.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.webengine.app.HeaderFixFilter.run(HeaderFixFilter.java:62) [nuxeo-webengine-core-10.10-HF30.jar:?] at org.nuxeo.ecm.webengine.jaxrs.HttpFilter.doFilter(HttpFilter.java:49) [nuxeo-webengine-jaxrs-10.10-HF08.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69) [nuxeo-core-management-jtajca-10.10.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:134) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:640) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:431) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:67) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:64) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.restapi.server.APIServlet.service(APIServlet.java:50) [nuxeo-rest-api-server-10.10-HF32.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [servlet-api-9.0.38.jar:4.0.FR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-websocket-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.elasticsearch.ElasticSearchFilter.doFilter(ElasticSearchFilter.java:55) [nuxeo-elasticsearch-core-10.10-HF32.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:43) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoStandbyFilter.doFilter(NuxeoStandbyFilter.java:67) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69) [nuxeo-core-management-jtajca-10.10.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:120) [nuxeo-platform-ui-web-10.10-HF25.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:134) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsCsrfFilter.doFilter(NuxeoCorsCsrfFilter.java:255) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:67) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:64) [nuxeo-platform-web-common-10.10-HF27.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:555) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [catalina-9.0.38.jar:9.0.38] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina-9.0.38.jar:9.0.38] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-coyote-9.0.38.jar:9.0.38] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote-9.0.38.jar:9.0.38] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote-9.0.38.jar:9.0.38] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-coyote-9.0.38.jar:9.0.38] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote-9.0.38.jar:9.0.38] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util-9.0.38.jar:9.0.38] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
- is related to
-
WEBUI-75 500 error blocks import of all uploads
- Resolved
- Is referenced in