-
Type:
Bug
-
Status: Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 3.11.0
-
Fix Version/s: NoFixVersionApplicable
-
Component/s: Technical / Transverse
-
Tags:
-
Sprint:NOS 11.1.9 - 2019-05 02
-
Story Points:2
Problem
There's a critical (if not blocker) issue to handle, it's about Amazon Java SDK. I was working with @rsertelon to diagnostic the problem. The scenario for reproduction: perform a mp-listall operation on dev02. You can observe that all remote packages HTTP requests end with error 500. It means that it impacts all the customers and nothing will work when we deploy 3.11.0+ on production (currently 3.10.0 on prod):
nuxeo-server-9.10-tomcat $ ./bin/nuxeoctl mp-listall ... All packages: Unable to fetch remote packages list: Server returned a code 500 Unable to fetch remote packages list: Server returned a code 500 Unable to fetch remote packages list: Server returned a code 500
Datadog warning:
java.lang.NoSuchMethodError: com.amazonaws.services.s3.internal.ServiceUtils.isMultipartUploadETag(Ljava/lang/String;)Z
at org.nuxeo.ecm.core.storage.sql.S3BinaryManager$S3FileStorage.fetchFile(S3BinaryManager.java:485)
at org.nuxeo.ecm.core.blob.binary.CachingBinaryManager.getFile(CachingBinaryManager.java:181)
at org.nuxeo.ecm.core.blob.binary.LazyBinary.getFile(LazyBinary.java:67)
at com.nuxeo.connect.packagerepository.PackageDocumentAdapter.updatePropertiesFromBinary(PackageDocumentAdapter.java:644)
at com.nuxeo.connect.packagerepository.PackageDocumentAdapter.updatePropertiesFromSQLBlob(PackageDocumentAdapter.java:636)
at com.nuxeo.connect.packagerepository.PackageDocumentAdapter.initFileAttribute(PackageDocumentAdapter.java:620)
at com.nuxeo.connect.packagerepository.PackageDocumentAdapter.getSourceDigest(PackageDocumentAdapter.java:566)
at com.nuxeo.connect.gateway.jaxrs.SimpleDownloadablePackage.<init>(SimpleDownloadablePackage.java:19)
at com.nuxeo.connect.gateway.jaxrs.GatewayProvider.wrap(GatewayProvider.java:235)
at com.nuxeo.connect.gateway.jaxrs.GatewayProvider.getDownloads(GatewayProvider.java:323)
at com.nuxeo.connect.gateway.jaxrs.GatewayProvider.getDownloads(GatewayProvider.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
The possible cause of the problem is the classpath conflicts on Amazon SDK Java S3 (`aws-java-sdk-s3`). On NCO side, you use Nuxeo Addon "Amazon S3 1.7.3" [1] which depends on Nuxeo Server 8.10 [2], which uses Amazon SDK Java S3 1.10.22 [3][4]. While on the NOS side, we depends on Nuxeo Server 8.10 HF44 [5], which users Amazon SDK Java S3 1.11.481 [6]. Actually, on Amazon SDK side, the method `ServiceUtils#isMultipartUploadETag` has been removed from the library in version 1.10.41 [7]. Here's a graph for better illustrate the problem:
NOS Prod (OK) NOS 3.10.1 -------------------------------------------------> Nuxeo Server 8.10 HF36 --> AWS SDK Java 1.10.22 NCO 5.0.1 ---> Nuxeo Addon amazon-s3-online-storage 1.7.3 --> Nuxeo Server 8.10 -------> AWS SDK Java 1.10.22 NOS Dev 02 (Conflicting) NOS 3.11.0+ ------------------------------------------------> Nuxeo Server 8.10 HF44 --> AWS SDK Java 1.11.481 NCO 5.0.1 ---> Nuxeo Addon amazon-s3-online-storage 1.7.3 --> Nuxeo Server 8.10 -------> AWS SDK Java 1.10.22
Solution
After discussion with Cloud Team, the solution is to upgrade NCP NOS platform v5.0.2 and NOS v3.11.0 at the same time. NCP NOS v5.0.2 upgraded the addon amazon-s3-online-storage to version 1.7.6.
References
[1]: https://github.com/nuxeo/nos-platform/blob/v5.0.1/ansible/group_vars/all.yml#L466
[2]: https://github.com/nuxeo/marketplace-amazon-s3/blob/release-1.7.3/pom.xml#L6
[3]: https://github.com/nuxeo/nuxeo/blob/release-8.10/pom.xml#L176
[4]: https://github.com/nuxeo/nuxeo/blob/release-8.10/pom.xml#L4345-L4349
[5]: https://github.com/nuxeo/nos/blob/release-3.11.0/pom.xml#L30
[6]: https://github.com/nuxeo/nuxeo/blob/release-8.10-HF44/pom.xml#L179
[7]: https://github.com/aws/aws-sdk-java/commit/d34bb79de426d03e9624daa3444374fa31c39d94#diff-0c9afa1bae3935d900981c067fe9b098
