Uploaded image for project: 'Nuxeo Drive '
  1. Nuxeo Drive
  2. NXDRIVE-768

Fix encoding in report generation

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.1
    • Component/s: Tests

      Description

      When running into the random failure described by NXDRIVE-719 under Linux, we get this error:

      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/tests/test_windows.py .FE
      15:13:41 [SLAVE] [INFO]      [exec] =========================== short test summary info ============================
      15:13:41 [SLAVE] [INFO]      [exec] ERROR nuxeo-drive-client/tests/test_windows.py::TestWindows::test_local_replace
      15:13:41 [SLAVE] [INFO]      [exec] FAIL nuxeo-drive-client/tests/test_windows.py::TestWindows::test_local_replace
      15:13:41 [SLAVE] [INFO]      [exec] ==================================== ERRORS ====================================
      15:13:41 [SLAVE] [INFO]      [exec] _____________ ERROR at teardown of TestWindows.test_local_replace ______________
      15:13:41 [SLAVE] [INFO]      [exec] self = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] result = <TestCaseFunction 'test_local_replace'>
      15:13:41 [SLAVE] [INFO]      [exec]     def run(self, result=None):
      15:13:41 [SLAVE] [INFO]      [exec]         orig_result = result
      15:13:41 [SLAVE] [INFO]      [exec]         if result is None:
      15:13:41 [SLAVE] [INFO]      [exec]             result = self.defaultTestResult()
      15:13:41 [SLAVE] [INFO]      [exec]             startTestRun = getattr(result, 'startTestRun', None)
      15:13:41 [SLAVE] [INFO]      [exec]             if startTestRun is not None:
      15:13:41 [SLAVE] [INFO]      [exec]                 startTestRun()
      15:13:41 [SLAVE] [INFO]      [exec]         self._resultForDoCleanups = result
      15:13:41 [SLAVE] [INFO]      [exec]         result.startTest(self)
      15:13:41 [SLAVE] [INFO]      [exec]         testMethod = getattr(self, self._testMethodName)
      15:13:41 [SLAVE] [INFO]      [exec]         if (getattr(self.__class__, "__unittest_skip__", False) or
      15:13:41 [SLAVE] [INFO]      [exec]             getattr(testMethod, "__unittest_skip__", False)):
      15:13:41 [SLAVE] [INFO]      [exec]             # If the class or method was skipped.
      15:13:41 [SLAVE] [INFO]      [exec]             try:
      15:13:41 [SLAVE] [INFO]      [exec]                 skip_why = (getattr(self.__class__, '__unittest_skip_why__', '')
      15:13:41 [SLAVE] [INFO]      [exec]                             or getattr(testMethod, '__unittest_skip_why__', ''))
      15:13:41 [SLAVE] [INFO]      [exec]                 self._addSkip(result, skip_why)
      15:13:41 [SLAVE] [INFO]      [exec]             finally:
      15:13:41 [SLAVE] [INFO]      [exec]                 result.stopTest(self)
      15:13:41 [SLAVE] [INFO]      [exec]             return
      15:13:41 [SLAVE] [INFO]      [exec]         try:
      15:13:41 [SLAVE] [INFO]      [exec]             success = False
      15:13:41 [SLAVE] [INFO]      [exec]             try:
      15:13:41 [SLAVE] [INFO]      [exec]                 self.setUp()
      15:13:41 [SLAVE] [INFO]      [exec]             except SkipTest as e:
      15:13:41 [SLAVE] [INFO]      [exec]                 self._addSkip(result, str(e))
      15:13:41 [SLAVE] [INFO]      [exec]             except KeyboardInterrupt:
      15:13:41 [SLAVE] [INFO]      [exec]                 raise
      15:13:41 [SLAVE] [INFO]      [exec]             except:
      15:13:41 [SLAVE] [INFO]      [exec]                 result.addError(self, sys.exc_info())
      15:13:41 [SLAVE] [INFO]      [exec]             else:
      15:13:41 [SLAVE] [INFO]      [exec]                 try:
      15:13:41 [SLAVE] [INFO]      [exec]                     testMethod()
      15:13:41 [SLAVE] [INFO]      [exec]                 except KeyboardInterrupt:
      15:13:41 [SLAVE] [INFO]      [exec]                     raise
      15:13:41 [SLAVE] [INFO]      [exec]                 except self.failureException:
      15:13:41 [SLAVE] [INFO]      [exec]                     result.addFailure(self, sys.exc_info())
      15:13:41 [SLAVE] [INFO]      [exec]                 except _ExpectedFailure as e:
      15:13:41 [SLAVE] [INFO]      [exec]                     addExpectedFailure = getattr(result, 'addExpectedFailure', None)
      15:13:41 [SLAVE] [INFO]      [exec]                     if addExpectedFailure is not None:
      15:13:41 [SLAVE] [INFO]      [exec]                         addExpectedFailure(self, e.exc_info)
      15:13:41 [SLAVE] [INFO]      [exec]                     else:
      15:13:41 [SLAVE] [INFO]      [exec]                         warnings.warn("TestResult has no addExpectedFailure method, reporting as passes",
      15:13:41 [SLAVE] [INFO]      [exec]                                       RuntimeWarning)
      15:13:41 [SLAVE] [INFO]      [exec]                         result.addSuccess(self)
      15:13:41 [SLAVE] [INFO]      [exec]                 except _UnexpectedSuccess:
      15:13:41 [SLAVE] [INFO]      [exec]                     addUnexpectedSuccess = getattr(result, 'addUnexpectedSuccess', None)
      15:13:41 [SLAVE] [INFO]      [exec]                     if addUnexpectedSuccess is not None:
      15:13:41 [SLAVE] [INFO]      [exec]                         addUnexpectedSuccess(self)
      15:13:41 [SLAVE] [INFO]      [exec]                     else:
      15:13:41 [SLAVE] [INFO]      [exec]                         warnings.warn("TestResult has no addUnexpectedSuccess method, reporting as failures",
      15:13:41 [SLAVE] [INFO]      [exec]                                       RuntimeWarning)
      15:13:41 [SLAVE] [INFO]      [exec]                         result.addFailure(self, sys.exc_info())
      15:13:41 [SLAVE] [INFO]      [exec]                 except SkipTest as e:
      15:13:41 [SLAVE] [INFO]      [exec]                     self._addSkip(result, str(e))
      15:13:41 [SLAVE] [INFO]      [exec]                 except:
      15:13:41 [SLAVE] [INFO]      [exec]                     result.addError(self, sys.exc_info())
      15:13:41 [SLAVE] [INFO]      [exec]                 else:
      15:13:41 [SLAVE] [INFO]      [exec]                     success = True
      15:13:41 [SLAVE] [INFO]      [exec]                 try:
      15:13:41 [SLAVE] [INFO]      [exec] >                   self.tearDown()
      15:13:41 [SLAVE] [INFO]      [exec] cleanUpSuccess = True
      15:13:41 [SLAVE] [INFO]      [exec] orig_result = <TestCaseFunction 'test_local_replace'>
      15:13:41 [SLAVE] [INFO]      [exec] result     = <TestCaseFunction 'test_local_replace'>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] success    = False
      15:13:41 [SLAVE] [INFO]      [exec] testMethod = <bound method TestWindows.test_local_replace of <tests.test_windows.TestWindows testMethod=test_local_replace>>
      15:13:41 [SLAVE] [INFO]      [exec] ../deploy-dir/.pyenv/versions/2.7.13/lib/python2.7/unittest/case.py:358: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec]     def tearDown(self):
      15:13:41 [SLAVE] [INFO]      [exec]         unittest.TestCase.tearDown(self)
      15:13:41 [SLAVE] [INFO]      [exec]         if not self.tearedDown:
      15:13:41 [SLAVE] [INFO]      [exec] >           self.tearDownApp()
      15:13:41 [SLAVE] [INFO]      [exec] self       = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/tests/common_unit_test.py:559: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] server_profile = None
      15:13:41 [SLAVE] [INFO]      [exec]     def tearDownApp(self, server_profile=None):
      15:13:41 [SLAVE] [INFO]      [exec]         if self.tearedDown:
      15:13:41 [SLAVE] [INFO]      [exec]             return
      15:13:41 [SLAVE] [INFO]      [exec]         if sys.exc_info() != (None, None, None):
      15:13:41 [SLAVE] [INFO]      [exec] >           self.generate_report()
      15:13:41 [SLAVE] [INFO]      [exec] self       = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] server_profile = None
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/tests/common_unit_test.py:565: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec]     def generate_report(self):
      15:13:41 [SLAVE] [INFO]      [exec]         if "REPORT_PATH" not in os.environ:
      15:13:41 [SLAVE] [INFO]      [exec]             return
      15:13:41 [SLAVE] [INFO]      [exec]         report_path = os.path.join(os.environ["REPORT_PATH"], self.id())
      15:13:41 [SLAVE] [INFO]      [exec] >       self.manager_1.generate_report(report_path)
      15:13:41 [SLAVE] [INFO]      [exec] report_path = '/opt/jenkins/workspace/-766-conflict-local-resolve-4BUUDYKOVSB7YE2CAS7SBIGXNTN3QKJ3TRXAPSFJOTILXQTHHNAQ/sources/tests.test_windows.TestWindows.test_local_replace'
      15:13:41 [SLAVE] [INFO]      [exec] self       = <tests.test_windows.TestWindows testMethod=test_local_replace>
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/tests/common_unit_test.py:685: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <nxdrive.manager.Manager object at 0x7fb03da62b98>
      15:13:41 [SLAVE] [INFO]      [exec] path = '/opt/jenkins/workspace/-766-conflict-local-resolve-4BUUDYKOVSB7YE2CAS7SBIGXNTN3QKJ3TRXAPSFJOTILXQTHHNAQ/sources/tests.test_windows.TestWindows.test_local_replace'
      15:13:41 [SLAVE] [INFO]      [exec]     def generate_report(self, path=None):
      15:13:41 [SLAVE] [INFO]      [exec]         from nxdrive.report import Report
      15:13:41 [SLAVE] [INFO]      [exec]         report = Report(self, path)
      15:13:41 [SLAVE] [INFO]      [exec] >       report.generate()
      15:13:41 [SLAVE] [INFO]      [exec] Report     = <class 'nxdrive.report.Report'>
      15:13:41 [SLAVE] [INFO]      [exec] path       = '/opt/jenkins/workspace/-766-conflict-local-resolve-4BUUDYKOVSB7YE2CAS7SBIGXNTN3QKJ3TRXAPSFJOTILXQTHHNAQ/sources/tests.test_windows.TestWindows.test_local_replace'
      15:13:41 [SLAVE] [INFO]      [exec] report     = <nxdrive.report.Report object at 0x7fb03deeeb50>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <nxdrive.manager.Manager object at 0x7fb03da62b98>
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/nxdrive/manager.py:816: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <nxdrive.report.Report object at 0x7fb03deeeb50>
      15:13:41 [SLAVE] [INFO]      [exec]     def generate(self):
      15:13:41 [SLAVE] [INFO]      [exec]         log.debug("Create report '%s'", self._report_name)
      15:13:41 [SLAVE] [INFO]      [exec]         log.debug("Manager metrics: '%s'", self._manager.get_metrics())
      15:13:41 [SLAVE] [INFO]      [exec]         with ZipFile(self._zipfile, 'w') as myzip:
      15:13:41 [SLAVE] [INFO]      [exec]             dao = self._manager.get_dao()
      15:13:41 [SLAVE] [INFO]      [exec]             self.copy_db(myzip, dao)
      15:13:41 [SLAVE] [INFO]      [exec]             for engine in self._manager.get_engines().values():
      15:13:41 [SLAVE] [INFO]      [exec]                 log.debug("Engine metrics: '%s'", engine.get_metrics())
      15:13:41 [SLAVE] [INFO]      [exec]                 self.copy_db(myzip, engine.get_dao())
      15:13:41 [SLAVE] [INFO]      [exec]                 # Might want threads too here
      15:13:41 [SLAVE] [INFO]      [exec]             self.copy_logs(myzip)
      15:13:41 [SLAVE] [INFO]      [exec] >           myzip.writestr("debug.log", self._export_logs().encode('utf-8', errors="ignore").strip())
      15:13:41 [SLAVE] [INFO]      [exec] dao        = <nxdrive.engine.dao.sqlite.ManagerDAO object at 0x7fb03da622b0>
      15:13:41 [SLAVE] [INFO]      [exec] engine     = <nxdrive.engine.engine.Engine object at 0x7fb048906a68>
      15:13:41 [SLAVE] [INFO]      [exec] myzip      = <zipfile.ZipFile object at 0x7fb03f399d50>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <nxdrive.report.Report object at 0x7fb03deeeb50>
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/nxdrive/report.py:81: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <nxdrive.report.Report object at 0x7fb03deeeb50>
      15:13:41 [SLAVE] [INFO]      [exec]     def _export_logs(self):
      15:13:41 [SLAVE] [INFO]      [exec]         logs = u""
      15:13:41 [SLAVE] [INFO]      [exec]         logger = get_logger(None)
      15:13:41 [SLAVE] [INFO]      [exec]         handler = get_handler(logger, "memory")
      15:13:41 [SLAVE] [INFO]      [exec]         log_buffer = handler.get_buffer(MAX_LOG_DISPLAYED)
      15:13:41 [SLAVE] [INFO]      [exec]         for record in log_buffer:
      15:13:41 [SLAVE] [INFO]      [exec]             try:
      15:13:41 [SLAVE] [INFO]      [exec]                 log = handler.format(record).decode("utf-8", errors="replace")
      15:13:41 [SLAVE] [INFO]      [exec]             except UnicodeEncodeError:
      15:13:41 [SLAVE] [INFO]      [exec] >               log = handler.format(record)
      15:13:41 [SLAVE] [INFO]      [exec] handler    = <nxdrive.logging_config.CustomMemoryHandler object at 0x7fb054b46f50>
      15:13:41 [SLAVE] [INFO]      [exec] log        = '2017-03-16 15:07:58,956 3853 140395351795456 DEBUG    nxdrive.engine.queue_manager creating additional file processor'
      15:13:41 [SLAVE] [INFO]      [exec] log_buffer = [<logging.LogRecord object at 0x7fb03f399210>, <logging.LogRecord object at 0x7fb03f3996d0>, <logging.LogRecord object...ct at 0x7fb04426dad0>, <logging.LogRecord object at 0x7fb04426d110>, <logging.LogRecord object at 0x7fb044d83950>, ...]
      15:13:41 [SLAVE] [INFO]      [exec] logger     = <logging.RootLogger object at 0x7fb05114da90>
      15:13:41 [SLAVE] [INFO]      [exec] logs       = "2017-03-16 15:10:02,662 3853 140395045488384 DEBUG    nxdrive.report     Engine metrics: '{'files_size': 0, 'sync_fil...2017-03-16 15:07:58,956 3853 140395351795456 DEBUG    nxdrive.engine.queue_manager creating additional file processor
      15:13:41 [SLAVE] [INFO]      [exec] "
      15:13:41 [SLAVE] [INFO]      [exec] record     = <logging.LogRecord object at 0x7fb03e422850>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <nxdrive.report.Report object at 0x7fb03deeeb50>
      15:13:41 [SLAVE] [INFO]      [exec] nuxeo-drive-client/nxdrive/report.py:66: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <nxdrive.logging_config.CustomMemoryHandler object at 0x7fb054b46f50>
      15:13:41 [SLAVE] [INFO]      [exec] record = <logging.LogRecord object at 0x7fb03e422850>
      15:13:41 [SLAVE] [INFO]      [exec]     def format(self, record):
      15:13:41 [SLAVE] [INFO]      [exec]         """
      15:13:41 [SLAVE] [INFO]      [exec]             Format the specified record.
      15:13:41 [SLAVE] [INFO]      [exec]             If a formatter is set, use it. Otherwise, use the default formatter
      15:13:41 [SLAVE] [INFO]      [exec]             for the module.
      15:13:41 [SLAVE] [INFO]      [exec]             """
      15:13:41 [SLAVE] [INFO]      [exec]         if self.formatter:
      15:13:41 [SLAVE] [INFO]      [exec]             fmt = self.formatter
      15:13:41 [SLAVE] [INFO]      [exec]         else:
      15:13:41 [SLAVE] [INFO]      [exec]             fmt = _defaultFormatter
      15:13:41 [SLAVE] [INFO]      [exec] >       return fmt.format(record)
      15:13:41 [SLAVE] [INFO]      [exec] fmt        = <logging.Formatter object at 0x7fb054b46c50>
      15:13:41 [SLAVE] [INFO]      [exec] record     = <logging.LogRecord object at 0x7fb03e422850>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <nxdrive.logging_config.CustomMemoryHandler object at 0x7fb054b46f50>
      15:13:41 [SLAVE] [INFO]      [exec] ../deploy-dir/.pyenv/versions/2.7.13/lib/python2.7/logging/__init__.py:734: 
      15:13:41 [SLAVE] [INFO]      [exec] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      15:13:41 [SLAVE] [INFO]      [exec] self = <logging.Formatter object at 0x7fb054b46c50>
      15:13:41 [SLAVE] [INFO]      [exec] record = <logging.LogRecord object at 0x7fb03e422850>
      15:13:41 [SLAVE] [INFO]      [exec]     def format(self, record):
      15:13:41 [SLAVE] [INFO]      [exec]         """
      15:13:41 [SLAVE] [INFO]      [exec]             Format the specified record as text.
      15:13:41 [SLAVE] [INFO]      [exec]             The record's attribute dictionary is used as the operand to a
      15:13:41 [SLAVE] [INFO]      [exec]             string formatting operation which yields the returned string.
      15:13:41 [SLAVE] [INFO]      [exec]             Before formatting the dictionary, a couple of preparatory steps
      15:13:41 [SLAVE] [INFO]      [exec]             are carried out. The message attribute of the record is computed
      15:13:41 [SLAVE] [INFO]      [exec]             using LogRecord.getMessage(). If the formatting string uses the
      15:13:41 [SLAVE] [INFO]      [exec]             time (as determined by a call to usesTime(), formatTime() is
      15:13:41 [SLAVE] [INFO]      [exec]             called to format the event time. If there is exception information,
      15:13:41 [SLAVE] [INFO]      [exec]             it is formatted using formatException() and appended to the message.
      15:13:41 [SLAVE] [INFO]      [exec]             """
      15:13:41 [SLAVE] [INFO]      [exec]         record.message = record.getMessage()
      15:13:41 [SLAVE] [INFO]      [exec]         if self.usesTime():
      15:13:41 [SLAVE] [INFO]      [exec]             record.asctime = self.formatTime(record, self.datefmt)
      15:13:41 [SLAVE] [INFO]      [exec]         try:
      15:13:41 [SLAVE] [INFO]      [exec] >           s = self._fmt % record.__dict__
      15:13:41 [SLAVE] [INFO]      [exec] E           UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 56: ordinal not in range(128)
      15:13:41 [SLAVE] [INFO]      [exec] record     = <logging.LogRecord object at 0x7fb03e422850>
      15:13:41 [SLAVE] [INFO]      [exec] self       = <logging.Formatter object at 0x7fb054b46c50>
      15:13:41 [SLAVE] [INFO]      [exec] ../deploy-dir/.pyenv/versions/2.7.13/lib/python2.7/logging/__init__.py:469: UnicodeEncodeError
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 4 hours
                  4h