Project

General

Profile

Actions

Bug #5632

open

better test result summary / notifications

Added by laforge 2 months ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
-
Start date:
07/26/2022
Due date:
% Done:

10%

Spec Reference:

Description

10:55 < fixeria> regarding the testsuite monitoring in Jenkins...
11:19 < fixeria> not sure if you guys still use RSS/Atom, but this is a convinient way to monitor test 
                 results
11:20 < fixeria> https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/rssFailed
11:21 < fixeria> this one is specifically for ttcn3-bts-test, but you can also monitor all TTCN-3 stuff
11:21 < fixeria> https://jenkins.osmocom.org/jenkins/view/TTCN3/rssFailed
11:23 < LaF0rge> fixeria: i did a quick look, but this just gives you the high level overview of which job 
                 failed, right?
11:24 < LaF0rge> fixeria: what we'd want is down on the junit-xml level of granularity where we can track 
                 that sudenyl TC_foo_bar inside a jenkins job is failing
11:29 < LaF0rge> stuff like 
https://jenkins.osmocom.org/jenkins/job/ttcn3-bts-test/lastCompletedBuild/testReport/api/xml
11:29 < fixeria> LaF0rge: it simply says: N more tests failing, N less
11:29 < LaF0rge> fixeria: that cumulative count can be misleading if some old failures are passing but some 
                 new failures are added.
11:29 < LaF0rge> lynxis: ^^ 
11:30 < LaF0rge> I think those testReport/api/xml URLs should cointain what we need
11:30 < LaF0rge> it even has the errorStackTrace with the verbose error message [if there is one]
11:31 < LaF0rge> and it has 'age' and 'failedSince' fields
Actions #1

Updated by laforge 2 months ago

  • Description updated (diff)
Actions #2

Updated by lynxis about 1 month ago

  • Status changed from New to In Progress
Actions #3

Updated by lynxis about 1 month ago

  • % Done changed from 0 to 10

I'll implement a irc bot and command line utility to check for ttcn3 failures.

  • use jenkins api to retrieve the junit xml file
  • parse junit xml files into junit object
  • convert junit object into data structure to get a sliding window over 10 days/jobs for
    • "good" (first job succeed after fails, but job wasn't unstable)
    • "fail" (first job fails after good, but job wasn't unstable)
    • "unstable" (many fails & succeeds alternating (2x in a row is fine)
  • generate a summary for every jobs
  • generate a summary for all jobs
  • connect a irc bot to the output
  • check jenkins for updates every hour
Actions #4

Updated by laforge about 1 month ago

If possible, make the ir bot only one consumer of sll that information. As I mentioned, we also want some kind of overview web page at some point. You don't have to work on that if you dont want to, but it would be good to design the software in a way that multiplr consumers for the information exist so it can be extended later.

As a bonus, also considering obs.osmocom.org failures would be great

Actions #5

Updated by neels about 1 month ago

Maybe this bit of code can help:

https://gitea.osmocom.org/ttcn3/osmo-ttcn3-hacks/src/branch/master/compare-results.py

the proper way would be to do actual XML parsing though, this script cheats
around that with regexes, depending on the actual line feeds that jenkins
typically produces.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)