resource objects leaked or destroyed too late
It was noticed during development of nanoBTS support that some objects are being kept alive after suite.objects_cleanup() is called.
It was noticed because nanobts keeps a PowerSupplySispm which in turn used to keep (changed now) a usbdevice object from libusb which in turn kept the usb device "reclaimed" during the lifespan of the object. When the 2nd test was run, even on a differnet suite, the new PowerSupplySispm failed due to libusb returned "Resource Busy", because the previous nanoBTS+PowerSupplySispm were still alive.
Adding the following code to suite.objects_cleanup() helped showing that devices are being kept alive at that point, because rfcount is > 1 (it should be 1 since we should only have 1 reference to it in "obj" var after cleanup() is called. Actually this is not entirely true since other resource objects could be referencing them, so we should first call cleanup() on all of them, then keep them in another array, then check the refcount of each of them to make sure they are going to be cleared.
My bet is that they are still kept referenced by the test .py file which used them, since the test is run in the file directly and not in a function with a scope. I think we should either find a way to "unload" that file, and if not easy to do, then call a function inside that file instead of running the file directly. The test could still leak objects by having global objects in the file, but we could check it with the refcount once the test/suite is finished.
diff --git a/src/osmo_gsm_tester/log.py b/src/osmo_gsm_tester/log.py index 7c4ae44..ca56781 100644 --- a/src/osmo_gsm_tester/log.py +++ b/src/osmo_gsm_tester/log.py @@ -397,6 +397,9 @@ class Origin: if find_parent: self._set_parent(Origin.find_on_stack(except_obj=self)) + def __del__(self): + self.log('PESPIN: OBJ DESTROYED!') + def _set_parent(self, parent): # make sure to avoid loops p = parent --- a/src/osmo_gsm_tester/suite.py +++ b/src/osmo_gsm_tester/suite.py @@ -20,6 +20,7 @@ import os import sys import time +import gc import pprint from . import config, log, template, util, resource, schema, event_loop, test from . import osmo_nitb, osmo_hlr, osmo_mgcpgw, osmo_mgw, osmo_msc, osmo_bsc, osmo_stp, osmo_ggsn, osmo_sgsn, modem, esme @@ -100,6 +101,9 @@ class SuiteRun(log.Origin): obj.cleanup() except Exception: log.log_exn() + refc = sys.getrefcount(obj) + if refc > 1: + self.err('refc for %r: %d > 1, probably going to be leaked. referrers: %r' % (repr(obj), refc, gc.get_referrers(obj))) def mark_start(self): self.start_timestamp = time.time()