I am wondering if anyone has insights into the massive memory consumption that occurs with Script Debugger 8 (SD8) under Catalina.
(There is a previous thread in April 2020 with a similar vein)
The premise is this:
I have about 100,000 HTML files with structured data sitting in DevonTHINK. I am inventarising these for a subset of this structured data in an Excel table. This works as follows: I take the selection of HTML files and iterate for each to read the source and then extract the structured data, setting the appropriate cells in Excel with this data. To extract the data from the HTML files, I use AppleScript’s OFFSET function to find the enclosing text brackets, if you like, so that I can determine the appropriate column heading (-> column number), alternate/sub heading and actual data item.
(I use OFFSET to find the start of the text, then cut off all the text before it, and work on the raminder)
Per ~25K HTML file this tallies up about 100-250 events - most of which are finding the OFFSET for the text brackets as well as stripping out links and such. Very few events are involved in reading the file from DevonTHINK and putting the data in Excel.
I tried this in SD8 and Catalina and this caused something to barf; SD8 was not getting any more response after about 25% of all documents processed (‘An AppleEvent timed out’). Not that DevonTHINK hung up; something just stopped working. I did notice SD8 was using a lot of memory by this point (ie several GBs).
So instead, I batch-process a selection of files from DevonTHINK. This works OK… but for some reason SD8’s memory usage BALLOONS. I suppose this is fine when manipulating x0,000 files… but… the memory is never released. I run the script successively on several batches, and at one point SD8 has consumed all system memory and more… totalling more than 32 GB. This of course causes the system to swap like a mad hatter slowing down everything. It isn’t until I quit SD8 that things return to normal. So, now I’m resigned to do several batches and then quit and restart SD8.
(the 100,000 HTML files total about 2.5GB)
I had noticed SD8’s memory consumption increase over the period I use it (without quitting in between) before but this is excessive and I am wondering is there is a memory leak somehow?
The problem does not occur when using stock Script Editor under Catalina. Script Editor does run much slower.
The bizarre thing is, it also does not occur when using SD8 under Mojave. (Same database, same Excel file). Indeed – under Mojave the script runs at least twice as fast. (and this on a 2014 iMac with 8GB memory vs. a late-2019 MacBook Pro with 32 GB)
The only other thing that is different is the version of Excel: 2019 on the MacBook Pro/Catalina, 2016 on the iMac/Mojave. But gauging by the rate at which data appears in the Excel file, I don’t see a difference there. (And would still not explain why stock Script Editor has no problems)
(Script Debugger 8 setup & prefs are the same between machines)