The TL;DR is that the architecture of the AppleScript interpreter goes back to the early 1990’s and was designed to make use of only a small amount of memory.
It’s a delightful glue language for working on a miniature scale, but begins to error, intermittently or ineluctably, when:
Code begins to grow long (c. 1000 loc +)
Data structures grow large (as you have seen)
Recursion goes deep (as above).
If you need to work with Apple Events, and with the Foundation, Appkit etc libraries, at at a slightly larger scale, then you may need to experiment with the JavaScript interface to osascript. (You can begin to the explore JS Automation library from Script Editor by switching the language tab at top left).
Thanks again. In fact, I am using ASObjC more and more for arrays operations by coding examples from you, your book, and others in MacScripter. It seems the only limitation for ASObjC is that it can’t operate on list (or list of list) that contains reference objects of an application.
To be fair, your script only starts counting a level or two down and i and (presumably) the error return address are stored on the stack each time. If the script’s rewritten this way …
property i : 0
on recursionDepth()
set i to i + 1
recursionDepth()
end recursionDepth
on run
set i to 0
try
recursionDepth()
on error
"Recursion limit encountered at " & i
end try
end run
… the result is "Recursion limit encountered at 734". Admittedly, though, that’s still a long way short of 31721. On the other hand, 734 return addresses isn’t very much actual memory at all and I wonder if the result has any real meaning.
Thanks!
Can u kindly point me to where I can read/learn about such info?
Or it’s could be as simple as setting the lists back to {} after they are not used anymore or something like that? I never have issue (yet) in running the script, but stack overflow happened while I’m in the debug mode of SD occasionally.
In my case, I don’t use recursions much and only for less that 4 levels (for folders hierarchy) if they are used. I do have to manipulate list of lists that are usually less than 1000 rows and with less than 20 items in each row and of which some of the items are also a list, or records of an app, or a string with several hundreds to a few thousands words.
Not looking for a specific answer but a pointer to the right direction…
Chewing over results is the business of ideology – more revealing of the chewer than of what lies behind the results
Experimentation is in the business of asking questions, and these questions are comparative:
What happens if we apply the same method to different languages and contexts ?
You feel that adjusting the method (but only applying it to one language) asks nature a useful question ?
(To be honest, the variation, even for AS in isolation, is still not great, and still reveals comparatively little space for a language to breathe in.)
(comparative being the relevant category)
Even in Script Editor, we do have much more resource to play with (code length and data size as well as recursion depth) if we reach for JavaScript.
You are doing very interesting work with the automation of DEVONthink, but your projects are now reaching the scale at which AppleScript becomes less reliable and less useful. Perhaps less relevant.
I would certainly recommend continued use of AppleScript for smaller things, not least because of the excellent Script Debugger product, but you probably would need to reach for a different tool if you wanted to pursue those larger and more ambitious DEVONthink projects.
(We would be misleading you if we gave the impression that scrupulously avoiding recursion would suffice to make AppleScript significantly more scaleable.
The shallow recursion stack is a useful and measurable indication of the limited memory available to the AppleScript interpreter, but it is not the only constraint).
PS you are far from alone in hitting these limits, there are various discussions on Stack Overflow (if Nigel will forgive the hint of something recursive here), for example:
I first became conscious if it while writing a large script for search and querying the OmniFocus database.
It impinged again with a DSL for quicker reorganising and formatting of nested diagrams in OmniGraffle.
JavaScript I first tried when TaskPaper moved the bulk of its scripting into a JS Context. Since then Omni group has done the same with OmniGraffle, OmniOutliner, and most recently OmniFocus, which, inter alia, allows them to provide rich scriptability for these applications on iOS as well as macOS.
JS rewards a bit of experimentation I think – a useful complement to AppleScript.
Got it. I’m just doing stuff for fun. Will learn new things eventually but probably will want to thoroughly understand about AppleScript and ASObjC within the environment first. More importantly, I just need to learn to code efficiently first!
Fooling around with my script in post #25, I get results like the ones below. The first column shows the depths of recursion reported by the script and the second the number of local variables set in the recursionDepth() handler when those depths were achieved. The bracketed figures at the end are the reduction in recursion depth each time.
734 No locals set
672 1 local of any class or size (-62)
621 2 locals set ditto or 1 parameter passed ditto (-51)
576 3 locals set (-45)
538 4 locals set or 2 parameters passed (-38)
504 5 locals set (-34)
474 6 locals set or 3 parameters passed (-30)
448 7 locals set (-26)
424 8 locals set or 4 parameters passed (-24)
These results tend to support my understanding (dating from when I used to dabble in 6502 and 68000 Assembler 25 years and more ago) that the stack is only used to store return addresses and pointers to local variable values. The values themselves aren’t stored on the stack and so their classes and sizes are immaterial. The reason why one parameter variable is equivalent to two set in the handler is that the parameter pointers are pushed onto the stack as part of each handler call and are stacked again for preservation when the handler’s called from within itself with fresh parameter pointers. The gradually reducing difference between the differences is because the recursion depths are the number of times that the whole number of bytes stacked at each call goes into the total stack size.
The recursion depth results above are the straight-down maxima achieved under the conditions shown. They’re actually pretty generous for normal use. A non-recursive script should never need to descend that far. The best use for recursion is with multiply-branching processes, to simplify what would otherwise be complex tasks. Each recursion branch is completed, releasing the stack space it uses, before the next branch from that level is entered. So although the amount of recursion going on can be huge, the actual depth reached and the amount of stack in use at any one time usually isn’t. A handler with only one recursion point would be better handled as a repeat even if the recursion depth was only expected to be shallow.
Since you’re only getting a stack error in SD’s debug mode, the problem has to be more with what SD has to do to the script to impose that mode than with the AppleScript stack itself.
With regard to setting lists back to {} when they’re no longer needed, this only applies to a running script’s run-handler variables, top-level properties, and globals, whose final values are saved back to the script’s file when it finishes. This can cause bloat and even errors when the total amount of data involved is large. I myself try to keep variables local where possible so that their values are simply forgotten when the script finishes.
Thank you very much for the detail explanation. When I was in uni almost 40 years ago, Pascal was my thing. I never gets comfortable with assembly and C is a total alien to me (never got the syntax into my sense). Then I was spoiled by VBA and its editing environment, all methods and properties are listed after the “.” of an object.
While I am certain that @ComplexPoint is giving me valuable advise from his experience that I really should seriously learn JavaScript for expanding the scope of programmability, I share the same conclusion with you that my AppleScript program has yet to hit the wall (given that the stack overflow only occurs in debug mode) coz many fine-tunings can be done. For example, transforming recursive into repeat loops and let ASObjC handles all manipulations of deep and wide arrays. I guess the real test is when the LOL is gradually increased to several thousands of items instead of nearly a thousand - but I suspect that I will be reaching a point that I can’t debug the full program and will have to debug by part in the SD’s debug mode very soon.
Regarding properties, I am using quite a lot of properties to define lists in the script object within the handlers. Will those properties within the handlers reset to null after the handlers finish their jobs?
Yes. Sort of. The script objects themselves are assigned to local variables (unless the variables have been otherwise specified, of course), so they and their properties are discarded as soon as the handlers exit.
Mileage varies (particularly depending on whether people are coding for themselves or for clients), but productivity is clearly a central value, especially if we are coding in our spare time, or hoping to believe our own propaganda – that our scripts “save time” : - )
My personal ranking would be:
productivity
reliability
performance
with the last trailling in pretty late. Much more time is wasted by scripts that error, or take too long to write, than by scripts that don’t run fast.
That makes me feel just a little uneasy when others are told that they should learn how to script more efficiently, or devote effort to working around the failings of some particular tool.
A casual scripter should never need to open the hood and tinker with the engine, or have to soup it up. Better to use a different vehicle.
Some people identify strongly very with their tools (even to the point of T shirts, mugs and bumper stickers), but I think that’s bound to be a mistake.
Better to have a tool-box from which you can choose the thing that works with least effort, and most reliability, for the job.
Thanks again for the advise!
I sure will put learning a more effective language in my agenda.
Just that you know I have a very different perspective. I program for fun and to kill time. Image those people who spend years to restore an old car, or spending one year to build a sophisticated car or ship models - I’m one of them. It’s not about the performance of the car but the understanding of the car down to the nuts and bolts and the fine-tuning of it to its optimum…
Cheers
In that case, I would recommend experimenting not just with JavaScript (Automation library supplied by Apple), but also with, at the very least:
Python (excellent libraries for all kinds of things)
Haskell (fascinating insight into problem solving and the underlying math of computation, fast compiled code)
Rust (excellent introductory documentation, at least one osascript ‘crate’, much more productive (and reliable) than C, and, if you ever genuinely need it, extremely fast compiled code.
On the other hand, 734 return addresses isn’t very much actual memory at all and I wonder if the result has any real meaning. 
