Suggestions for Splitting a Large Script Library to Avoid errOSAInternalTableOverflow – "Umbrella" Script Libraries?

AppleScript
,
#1

Hello – I’m hoping for some suggestions for the best way to split up a large script library, now that I’m bumping into the ceiling of allowable AppleScript size (see Table Overflow Errors | Late Night Software).

I have a personal script library (called General Library) that is imported into all of my AppleScripts, and mainly deals with extensions to the core AppleScript types & file management. It’s now over 9500 LoC and has reached the point that even adding comment lines gives an errOSAInternalTableOverflow error when saving.

I do have ~10 other script libraries that I’ve written for specific purposes (e.g. ASOC Library, Citrix Library, Media Library, System Library, etc.), so I’m not averse to splitting it up. However, I’ve never found a good way to keep the calling process straightforward (e.g. does makeListOfStrings() live in the hypothetical String Library or List Library? Right now it’s just General Library.)

My question is this – Is there a good way to create an “Umbrella Library”, so that the calling scripts can still refer to General Library and the library does the work of calling into the smaller libraries?

I have thought of a couple of options, but nothing seems like a great solution:

  1. Form a chain of parent relationships, by setting the parent of General Libary to String Library, the parent of String Libary to List Library, etc.

    This works (callers still just call “General Library”, and doesn’t require a re-write of all of my calling scripts), but seems like a really bad idea. It also breaks autocomplete.

  2. Create a new “General Library” whose only job is to call into the other libraries (e.g. General Library's makeListOfStrings() would simply call String Libraries's makeListOfStrings()). This would also add complexity, although it could probably be generated programmatically from the other libraries. And while it would reduce the number of objects leading to the errOSAInternalTableOverflow, it’s also just kicking that same can further down the road.

Unfortunately, I haven’t had any brilliant ideas that wouldn’t involve a complete re-writing of essentially all of my scripts. Obviously, this is something I’m not eager to do, and only want to do once (or not at all).

Any ideas would be very welcomed!

2 Likes
#2

If it’s splitting a string into text items, it belongs in the String Library. If it’s coercing each item in a list to a string, it belongs in the List Library.

If you’re importing the General Library into all of your scripts, I how much of your 8,000+ lines of code do you estimate is being actively utilised by any one script picked at random ? I imagine that it probably needs splitting up regardless of the compiler’s limitations or any other changes you end up making.

I think it then becomes logistically (and mentally) more digestible to load each smaller library into your scripts as deemed necessary, which you can assign to a property using load script, e.g.:

property string : load script "/path/to/string library.scpt"

It’s not going to feel as semantically elegant to write my string's listOfStrings(…) rather than simply listOfStrings(…), but it affords greater organisation, manageability and efficiency (seriously, 8,000 lines in any programming language is hardcore, which is the opposite of any scripting languages’s remit).

#3

Thanks CJK!

I’d say the majority of my scripts don’t cover too much of the library, but enough that if I split up the library, I’d probably still have to import all of the sub-libraries into each script.

For example, I use the following quite frequently:
slice, for python-like string & list slicing
wrap, for wrapping handlers to allow them to be used as first-class objects
dictWithKeysAndValues, a custom dictionary type
getFile, to coerce any of the 7 AppleScript file types to «class furl»
loadVariableFromFile & storeVariableInFile, for reliable object persistence
logToWindow, to log to a tty
getKeyboardModifierKeys, to allow for modifier key reporting

Full List

— MATH HANDLERS —
abs(n)
sign(n)
zeroSign(n)
copySign(n, i)
roundTo(n, decimal_places)
trunc(n)
reverseTrunc(n)
floor(n)
ceil(n)
roundIfRound(n)
min(L)
max(L)
sum(L)
average(L)
median(L)
mode(L)
sqrt(n)
logB(n, b)
log2(n)
log10(n)
ln(n)
degrees(radians)
radians(degrees)
sin(n)
cos(n)
tan(n)
asin(n)
acos(n)
atan(n)
gcd(n1, n2)
hypot(L)
convertIntegerToBinary(n)
convertBinaryToInteger(s)
bitwiseNOT(s1, s2)
bitwiseAND(s1, s2)
bitwiseOR(s1, s2)
bitwiseXOR(s1, s2)
bitwiseShift(s, n)
_convertUnits(n, from_units, to_units)

— STRING HANDLERS —
sliceString(s, slice_start, slice_end)
countSubstrings(search_string, s)
getOffset(search_string, s)
getOffsetWithEndpoints(search_string, s, search_start, search_end)
getLastOffset(search_string, s)
getLastOffsetWithEndpoints(search_string, s, search_start, search_end)
getAllOffsets(search_string, s)
_getLowestOffset(L, s)
_getHighestOffset(L, s)
reverseIndex(i, C_length)
reverseString(s)
containsOneOf(s, L)
containsOneOfAt(s, L)
chr(n)
ord(char)
getUnicodeName(chars)
convertNumberToLetter(n)
convertNumberToLetters(n, C)
convertLetterToNumber(s)
convertLettersToNumber(s, C)
_convertArabicToRoman(n)
_convertRomanToArabic(s)
_convertTimeToNumber(s)
_convertNumberToTime(n)
extractInteger(s)
extractLastInteger(s)
extractAllIntegers(s)
splitStringAtIntegerBoundaries(s)
isUppercase(s)
isLowercase(s)
isLetter(s)
isInteger(s)
isNumber(s)
isNaturallyLessThan(s1, s2)
uppercase(s)
lowercase(s)
_titleCase(s)
_sentenceCase(s)
properEnglishTitleCase(s)
strictEnglishTitleCase(s)
flexibleEnglishTitleCase(s)
replaceText(s, search_string, replacement_string)
expandText(s, string_expansion_pairs)
translateCharacters(s, old_characters, new_characters)
parseString(s, parse_from, parse_to)
trimString(s, trim_from, trim_to, mode)
truncateString(s, search_string, truncate_end)
truncateStringAfterSentenceBeforeIndex(s, maximum_length)
stripString(s, strip_characters)
stripWhitespace(s)
padString(s, char, n)
padStringWithZeroes(integer_s, desired_length)
centreString(s, char, n)
condenseString(s, n)
shortenString(s, n)
removeRepeatingCharacters(s, search_characters, max_repeats)
repeatCharacter(char, n)
makeStringOfDigits(s_length)
makeStringOfLetters(s_length)
_incrementNumber(s, search_from_back)
_incrementVersionString(version_string, position_to_increment)
splitString(s, string_delimiters)
joinList(L, string_delimiter)
_listAsString(L)
_joinItemList(L, string_delimiter)

— LIST HANDLERS —
sliceList(L, slice_start, slice_end)
countOccurrences(an_item, L)
getIndex(an_item, L)
getIndexWithEndpoints(an_item, L, starting_index, ending_index)
getLastIndex(an_item, L)
getLastIndexWithEndpoints(an_item, L, starting_index, ending_index)
getAllIndices(an_item, L)
asList(O)
mapList(L, f)
getMappedList(L, f)
getFilteredList(L, f)
getReducedList(L, f)
reverseList(L)
getReversedList(L)
zipLists(LL)
zipPairedLists(L1, L2)
unzipPairs(L)
_splitList(L, sublist_count)
_joinLists(LL)
deinterlaceList(L, sublist_count)
_interlaceLists(LL)
hasListIntersection(L1, L2)
hasSymmetricListDifference(L1, L2)
getListUnion(L1, L2)
getListIntersection(L1, L2)
getListDifference(L1, L2)
getSymmetricListDifference(L1, L2)
itemsAreUnique(L)
getUniqueItems(L)
getRepeatedItems(L)
shuffleList(L)
getShuffledList(L)
makeList(list_length, value)
makeListOfNumbers(list_length)
makeListOfLetters(list_length)
makeListByDeletingItemAtIndex(i, L)
_convertListToString(L)
sortList(L)
reverseSortList(L)
getSortedList(L)
getReverseSortedList(L)
sortListNaturally(L)
getNaturallySortedList(L)
keyIsNaturallyLessThanKey(L1, L2)
_sortListBySublistItem(L, sublist_index)
_getSortedListBySublistItem(L, sublist_index)
sortListUsingHandler(L, sort_handler)
getSortedListUsingHandler(L, sort_handler)
sortListUsingKey(L, generate_key_handler)
getSortedListUsingKey(L, generate_key_handler)
_sortListByProperty(L, a_property)
getSortedListByProperty(L, a_property)
_sortListByCondition(L, sort_condition)
_getSortedListByCondition(L, sort_condition)
sortLists(LL)
getSortedLists(LL)

— DICTIONARY HANDLERS —
emptyDict()
dictWithKeysAndValues(keys, values)
dictWithKeysAndDefaultValue(keys, default_value)
dictWithPairs(kv_pairs)
dictWithInterlacedPairs(interlaced_pairs)

— RECORD HANDLERS —
_getRecordValue(k, R)
_setRecordValue(kv, R)
_addRecordValue(kv, R)
_changeRecordValue(kv, R)
_getRecordKeys(R)
_getRecordValues(R)
_convertRecordToLists(R)
_convertListsToRecord(LL)

— FILE HANDLERS —
getPath(file_specifier)
getHFSPath(file_specifier)
getFile(file_specifier)
getAlias(file_specifier)
getFileReference(file_specifier)
getFinderItem(file_specifier)
getDiskItem(file_specifier)
getPathTo(O)
getNameOf(O)
getFileNameOf(O)
getParentFolderOf(O)
fileExists(file_specifier)
filesExist(file_specifier_list)
symbolicLinkExists(file_path)
isFile(file_specifier)
isFolder(file_specifier)
isPackage(file_specifier)
expandTilde(file_path)
normalizePath(file_path)
normalizePathAndExpandTilde(file_path)
getPathComponents(file_path)
joinPathComponents(path_components)
appendPathComponent(parent_path, new_path_component)
splitFilePath(file_path)
getParentPath(file_path)
getFileName(file_path)
splitFileName(file_name_ext)
splitFileParts(file_path)
joinFileParts(parent_path, file_name, file_ext)
getFileStem(file_path)
removeTrailingPathSeparator(file_path)
appendTrailingPathSeparator(file_path)
removeFileExtension(file_name_ext)
getFileExtension(file_name_ext)
addFileExtension(file_name, file_ext)
changeFileName(file_name_ext, new_name)
appendToFileName(file_name_ext, s)
asDateString(D)
daysInMonth(D)
daysInMonthForYear(month_number, year_number)
getDateString()
getTimeString()
appendTimeString(file_name_ext)
changeFileExtension(file_name_ext, new_ext)
getSafeName(file_name_ext)
getCleanName(file_name_ext)
getOriginalName(file_name_ext)
getOriginalPath(file_path)
getNextAvailableStem(parent_path, file_name, file_ext)
getNextAvailableName(parent_path, file_name_ext)
getNextAvailablePath(file_path)
promptToOverwriteFile(file_path)

— FILE MANAGEMENT HANDLERS —
renameFile(file_path, new_name_ext)
duplicateFile(file_path)
moveFile(file_path, new_file_path)
forceMoveFile(file_path, new_file_path)
moveFileToFolder(file_path, new_parent_path)
copyFile(file_path, new_file_path)
forceCopyFile(file_path, new_file_path)
copyFileToFolder(file_path, new_parent_path)
updateFile(file_path, source_file_path)
syncFile(file_path_1, file_path_2)
copyFolderHierarchy(folder_path, new_folder_path)
_mergeFolders(folder_1_path, folder_2_path)
archiveFile(file_path)
unarchiveFile(file_path)
_backUpFile(file_path)
deleteFile(file_path)
removeFile(file_path)
removeEmptyFolder(file_path)
removeFolder(file_path)
removeFolderContents(file_path)
_moveFileWithAdministratorPrivileges(a, b, user_password)
makeFile(file_path)
makeFolder(file_path)
makeFolderWithIntermediates(file_path)
makeTemporaryFolder()
_makeFolderTree(file_path, name_tree)
_getFolderTree(file_path)
renameFileSafely(file_path, new_name)
moveFileSafely(file_path, new_path)
moveFileToFolderSafely(file_path, new_parent_path, new_stem)
moveFolderContentsToFolderSafely(source_folder_path, destination_folder_path)
copyFileSafely(file_path, new_path)
copyFileToFolderSafely(file_path, new_parent_path, new_stem)
copyFolderContentsToFolderSafely(source_folder_path, destination_folder_path)
makeFileSafely(file_path)
makeFolderSafely(file_path)
makeHardLink(file_path, link_path)
countHardLinks(file_path)
isHardLinkOf(file_path_1, file_path_2)
_getHardLinks(file_path)
makeSymbolicLink(target_path, link_path)
forceMakeSymbolicLink(target_path, link_path)
isSymbolicLink(file_path)
checkSymbolicLink(link_path)
checkSymbolicLinks(file_path)
readSymbolicLink(link_path)
resolveSymbolicLink(link_path)
resolveSymbolicLinks(file_path)
updateSymbolicLink(link_path, new_target_path)
recreateSymbolicLink(link_path, hide_extension)
makeFinderAlias(file_path, alias_path)
isFinderAlias(file_path)
checkFinderAlias(alias_path)
resolveFinderAlias(alias_path)
_updateFinderAlias(alias_path, new_target_path)
isLink(file_path)
checkLink(link_path)
checkLinks(file_path)
resolveLink(link_path)
resolveLinks(file_path)
_updateLink()
getVolumeDeviceID(volume_path)
getPathWithInode(inode_string)
_isDotFile(file_path)
fileIsHidden(file_path)
hideFile(file_path)
unhideFile(file_path)
fileExtensionIsHidden(file_path)
hideFileExtension(file_path)
showFileExtension(file_path)
fileIsLocked(file_path)
lockFile(file_path)
unlockFile(file_path)
fileIsExecutable(file_path)
makeFileExecutable(file_path)
makeFileUnexecutable(file_path)
executeFile(file_path)
fileIsNewerThan(file_path_1, file_path_2)
fileIsOlderThan(file_path_1, file_path_2)
folderIsEmpty(folder_path)
folderIsEmptyIncludingInvisibles(folder_path)
listFolder(folder_path)
listFolderIncludingInvisibles(folder_path)
listFolderPaths(folder_path)
listFolderPathsIncludingInvisibles(folder_path)
getFirstPathInFolderMatchingName(parent_folder_path, name_match)
_getSubfolderByNameMatch(parent_folder_path, name_match)
findFiles(folder_path)
findFilesIncludingInvisibles(folder_path)
findFilesWithoutRecursion(folder_path)
findFilesWithoutRecursionIncludingInvisibles(folder_path)
findFinderAliases(folder_path)
findSymbolicLinks(folder_path)
readFile(file_specifier)
readFileIntoList(file_specifier)
writeToFile(file_specifier, text_to_write)
appendToFile(file_specifier, text_to_append)
appendToExistingFile(file_specifier, text_to_append)
openFile(file_path)
openFileInApplication(file_path, application_name)
previewFile(file_path)
revealFile(file_path)
revealFiles(file_list)
revealFileInBackground(file_path)
revealFilesInBackground(file_list)
getSelectedPath()
getSelectedPaths()
getInsertionPath()
getFrontFinderWindowTarget()
setInsertionPath(target_path)
selectFile(file_path)
selectFiles(file_list)
reselectFiles()
makeNewFinderWindow()
_isRecognizedFileExtension(file_ext)
_hasRecognizedFileExtension(file_name_ext)

— OTHER HANDLERS —
nothing()
isNothing(object_reference)
attempt(reference_reference)
emptyScript()
isApplication(O)
isReference(some_value)
dereference(some_value)
length__(O)
isEqualTo(value_1, value_2)
isEqualReferenceTo(value_1, value_2)
_isIdenticalTo(value_1, value_2)
referenceExists(some_reference)
unwrapReference(some_reference, default_value)
_init(script_object)
assert(condition)
assertValue(condition, return_value)
tif(condition, value_if_true, value_if_false)
setTIDs(new_TIDs)
resetTIDs()
getCurrentDate()
getBestTypeOf(O)
_getHandlerName(a_handler)
callHandlerWithArguments(f, argument_list)
wrap(O)
_testHandler(handler_name, list_of_arguments_and_expected_result)
eval(script_text)
evalWithParameters(script_text, parameters)
compileScriptText(script_text, output_path)
compileScriptFile(file_path)
recompileScriptFile(file_path)
decompileScriptFile(file_path)
compileScriptFileTo(file_path, output_path)
recompileScriptFileTo(file_path, output_path)
decompileScriptFileTo(file_path, output_path)
scriptFileIsDebugFormat(file_path)
loadVariableFromFile(file_specifier, variable_class)
loadScriptFromFile(file_specifier)
storeVariableInFile(file_specifier, variable_to_store)
loadListFromFile(file_specifier)
loadRecordFromFile(file_specifier)
escapeInput()
_escapeString(s)
escapeStringForShell(s)
_escapeStringForPython(s)
promptForPassword()
confirmListChanges(old_strings, new_strings, dialog_title, dialog_description)
doShellScript(script_text)
doShellScriptIgnoringErrors(script_text)
doShellScriptReturningStandardError(script_text)
doShellScriptCombiningStandardError(script_text)
doShellScriptInBackground(script_text)
doShellScriptInTerminal(script_text)
doShellScriptInBackgroundTerminal(script_text)
doShellScriptInTerminalWithPrompt(script_text)
logToWindow(s, output_tty)
quotedForm(C)
runScript(script_path)
runScriptInBackground(script_path)
_runScriptWithParameters(script_path, argument_list)
runScriptInBackgroundWithParameters(script_path, argument_list)
listBackgroundScriptInvocations()
runWorkflow(workflow_path)
runWorkflowInBackground(workflow_path)
runWorkflowWithInput(workflow_path, input)
runWorkflowInBackgroundWithInput(workflow_path, input)
getClipboard()
getClipboardPaths()
setClipboard(s)
copyToClipboard(a_string)
py(expression_text)
python(script_text)
python2(script_text)
python3(script_text)
_getWindowSize(a_window)
_setWindowSize(a_window, window_width, window_height)
getScreenSize()
playTaskCompleteChime()
playErrorChime()
playInteractionRequiredChime()
_logToFile(text_to_log)
_displayText(s)
getFrontmostApplication()
getFrontmostApplicationName()
getFrontmostApplicationPath()
getFrontmostApplicationProcess()
hideFrontmostApplication()
getApplicationProcess(application_identifier)
getProcessID(application_identifier)
processExists(pid)
getModifierKeys()
modifierKeysArePressed(modifier_keys)
waitForModifierKeyRelease(modifier_keys)
runInScriptMonitor(script_path)
runScriptInMenu(script_path)
registerCaller(script_object)
caller()
getSystemVersion()
truncateTrailingVersionZeroes(s)
versionIsEqualTo(s1, s2)
versionIsGreaterThan(s1, s2)
versionIsGreaterThanOrEqualTo(s1, s2)
getHostname()
getComputerName()
isInternetReachable()
getWiFiNetworkName()
getWiFiBSSID()
sleepDisplays()
shutdownWithConfirmation()
isSIPDisabled()
getDefaultsPaths(bundle_id)

I used to do something similar to this before use was a thing, but this causes the full library to be embedded as an object in each calling script (and requires recompilation of the calling script in order for the library to be updated).

I do have my “General Library” organized into logical sections, so the organization itself isn’t really the problem. Quite a few of the handlers call into each other, so I’m concerned how reliable that will be with AppleScript’s library loading mechanism.

If I do split up the library that way, it will require me to re-write all my calling scripts (e.g. change "General Library"'s someFunction() to "String Library"'s someFunction()) and a good chunk of the library itself (e.g. if someFunction() calls an unqualified someOtherFunction() that both used to be in the same library).

#4

I was going to say don’t use the load script method when use script is now available, but you got there before me.

I don’t have a solution for the pain of re-writing all/many of your scripts after making the changes, but I do have a lot of sympathy.

Maybe you’re already aware, but one thing that might help is that libraries can use other script libraries. So the obvious (though not necessarily best or cleverest) solution is to break your general library into multiple smaller libraries. The way you categorise which handlers go in which library might end up being somewhat arbitrary, but that’s an unfortunate consequence of AppleScript’s script object limit being somewhat arbitrary. I’m sure you’ll eventually learn/remember which goes in which. In any event, if a handler in one library calls into a handler from another library, the first library just needs to use the second library (and you need to update the call in the first library to explicitly reference the second library). This will be the painful point of rewriting.

The idea of keeping your existing general library as a wrapper for your other libraries sounds good to me, at least as a transitional measure for existing scripts. As you create new library handlers and update or write new scripts, you should use the underlying libraries. But, in the meantime for compatibility purposes, after you copy all the general library’s handlers into new libraries, you can go through and gut each handler in the general library and replace its body with a call to the handler in the new library. Again, it will be a lot of boring typing—if you can automate it even better—but I think you should only need to do it once as from then on you should just use the new smaller underlying libraries.

One other tip: if you have an applet or something that uses the general library, you might want to also use all libraries used by the general library even if that applet doesn’t use those libraries directly. That way, when you export a run only version for deployment, SD will automatically bundle all the needed script libraries with your app.

#5

Thanks, AppleScripter. I think that’s what I’ll end up doing – gradually moving things into Script Libraries (as a 2.0 API), and updating the “General Library” so that they call into the new libraries. Eventually I’ll get rid of the “General Library” once enough of my scripts have been updated.

I’m glad someone understands my pain. :smiley: