Suggestions for Splitting a Large Script Library to Avoid errOSAInternalTableOverflow – "Umbrella" Script Libraries?

Hello – I’m hoping for some suggestions for the best way to split up a large script library, now that I’m bumping into the ceiling of allowable AppleScript size (see Table Overflow Errors | Late Night Software).

I have a personal script library (called General Library) that is imported into all of my AppleScripts, and mainly deals with extensions to the core AppleScript types & file management. It’s now over 9500 LoC and has reached the point that even adding comment lines gives an errOSAInternalTableOverflow error when saving.

I do have ~10 other script libraries that I’ve written for specific purposes (e.g. ASOC Library, Citrix Library, Media Library, System Library, etc.), so I’m not averse to splitting it up. However, I’ve never found a good way to keep the calling process straightforward (e.g. does makeListOfStrings() live in the hypothetical String Library or List Library? Right now it’s just General Library.)

My question is this – Is there a good way to create an “Umbrella Library”, so that the calling scripts can still refer to General Library and the library does the work of calling into the smaller libraries?

I have thought of a couple of options, but nothing seems like a great solution:

  1. Form a chain of parent relationships, by setting the parent of General Libary to String Library, the parent of String Libary to List Library, etc.

    This works (callers still just call “General Library”, and doesn’t require a re-write of all of my calling scripts), but seems like a really bad idea. It also breaks autocomplete.

  2. Create a new “General Library” whose only job is to call into the other libraries (e.g. General Library's makeListOfStrings() would simply call String Libraries's makeListOfStrings()). This would also add complexity, although it could probably be generated programmatically from the other libraries. And while it would reduce the number of objects leading to the errOSAInternalTableOverflow, it’s also just kicking that same can further down the road.

Unfortunately, I haven’t had any brilliant ideas that wouldn’t involve a complete re-writing of essentially all of my scripts. Obviously, this is something I’m not eager to do, and only want to do once (or not at all).

Any ideas would be very welcomed!

2 Likes

If it’s splitting a string into text items, it belongs in the String Library. If it’s coercing each item in a list to a string, it belongs in the List Library.

If you’re importing the General Library into all of your scripts, I how much of your 8,000+ lines of code do you estimate is being actively utilised by any one script picked at random ? I imagine that it probably needs splitting up regardless of the compiler’s limitations or any other changes you end up making.

I think it then becomes logistically (and mentally) more digestible to load each smaller library into your scripts as deemed necessary, which you can assign to a property using load script, e.g.:

property string : load script "/path/to/string library.scpt"

It’s not going to feel as semantically elegant to write my string's listOfStrings(…) rather than simply listOfStrings(…), but it affords greater organisation, manageability and efficiency (seriously, 8,000 lines in any programming language is hardcore, which is the opposite of any scripting languages’s remit).

Thanks CJK!

I’d say the majority of my scripts don’t cover too much of the library, but enough that if I split up the library, I’d probably still have to import all of the sub-libraries into each script.

For example, I use the following quite frequently:
slice, for python-like string & list slicing
wrap, for wrapping handlers to allow them to be used as first-class objects
dictWithKeysAndValues, a custom dictionary type
getFile, to coerce any of the 7 AppleScript file types to «class furl»
loadVariableFromFile & storeVariableInFile, for reliable object persistence
logToWindow, to log to a tty
getKeyboardModifierKeys, to allow for modifier key reporting

Full List

— MATH HANDLERS —
abs(n)
sign(n)
zeroSign(n)
copySign(n, i)
roundTo(n, decimal_places)
trunc(n)
reverseTrunc(n)
floor(n)
ceil(n)
roundIfRound(n)
min(L)
max(L)
sum(L)
average(L)
median(L)
mode(L)
sqrt(n)
logB(n, b)
log2(n)
log10(n)
ln(n)
degrees(radians)
radians(degrees)
sin(n)
cos(n)
tan(n)
asin(n)
acos(n)
atan(n)
gcd(n1, n2)
hypot(L)
convertIntegerToBinary(n)
convertBinaryToInteger(s)
bitwiseNOT(s1, s2)
bitwiseAND(s1, s2)
bitwiseOR(s1, s2)
bitwiseXOR(s1, s2)
bitwiseShift(s, n)
_convertUnits(n, from_units, to_units)

— STRING HANDLERS —
sliceString(s, slice_start, slice_end)
countSubstrings(search_string, s)
getOffset(search_string, s)
getOffsetWithEndpoints(search_string, s, search_start, search_end)
getLastOffset(search_string, s)
getLastOffsetWithEndpoints(search_string, s, search_start, search_end)
getAllOffsets(search_string, s)
_getLowestOffset(L, s)
_getHighestOffset(L, s)
reverseIndex(i, C_length)
reverseString(s)
containsOneOf(s, L)
containsOneOfAt(s, L)
chr(n)
ord(char)
getUnicodeName(chars)
convertNumberToLetter(n)
convertNumberToLetters(n, C)
convertLetterToNumber(s)
convertLettersToNumber(s, C)
_convertArabicToRoman(n)
_convertRomanToArabic(s)
_convertTimeToNumber(s)
_convertNumberToTime(n)
extractInteger(s)
extractLastInteger(s)
extractAllIntegers(s)
splitStringAtIntegerBoundaries(s)
isUppercase(s)
isLowercase(s)
isLetter(s)
isInteger(s)
isNumber(s)
isNaturallyLessThan(s1, s2)
uppercase(s)
lowercase(s)
_titleCase(s)
_sentenceCase(s)
properEnglishTitleCase(s)
strictEnglishTitleCase(s)
flexibleEnglishTitleCase(s)
replaceText(s, search_string, replacement_string)
expandText(s, string_expansion_pairs)
translateCharacters(s, old_characters, new_characters)
parseString(s, parse_from, parse_to)
trimString(s, trim_from, trim_to, mode)
truncateString(s, search_string, truncate_end)
truncateStringAfterSentenceBeforeIndex(s, maximum_length)
stripString(s, strip_characters)
stripWhitespace(s)
padString(s, char, n)
padStringWithZeroes(integer_s, desired_length)
centreString(s, char, n)
condenseString(s, n)
shortenString(s, n)
removeRepeatingCharacters(s, search_characters, max_repeats)
repeatCharacter(char, n)
makeStringOfDigits(s_length)
makeStringOfLetters(s_length)
_incrementNumber(s, search_from_back)
_incrementVersionString(version_string, position_to_increment)
splitString(s, string_delimiters)
joinList(L, string_delimiter)
_listAsString(L)
_joinItemList(L, string_delimiter)

— LIST HANDLERS —
sliceList(L, slice_start, slice_end)
countOccurrences(an_item, L)
getIndex(an_item, L)
getIndexWithEndpoints(an_item, L, starting_index, ending_index)
getLastIndex(an_item, L)
getLastIndexWithEndpoints(an_item, L, starting_index, ending_index)
getAllIndices(an_item, L)
asList(O)
mapList(L, f)
getMappedList(L, f)
getFilteredList(L, f)
getReducedList(L, f)
reverseList(L)
getReversedList(L)
zipLists(LL)
zipPairedLists(L1, L2)
unzipPairs(L)
_splitList(L, sublist_count)
_joinLists(LL)
deinterlaceList(L, sublist_count)
_interlaceLists(LL)
hasListIntersection(L1, L2)
hasSymmetricListDifference(L1, L2)
getListUnion(L1, L2)
getListIntersection(L1, L2)
getListDifference(L1, L2)
getSymmetricListDifference(L1, L2)
itemsAreUnique(L)
getUniqueItems(L)
getRepeatedItems(L)
shuffleList(L)
getShuffledList(L)
makeList(list_length, value)
makeListOfNumbers(list_length)
makeListOfLetters(list_length)
makeListByDeletingItemAtIndex(i, L)
_convertListToString(L)
sortList(L)
reverseSortList(L)
getSortedList(L)
getReverseSortedList(L)
sortListNaturally(L)
getNaturallySortedList(L)
keyIsNaturallyLessThanKey(L1, L2)
_sortListBySublistItem(L, sublist_index)
_getSortedListBySublistItem(L, sublist_index)
sortListUsingHandler(L, sort_handler)
getSortedListUsingHandler(L, sort_handler)
sortListUsingKey(L, generate_key_handler)
getSortedListUsingKey(L, generate_key_handler)
_sortListByProperty(L, a_property)
getSortedListByProperty(L, a_property)
_sortListByCondition(L, sort_condition)
_getSortedListByCondition(L, sort_condition)
sortLists(LL)
getSortedLists(LL)

— DICTIONARY HANDLERS —
emptyDict()
dictWithKeysAndValues(keys, values)
dictWithKeysAndDefaultValue(keys, default_value)
dictWithPairs(kv_pairs)
dictWithInterlacedPairs(interlaced_pairs)

— RECORD HANDLERS —
_getRecordValue(k, R)
_setRecordValue(kv, R)
_addRecordValue(kv, R)
_changeRecordValue(kv, R)
_getRecordKeys(R)
_getRecordValues(R)
_convertRecordToLists(R)
_convertListsToRecord(LL)

— FILE HANDLERS —
getPath(file_specifier)
getHFSPath(file_specifier)
getFile(file_specifier)
getAlias(file_specifier)
getFileReference(file_specifier)
getFinderItem(file_specifier)
getDiskItem(file_specifier)
getPathTo(O)
getNameOf(O)
getFileNameOf(O)
getParentFolderOf(O)
fileExists(file_specifier)
filesExist(file_specifier_list)
symbolicLinkExists(file_path)
isFile(file_specifier)
isFolder(file_specifier)
isPackage(file_specifier)
expandTilde(file_path)
normalizePath(file_path)
normalizePathAndExpandTilde(file_path)
getPathComponents(file_path)
joinPathComponents(path_components)
appendPathComponent(parent_path, new_path_component)
splitFilePath(file_path)
getParentPath(file_path)
getFileName(file_path)
splitFileName(file_name_ext)
splitFileParts(file_path)
joinFileParts(parent_path, file_name, file_ext)
getFileStem(file_path)
removeTrailingPathSeparator(file_path)
appendTrailingPathSeparator(file_path)
removeFileExtension(file_name_ext)
getFileExtension(file_name_ext)
addFileExtension(file_name, file_ext)
changeFileName(file_name_ext, new_name)
appendToFileName(file_name_ext, s)
asDateString(D)
daysInMonth(D)
daysInMonthForYear(month_number, year_number)
getDateString()
getTimeString()
appendTimeString(file_name_ext)
changeFileExtension(file_name_ext, new_ext)
getSafeName(file_name_ext)
getCleanName(file_name_ext)
getOriginalName(file_name_ext)
getOriginalPath(file_path)
getNextAvailableStem(parent_path, file_name, file_ext)
getNextAvailableName(parent_path, file_name_ext)
getNextAvailablePath(file_path)
promptToOverwriteFile(file_path)

— FILE MANAGEMENT HANDLERS —
renameFile(file_path, new_name_ext)
duplicateFile(file_path)
moveFile(file_path, new_file_path)
forceMoveFile(file_path, new_file_path)
moveFileToFolder(file_path, new_parent_path)
copyFile(file_path, new_file_path)
forceCopyFile(file_path, new_file_path)
copyFileToFolder(file_path, new_parent_path)
updateFile(file_path, source_file_path)
syncFile(file_path_1, file_path_2)
copyFolderHierarchy(folder_path, new_folder_path)
_mergeFolders(folder_1_path, folder_2_path)
archiveFile(file_path)
unarchiveFile(file_path)
_backUpFile(file_path)
deleteFile(file_path)
removeFile(file_path)
removeEmptyFolder(file_path)
removeFolder(file_path)
removeFolderContents(file_path)
_moveFileWithAdministratorPrivileges(a, b, user_password)
makeFile(file_path)
makeFolder(file_path)
makeFolderWithIntermediates(file_path)
makeTemporaryFolder()
_makeFolderTree(file_path, name_tree)
_getFolderTree(file_path)
renameFileSafely(file_path, new_name)
moveFileSafely(file_path, new_path)
moveFileToFolderSafely(file_path, new_parent_path, new_stem)
moveFolderContentsToFolderSafely(source_folder_path, destination_folder_path)
copyFileSafely(file_path, new_path)
copyFileToFolderSafely(file_path, new_parent_path, new_stem)
copyFolderContentsToFolderSafely(source_folder_path, destination_folder_path)
makeFileSafely(file_path)
makeFolderSafely(file_path)
makeHardLink(file_path, link_path)
countHardLinks(file_path)
isHardLinkOf(file_path_1, file_path_2)
_getHardLinks(file_path)
makeSymbolicLink(target_path, link_path)
forceMakeSymbolicLink(target_path, link_path)
isSymbolicLink(file_path)
checkSymbolicLink(link_path)
checkSymbolicLinks(file_path)
readSymbolicLink(link_path)
resolveSymbolicLink(link_path)
resolveSymbolicLinks(file_path)
updateSymbolicLink(link_path, new_target_path)
recreateSymbolicLink(link_path, hide_extension)
makeFinderAlias(file_path, alias_path)
isFinderAlias(file_path)
checkFinderAlias(alias_path)
resolveFinderAlias(alias_path)
_updateFinderAlias(alias_path, new_target_path)
isLink(file_path)
checkLink(link_path)
checkLinks(file_path)
resolveLink(link_path)
resolveLinks(file_path)
_updateLink()
getVolumeDeviceID(volume_path)
getPathWithInode(inode_string)
_isDotFile(file_path)
fileIsHidden(file_path)
hideFile(file_path)
unhideFile(file_path)
fileExtensionIsHidden(file_path)
hideFileExtension(file_path)
showFileExtension(file_path)
fileIsLocked(file_path)
lockFile(file_path)
unlockFile(file_path)
fileIsExecutable(file_path)
makeFileExecutable(file_path)
makeFileUnexecutable(file_path)
executeFile(file_path)
fileIsNewerThan(file_path_1, file_path_2)
fileIsOlderThan(file_path_1, file_path_2)
folderIsEmpty(folder_path)
folderIsEmptyIncludingInvisibles(folder_path)
listFolder(folder_path)
listFolderIncludingInvisibles(folder_path)
listFolderPaths(folder_path)
listFolderPathsIncludingInvisibles(folder_path)
getFirstPathInFolderMatchingName(parent_folder_path, name_match)
_getSubfolderByNameMatch(parent_folder_path, name_match)
findFiles(folder_path)
findFilesIncludingInvisibles(folder_path)
findFilesWithoutRecursion(folder_path)
findFilesWithoutRecursionIncludingInvisibles(folder_path)
findFinderAliases(folder_path)
findSymbolicLinks(folder_path)
readFile(file_specifier)
readFileIntoList(file_specifier)
writeToFile(file_specifier, text_to_write)
appendToFile(file_specifier, text_to_append)
appendToExistingFile(file_specifier, text_to_append)
openFile(file_path)
openFileInApplication(file_path, application_name)
previewFile(file_path)
revealFile(file_path)
revealFiles(file_list)
revealFileInBackground(file_path)
revealFilesInBackground(file_list)
getSelectedPath()
getSelectedPaths()
getInsertionPath()
getFrontFinderWindowTarget()
setInsertionPath(target_path)
selectFile(file_path)
selectFiles(file_list)
reselectFiles()
makeNewFinderWindow()
_isRecognizedFileExtension(file_ext)
_hasRecognizedFileExtension(file_name_ext)

— OTHER HANDLERS —
nothing()
isNothing(object_reference)
attempt(reference_reference)
emptyScript()
isApplication(O)
isReference(some_value)
dereference(some_value)
length__(O)
isEqualTo(value_1, value_2)
isEqualReferenceTo(value_1, value_2)
_isIdenticalTo(value_1, value_2)
referenceExists(some_reference)
unwrapReference(some_reference, default_value)
_init(script_object)
assert(condition)
assertValue(condition, return_value)
tif(condition, value_if_true, value_if_false)
setTIDs(new_TIDs)
resetTIDs()
getCurrentDate()
getBestTypeOf(O)
_getHandlerName(a_handler)
callHandlerWithArguments(f, argument_list)
wrap(O)
_testHandler(handler_name, list_of_arguments_and_expected_result)
eval(script_text)
evalWithParameters(script_text, parameters)
compileScriptText(script_text, output_path)
compileScriptFile(file_path)
recompileScriptFile(file_path)
decompileScriptFile(file_path)
compileScriptFileTo(file_path, output_path)
recompileScriptFileTo(file_path, output_path)
decompileScriptFileTo(file_path, output_path)
scriptFileIsDebugFormat(file_path)
loadVariableFromFile(file_specifier, variable_class)
loadScriptFromFile(file_specifier)
storeVariableInFile(file_specifier, variable_to_store)
loadListFromFile(file_specifier)
loadRecordFromFile(file_specifier)
escapeInput()
_escapeString(s)
escapeStringForShell(s)
_escapeStringForPython(s)
promptForPassword()
confirmListChanges(old_strings, new_strings, dialog_title, dialog_description)
doShellScript(script_text)
doShellScriptIgnoringErrors(script_text)
doShellScriptReturningStandardError(script_text)
doShellScriptCombiningStandardError(script_text)
doShellScriptInBackground(script_text)
doShellScriptInTerminal(script_text)
doShellScriptInBackgroundTerminal(script_text)
doShellScriptInTerminalWithPrompt(script_text)
logToWindow(s, output_tty)
quotedForm(C)
runScript(script_path)
runScriptInBackground(script_path)
_runScriptWithParameters(script_path, argument_list)
runScriptInBackgroundWithParameters(script_path, argument_list)
listBackgroundScriptInvocations()
runWorkflow(workflow_path)
runWorkflowInBackground(workflow_path)
runWorkflowWithInput(workflow_path, input)
runWorkflowInBackgroundWithInput(workflow_path, input)
getClipboard()
getClipboardPaths()
setClipboard(s)
copyToClipboard(a_string)
py(expression_text)
python(script_text)
python2(script_text)
python3(script_text)
_getWindowSize(a_window)
_setWindowSize(a_window, window_width, window_height)
getScreenSize()
playTaskCompleteChime()
playErrorChime()
playInteractionRequiredChime()
_logToFile(text_to_log)
_displayText(s)
getFrontmostApplication()
getFrontmostApplicationName()
getFrontmostApplicationPath()
getFrontmostApplicationProcess()
hideFrontmostApplication()
getApplicationProcess(application_identifier)
getProcessID(application_identifier)
processExists(pid)
getModifierKeys()
modifierKeysArePressed(modifier_keys)
waitForModifierKeyRelease(modifier_keys)
runInScriptMonitor(script_path)
runScriptInMenu(script_path)
registerCaller(script_object)
caller()
getSystemVersion()
truncateTrailingVersionZeroes(s)
versionIsEqualTo(s1, s2)
versionIsGreaterThan(s1, s2)
versionIsGreaterThanOrEqualTo(s1, s2)
getHostname()
getComputerName()
isInternetReachable()
getWiFiNetworkName()
getWiFiBSSID()
sleepDisplays()
shutdownWithConfirmation()
isSIPDisabled()
getDefaultsPaths(bundle_id)

I used to do something similar to this before use was a thing, but this causes the full library to be embedded as an object in each calling script (and requires recompilation of the calling script in order for the library to be updated).

I do have my “General Library” organized into logical sections, so the organization itself isn’t really the problem. Quite a few of the handlers call into each other, so I’m concerned how reliable that will be with AppleScript’s library loading mechanism.

If I do split up the library that way, it will require me to re-write all my calling scripts (e.g. change "General Library"'s someFunction() to "String Library"'s someFunction()) and a good chunk of the library itself (e.g. if someFunction() calls an unqualified someOtherFunction() that both used to be in the same library).