How to de-layer a list of lists

TimothyMurphy · January 13, 2023, 1:25am

Hi everyone, is there a clean way of converting a layered list of lists, which say contains 2 sublists, each with 6 further sublists, like the example below, into a list of lists with 12 sub items?
–From this:
{{{“Large Trunk”}, {“Large Suitcase”}, {“Large Bag”}, {“Large Purse”}, {“Small Trunk”}, {“Small Suitcase”}}, {{“Small Bag”}, {“Small Purse”}, {“Medium Trunk”}, {“Medium Suitcase”}, {“Medium Bag”}, {“Medium Purse”}}}

–To this: {{“Large Trunk”}, {“Large Suitcase”}, {“Large Bag”}, {“Large Purse”}, {“Small Trunk”}, {“Small Suitcase”}, {“Small Bag”}, {“Small Purse”}, {“Medium Trunk”}, {“Medium Suitcase”}, {“Medium Bag”}, {“Medium Purse”}}

I have toyed with flattening the list into a list of text and then messing around with text item delimiters and then looping back into a list, but i am sure there must be a better way? I have a more layered list than the one above, and I can’t work out how to recurse through 3 or 4 sub-layers…

Thanks for reading.

peavine · January 13, 2023, 2:30am

One solution can be found in the following thread, although it requires a few lines of additional code to make the list of individual items into a list of individual lists.

set theList to {{{"Large Trunk"}, {"Large Suitcase"}, {"Large Bag"}, {"Large Purse"}, {"Small Trunk"}, {"Small Suitcase"}}, {{"Small Bag"}, {"Small Purse"}, {"Medium Trunk"}, {"Medium Suitcase"}, {"Medium Bag"}, {{"Medium Purse"}, {"Added Item"}}}}

set theFlattenedList to flattenList(theList)
repeat with anItem in theFlattenedList
	set contents of anItem to anItem as list
end repeat
return theFlattenedList

on flattenList(theList)
	script util
		property flattenedList : theList
		property interimFlattenedList : missing value
	end script
	repeat while util's flattenedList's lists's length > 0
		set util's interimFlattenedList to {}
		repeat with currItem in util's flattenedList
			tell currItem's contents
				if its class = list then
					set util's interimFlattenedList to util's interimFlattenedList & it
				else
					set end of util's interimFlattenedList to it
				end if
			end tell
		end repeat
		set util's flattenedList to util's interimFlattenedList
	end repeat
	return util's flattenedList as list -- "as list" assures that the return value is dereferenced
end flattenList

The following works with your example but fails with more deeply layered lists (which the above-mentioned script appears to handle correctly):

use framework "Foundation"
use scripting additions

set theList to {{{"Large Trunk"}, {"Large Suitcase"}, {"Large Bag"}, {"Large Purse"}, {"Small Trunk"}, {"Small Suitcase"}}, {{"Small Bag"}, {"Small Purse"}, {"Medium Trunk"}, {"Medium Suitcase"}, {"Medium Bag"}, {"Medium Purse"}}}

set theArray to current application's NSArray's arrayWithArray:theList
set theList to (theArray's valueForKeyPath:"@unionOfArrays.self") as list

TimothyMurphy · January 13, 2023, 12:34pm

peavine, thank you very much for your thoughtful reply - which was very useful in allowing me to evaluate what worked best from my more specific requirements.

To be more specific, my list of layered lists is a list of layered lists of records - kind of like this:
{{{aDescription:“Large Trunk”}, {aDescription:“Large Suitcase”}, {aDescription:“Large Bag”}, {aDescription:“Large Purse”}, {aDescription:“Small Trunk”}, {aDescription:“Small Suitcase”}}, {{aDescription:“Small Bag”}, {aDescription:“Small Purse”}, {aDescription:“Medium Trunk”}, {aDescription:“Medium Suitcase”}, {aDescription:“Medium Bag”}, {aDescription:{“Medium Purse”}}}}

The first method you set out gave me this result, when the contents were 12 records:
{{“Large Trunk”}, {“Large Suitcase”}, {“Large Bag”}, {“Large Purse”}, {“Small Trunk”}, {“Small Suitcase”}, {“Small Bag”}, {“Small Purse”}, {“Medium Trunk”}, {“Medium Suitcase”}, {“Medium Bag”}, {{“Medium Purse”}}}

When the 12 contents were text:
{{“St Bernard”}, {“Wolfhound”}, {“Dogger”}, {“Alsatian”}, {“Mallinous”}, {“Setter”}, {“Labrador”}, {“Retriever”}, {“Spaniel”}, {“Poodle”}, {“Chihuahua”}, {“Terrier”}}

I notice that in the case of records the last item was enclosed in its own list. On my much larger list of a {4,18,14} layered list the result was flattened but not totally down to 1 layer.

The second method seems to work fine on text items, but with records I got back a flattened list of missing values. I guess NSArray on records may not be appropriate.

Following on your thread link into the macscripter site, applying Nigel Garvey’s handler, the result was perfect and it retained the record structure:
{{aDescription:“Large Trunk”}, {aDescription:“Large Suitcase”}, {aDescription:“Large Bag”}, {aDescription:“Large Purse”}, {aDescription:“Small Trunk”}, {aDescription:“Small Suitcase”}, {aDescription:“Small Bag”}, {aDescription:“Small Purse”}, {aDescription:“Medium Trunk”}, {aDescription:“Medium Suitcase”}, {aDescription:“Medium Bag”}, {aDescription:{“Medium Purse”}}}

This is exactly what i was looking for in this particular case. Running it on my complex set produced a fast result consistent with the above.

This was really useful for me to consider and be able to explore the application of different methods on a list of records and I would probably never have got there without your help and of course that of Nigel Garvey, BMose and Adrian Bell who supplied the other examples. Thanks to all!

TimothyMurphy · January 13, 2023, 1:02pm

To provide a wider context to my question on Layered lists of records, I attach an image above of a complex dialog box created using Shane’s (superb) Myriad Tables (many thanks!). The user enters some the required information for a given row, which is passed into a list of records. Some of the record values may contain a list of values, which means that a single row of data gets cloned according to the number of multiple items in a given record’s value. A single row like shown above can therefore expand very significantly, but then needs to be flattened out to a single layer list for processing each list item into a cURL request. The return values will be displayed also on a Myriad Table to allow sorting from high to low and from there either further processed or exported.

I hope this gives a good context that could be of help for other members.

peavine · January 13, 2023, 2:07pm

Tim. I’m glad you got that working. The ASObjC solution can be modified to work with your new example, and it appears to return the same result as Nigel’s script.

use framework "Foundation"
use scripting additions

set theList to {{{aDescription:"Large Trunk"}, {aDescription:"Large Suitcase"}, {aDescription:"Large Bag"}, {aDescription:"Large Purse"}, {aDescription:"Small Trunk"}, {aDescription:"Small Suitcase"}}, {{aDescription:"Small Bag"}, {aDescription:"Small Purse"}, {aDescription:"Medium Trunk"}, {aDescription:"Medium Suitcase"}, {aDescription:"Medium Bag"}, {aDescription:{"Medium Purse"}}}}

set theArray to current application's NSArray's arrayWithArray:theList
set theList to (theArray's valueForKeyPath:"@unionOfArrays.@self") as list

--> {{aDescription:"Large Trunk"}, {aDescription:"Large Suitcase"}, {aDescription:"Large Bag"}, {aDescription:"Large Purse"}, {aDescription:"Small Trunk"}, {aDescription:"Small Suitcase"}, {aDescription:"Small Bag"}, {aDescription:"Small Purse"}, {aDescription:"Medium Trunk"}, {aDescription:"Medium Suitcase"}, {aDescription:"Medium Bag"}, {aDescription:{"Medium Purse"}}}

Resource:

peavine · January 13, 2023, 2:46pm

Tim. I tested bmose’s script with your new example and it worked fine. The additional code I added to my earlier script to make a list of individual items into a list of individual lists is not required with the new example.

set theList to {{{aDescription:"Large Trunk"}, {aDescription:"Large Suitcase"}, {aDescription:"Large Bag"}, {aDescription:"Large Purse"}, {aDescription:"Small Trunk"}, {aDescription:"Small Suitcase"}}, {{aDescription:"Small Bag"}, {aDescription:"Small Purse"}, {aDescription:"Medium Trunk"}, {aDescription:"Medium Suitcase"}, {aDescription:"Medium Bag"}, {aDescription:{"Medium Purse"}}}}

set flattenedList to flattenList(theList)
--> {{aDescription:"Large Trunk"}, {aDescription:"Large Suitcase"}, {aDescription:"Large Bag"}, {aDescription:"Large Purse"}, {aDescription:"Small Trunk"}, {aDescription:"Small Suitcase"}, {aDescription:"Small Bag"}, {aDescription:"Small Purse"}, {aDescription:"Medium Trunk"}, {aDescription:"Medium Suitcase"}, {aDescription:"Medium Bag"}, {aDescription:{"Medium Purse"}}}

on flattenList(theList)
	script util
		property flattenedList : theList
		property interimFlattenedList : missing value
	end script
	repeat while util's flattenedList's lists's length > 0
		set util's interimFlattenedList to {}
		repeat with currItem in util's flattenedList
			tell currItem's contents
				if its class = list then
					set util's interimFlattenedList to util's interimFlattenedList & it
				else
					set end of util's interimFlattenedList to it
				end if
			end tell
		end repeat
		set util's flattenedList to util's interimFlattenedList
	end repeat
	return util's flattenedList as list -- "as list" assures that the return value is dereferenced
end flattenList

TimothyMurphy · January 13, 2023, 5:17pm

Thanks peavine, I realised I was getting missing values earlier in the ASObjC solution because I had “…valueForKeyPath:@unionOfArrays.self” instead of “…theArray’s valueForKeyPath:@unionOfArrays.@self” - this small (!) change fixed the missing value results and does give me the same result as Nigel’s script objects’ properties approach.

The second approach also works at my end and with a three layered list of records I get back a flat list of lists with 48 items.

So all 3 different approaches work from my practical perspective.

Thanks very much for the time, attention and feedback.

NigelGarvey · January 14, 2023, 10:36am

Hi both.

If you’re feeling speedfreakish, the recursive method mentioned in the other MacScripter topic mentioned in the one to which peavine linked appears to be about ten times as fast as the iterative one — in extreme cases, anyway.

on flatten(listOfLists)
	script o
		property fl : {}
		
		on flttn(l)
			script p
				property lol : l
			end script
			
			repeat with i from 1 to (count p's lol)
				set v to item i of p's lol
				if (v's class is list) then
					flttn(v)
				else
					set end of my fl to v
				end if
			end repeat
		end flttn
	end script
	
	tell o
		flttn(listOfLists)
		return its fl
	end tell
end flatten

set deeplyNestedList to {1, {2, {3, {4, {5, {6, {7, {8, {9, {10, {11, {12, {13, {14, {15, {16, {17, {18, {19, {20, {21, {22, {23, {24, {25, {26, {27, {28, {29, {30, {31, {32, {33, {34, {35, {36, {37, {38, {39, {40, {41, {42, {43, {44, {45, {46, {47, {48, {49, {50, {51, {52, {53, {54, {55, {56, {57, {58, {59, {60, {61, {62, {63, {64, {65, {66, {67, {68, {69, {70, {71, {72, {73, {74, {75, {76, {77, {78, {79, {80, {81, {82, {83, {84, {85, {86, {87, {88, {89, {90, {91, {92, {93, {94, {95, {96, {97, {98, {99, {100, {101, {102, {103, {104, {105, {106, {107, {108, {109, {110, {111, {112, {113, {114, {115, {116, {117, {118, {119, {120, {121, {122, {123, {124, {125, {126, {127, {128, {129, {130, {131, {132, {133, {134, {135, {136, {137, {138, {139, {140, {141, {142, {143, {144, {145, {146, {147, {148, {149, {150, {151, {152, {153, {154, {155, {156, {157, {158, {159, {160, {161, {162, {163, {164, {165, {166, {167, {168, {169, {170, {171, {172, {173, {174, {175, {176, {177, {178, {179, {180, {181, {182, {183, {184, {185, {186, {187, {188, {189, {190, {191, {192, {193, {194, {195, {196, {197, {198, {199, {200, {201, {202, {203, {204, {205, {206, {207, {208, {209, {210, {211, {212, {213, {214, {215, {216, {217, {218, {219, {220, {221, {222, {223, {224, {225, {226, {227, {228, {229, {230, {231, {232, {233, {234, {235, {236, {237, {238, {239, {240, {241, {242, {243, {244, {245, {246, {247, {248, {249, {250, {251, {252, {253, {254, {255, {256}, 257}}, 258}}, 259}}, 260}}, 261}}, 262}}, 263}}, 264}}, 265}}, 266}}, 267}}, 268}}, 269}}, 270}}, 271}}, 272}}, 273}}, 274}}, 275}}, 276}}, 277}}, 278}}, 279}}, 280}}, 281}}, 282}}, 283}}, 284}}, 285}}, 286}}, 287}}, 288}}, 289}}, 290}}, 291}}, 292}}, 293}}, 294}}, 295}}, 296}}, 297}}, 298}}, 299}}, 300}}, 301}}, 302}}, 303}}, 304}}, 305}}, 306}}, 307}}, 308}}, 309}}, 310}}, 311}}, 312}}, 313}}, 314}}, 315}}, 316}}, 317}}, 318}}, 319}}, 320}}, 321}}, 322}}, 323}}, 324}}, 325}}, 326}}, 327}}, 328}}, 329}}, 330}}, 331}}, 332}}, 333}}, 334}}, 335}}, 336}}, 337}}, 338}}, 339}}, 340}}, 341}}, 342}}, 343}}, 344}}, 345}}, 346}}, 347}}, 348}}, 349}}, 350}}, 351}}, 352}}, 353}}, 354}}, 355}}, 356}}, 357}}, 358}}, 359}}, 360}}, 361}}, 362}}, 363}}, 364}}, 365}}, 366}}, 367}}, 368}}, 369}}, 370}}, 371}}, 372}}, 373}}, 374}}, 375}}, 376}}, 377}}, 378}}, 379}}, 380}}, 381}}, 382}}, 383}}, 384}

flatten(deeplyNestedList)

ccstone · January 14, 2023, 3:04pm

Hey Nige,

Nice! That runs in about 0.015 seconds on my old i7 MacBook Air.

-Chris

peavine · January 14, 2023, 4:37pm

I had to put this to the test. My test list was Tim’s example concatenated to create a list containing 2048 sublists with each sublist containing 6 sublists and each of the 6 sublists containing a record.

The results were:

Nigel’s suggestion - 67 milliseconds
ASObjC solution - 261 milliseconds
bmose’s script - 783 milliseconds

To the extent possible, I compared the results of the three scripts and they were the same. The following is the ASObjC test script:

use framework "Foundation"
use scripting additions

-- untimed code
set theList to {{{aDescription:"Large Trunk"}, {aDescription:"Large Suitcase"}, {aDescription:"Large Bag"}, {aDescription:"Large Purse"}, {aDescription:"Small Trunk"}, {aDescription:"Small Suitcase"}}, {{aDescription:"Small Bag"}, {aDescription:"Small Purse"}, {aDescription:"Medium Trunk"}, {aDescription:"Medium Suitcase"}, {aDescription:"Medium Bag"}, {aDescription:{"Medium Purse"}}}}
repeat 10 times
	set theList to theList & theList
end repeat

-- start time
set startTime to current application's CACurrentMediaTime()

-- timed code
set newList to main(theList)
on main(theList)
	set theArray to current application's NSArray's arrayWithArray:theList
	set theList to (theArray's valueForKeyPath:"@unionOfArrays.@self") as list
end main

-- elapsed time
set elapsedTime to (current application's CACurrentMediaTime()) - startTime
set numberFormatter to current application's NSNumberFormatter's new()
if elapsedTime > 1 then
	numberFormatter's setFormat:"0.000"
	set elapsedTime to ((numberFormatter's stringFromNumber:elapsedTime) as text) & " seconds"
else
	(numberFormatter's setFormat:"0")
	set elapsedTime to ((numberFormatter's stringFromNumber:(elapsedTime * 1000)) as text) & " milliseconds"
end if

-- result
elapsedTime --> 261 milliseconds
# count newList --> 12288
# newList

TimothyMurphy · January 14, 2023, 11:16pm

Thanks Nigel, I did think it felt the fastest, but peavine’s numbers are amazing. Faster means more efficient I guess and I will study the method carefully to better understand what it is that makes it so much more efficient.

NigelGarvey · January 15, 2023, 1:46pm

Goodness! With a couple of minor modifications to the referencing and dereferencing, it’s about 85 times as fast as the iterative one (with the deeply nested list I used before):

on flatten(listOfLists)
	script o
		property fl : {}
		
		on flttn(l)
			script p
				property lol : l's contents
			end script
			
			repeat with v in p's lol
				if (v's class is list) then
					flttn(v)
				else
					set end of my fl to v
				end if
			end repeat
		end flttn
	end script
	
	o's flttn(listOfLists)
	return o's fl's contents
end flatten

set deeplyNestedList to {1, {2, {3, {4, {5, {6, {7, {8, {9, {10, {11, {12, {13, {14, {15, {16, {17, {18, {19, {20, {21, {22, {23, {24, {25, {26, {27, {28, {29, {30, {31, {32, {33, {34, {35, {36, {37, {38, {39, {40, {41, {42, {43, {44, {45, {46, {47, {48, {49, {50, {51, {52, {53, {54, {55, {56, {57, {58, {59, {60, {61, {62, {63, {64, {65, {66, {67, {68, {69, {70, {71, {72, {73, {74, {75, {76, {77, {78, {79, {80, {81, {82, {83, {84, {85, {86, {87, {88, {89, {90, {91, {92, {93, {94, {95, {96, {97, {98, {99, {100, {101, {102, {103, {104, {105, {106, {107, {108, {109, {110, {111, {112, {113, {114, {115, {116, {117, {118, {119, {120, {121, {122, {123, {124, {125, {126, {127, {128, {129, {130, {131, {132, {133, {134, {135, {136, {137, {138, {139, {140, {141, {142, {143, {144, {145, {146, {147, {148, {149, {150, {151, {152, {153, {154, {155, {156, {157, {158, {159, {160, {161, {162, {163, {164, {165, {166, {167, {168, {169, {170, {171, {172, {173, {174, {175, {176, {177, {178, {179, {180, {181, {182, {183, {184, {185, {186, {187, {188, {189, {190, {191, {192, {193, {194, {195, {196, {197, {198, {199, {200, {201, {202, {203, {204, {205, {206, {207, {208, {209, {210, {211, {212, {213, {214, {215, {216, {217, {218, {219, {220, {221, {222, {223, {224, {225, {226, {227, {228, {229, {230, {231, {232, {233, {234, {235, {236, {237, {238, {239, {240, {241, {242, {243, {244, {245, {246, {247, {248, {249, {250, {251, {252, {253, {254, {255, {256}, 257}}, 258}}, 259}}, 260}}, 261}}, 262}}, 263}}, 264}}, 265}}, 266}}, 267}}, 268}}, 269}}, 270}}, 271}}, 272}}, 273}}, 274}}, 275}}, 276}}, 277}}, 278}}, 279}}, 280}}, 281}}, 282}}, 283}}, 284}}, 285}}, 286}}, 287}}, 288}}, 289}}, 290}}, 291}}, 292}}, 293}}, 294}}, 295}}, 296}}, 297}}, 298}}, 299}}, 300}}, 301}}, 302}}, 303}}, 304}}, 305}}, 306}}, 307}}, 308}}, 309}}, 310}}, 311}}, 312}}, 313}}, 314}}, 315}}, 316}}, 317}}, 318}}, 319}}, 320}}, 321}}, 322}}, 323}}, 324}}, 325}}, 326}}, 327}}, 328}}, 329}}, 330}}, 331}}, 332}}, 333}}, 334}}, 335}}, 336}}, 337}}, 338}}, 339}}, 340}}, 341}}, 342}}, 343}}, 344}}, 345}}, 346}}, 347}}, 348}}, 349}}, 350}}, 351}}, 352}}, 353}}, 354}}, 355}}, 356}}, 357}}, 358}}, 359}}, 360}}, 361}}, 362}}, 363}}, 364}}, 365}}, 366}}, 367}}, 368}}, 369}}, 370}}, 371}}, 372}}, 373}}, 374}}, 375}}, 376}}, 377}}, 378}}, 379}}, 380}}, 381}}, 382}}, 383}}, 384}

flatten(deeplyNestedList)

ccstone · January 15, 2023, 5:31pm

Nice… Consistently in the neighborhood of 0.0035 seconds on my old system.