Speeding up do shell script

asobjc

(Phil Stokes) #1

This is really just a do shell script curiosity rather than a problem, but I’d welcome any thoughts.

I was looking for way to get user names and IDs from AppleScript and noticed that System Events is kind enough to give us user names, so starting there:

set userNames to {}
tell application "System Events"
	repeat with aUser in users
		set end of userNames to aUser's name & " "
	end repeat
end tell

However, System Events isn’t so generous to provide the user id numbers that I want to go along with them. For that, it seems we need to go to the shell. So, adding the above to a do shell script loop, we can get the required info* like this:

Script 1

set userNames to {}
set userIDs to {}
tell application "System Events"
	repeat with aUser in users
		set end of userNames to aUser's name & " "
	end repeat
end tell

repeat with i from 1 to count of userNames
	set aName to item i of userNames
	set userID to do shell script "id -u " & aName
	set end of userIDs to {aName, userID}
end repeat
userIDs

That gives the output I want, but it’s noticeably slow. Best time I could get out of it was about ~0.75-0.85 seconds**.

Now the fastest way to get the info I want directly in the shell is to do this:

dscl . list /Users UniqueID | egrep -v ^'_|daemon|nobody|root'

But surprisingly, wrapping that inside a ‘do shell script’ is even slower (~1.0) than the first script.

Script 2

do shell script "dscl . list /Users UniqueID | egrep -v ^'_|daemon|nobody|root'"

That surprised me because I though the first script’s slowness was probably down to the repeated do shell script calls in the loop, but no. There’s only one call here and it’s still slow. Just to be clear, this can’t be put down to the shell command. Try it directly in the shell and you’ll see just how fast it is when run natively.

But what came as even more of a surprise was this one, which turns out to be the fastest of all (<0.4):

Script 3

do shell script "userNames=$(osascript -e 'set usrNms to {}' -e 'tell app \"System Events\"' -e 'repeat with aUser in users' -e 'set end of usrNms to name of aUser & \" \"' -e 'end repeat' -e 'end tell' -e 'set str to items of usrNms as text');for u in $userNames; do printf $u\" \" ; id -u $u; done"

This crazy construction splits the applescript up into single lines and calls each line with osascript, then uses the shell to parse the result and iterate over each username. It is, in effect, a shell version of the first script, but THEN wrapped inside of a do shell script.

That such byzantine construction should turn out to be twice as fast as the first one and nearly three times faster than the second one is something I find completely baffling.

I don’t suppose anyone can offer any enlightenment on this curiosity?

Either way, the takeaway here for me is that it’s worth experimenting with your do shell script's if speed is at all an issue.

* I should point out that Script 2 will also reveal any hidden users, whereas Script 1 and Script 3 will not.
** Caveat on the timings: these are based on SD’s timer; I didn’t try the scripts in SE. I’d expect them to be individually faster, but presumably relatively the same to each other.


(Shane Stanley) #2

To be able to log stuff, script editors have to insert callbacks for when Apple events are handled. That unavoidably adds some overhead that distorts timings. And you’re comparing a script that sends many Apple events with others that send only one.

If you want to compare how long they will take from an applet — which is where I presume they will ultimately run — you really need to time them in an applet.

You should also get a more accurate value if you run them in my Script Geek.app. It’s more accurate simply because it doesn’t install any logging callbacks. When I compare your scripts 1 and 3 there, script 1 is faster —by around 10-15%.


(Phil Stokes) #3

I understand the point about logging; that’s why Script 2 is relevant.

If logging Apple Events was the only thing going on here, it doesn’t explain the disparity between Scripts 2 and 3.


(Jim Underwood) #4

I tend to not lose a lot of sleep over 0.35 sec, especially for such things as listing a very small list, like the number of user accounts on a Mac. Now days I find my personal time (i.e., programmer’s time) is worth a lot more than a few tenths of a second. LOL


(Phil Stokes) #5

The absolute times are of no importance. Understanding what’s going on under the hood to cause the relative differences could well be.

Even if not, curiosity about how things work at deeper levels is part of what both drove me into programming and keeps me interested.

I’m particularly fascinated by the apparent phenomenon of a speed increase when sending an applescript through osascript and do shell script in a script editor. It defies my understanding of how script editors work (presumably by creating an object with [[osascript alloc] initWithSource:] or similar, or the NSAppleScript class.


(Shane Stanley) #6

FWIW, I just put 2 and 3 in Script Geek and ran them 100 times. Script 3 took nearly 3x as long.


(Phil Stokes) #7

That’s at least more intuitive to me, but still doesn’t explain why it’s quicker in SD, given that we assume there’s no difference in the amount of apple events inserted by SD in Script 2 and Script 3.

But in some sense that does seem to be mystery solved. The speed increase I’m seeing in Script 3 appears to be peculiar to SD.

I just ran them all in Script Editor, and while Script 1 is marginally slower (there’s a perceptible delay between run and result) as I’d expect, there’s no perceptible delay between hitting the run button and seeing the result in SE for either Script 2 or Script 3. Both complete pretty much instantaneously just as if they were run in the shell.


(Shane Stanley) #8

In SD here, script 1 is taking ~0.44, script 2 ~0.15, and script 3 ~0.28. Isn’t that roughly in line with your expectations?


(Phil Stokes) #9

Yes, but not with my results yesterday.

However, I am seeing somewhat similar results to yours today. Just now I got 0.19, 0.06, and 0.19.

I did about a dozen runs yesterday getting the results I posted in the OP, but they were all within the same 15 minutes-ish period, so perhaps they were an anomaly caused by something local.


(Nigel Garvey) #10

I’m getting timings in line with Shane’s, although all the scripts are about twice as fast in Script Debugger than in Script Editor for some reason.

Here, for the exercise, is a version of script 1 that’s almost as fast as script 2:

tell application "System Events" to set userNames to name of users
set astid to AppleScript's text item delimiters
set AppleScript's text item delimiters to "; id "
set shellScript to "u=$(id " & userNames & ";); echo \"$u\" | sed -E 's/uid=([0-9]+)\\(([^)]+).+/\\2" & tab & "\\1/'"
set AppleScript's text item delimiters to astid
set userIDs to (do shell script shellScript)

(Phil Stokes) #11

Now that IS weird, indeed!

I’m getting 0.11 on yours, 0.06 on Script 2.


(Shane Stanley) #12

Script Debugger does make allowance for as much of the overhead as it’s possible to measure.


(Phil Stokes) #13

Thanks for the input everyone. I guess that’s “mystery solved”. :smiley:

So let me just round this off by going back to the purpose of this script. Why would anyone want a list of users and ids? Most people probably wouldn’t, but for sys admin and security purposes it can be useful data.

For those purposes, and as I mentioned in a note in the OP, you’d really want to use Script 2 as that also reveals the existence of any hidden login users. These are login accounts that do not appear in System Preferences ‘Users & Groups’ list or at the Login screen (hidden login users, if there are any, are also recorded in /Library/Preferences/com.apple.loginwindow.plist).

For the same purposes, it would also be useful to have a list of any previous users that have since been deleted. These, if there are any, are recorded in the /Library/Preferences/com.apple.preferences.accounts.plist. I don’t know of a way to get that from dscl, but we can read and parse the plist quickly enough and add it to our previous list of users.

Thus Script 5 (counting Nigel’s input as Script 4), gives us a list of all login user accounts and their user ids, past and present (oh, and for the record, I’m getting 0.13 in SD for this one! :sunglasses: ):

set cut to " = "
set delNames to {}
set deletedUsers to {}
set currentUsers to "Login User List:" & return

set currentUsers to currentUsers & (do shell script "dscl . list /Users UniqueID | egrep -v ^'_|daemon|nobody|root'")
set currentUsers to currentUsers & return & return & "Deleted User List:" & return

try
	set deletedUsers to paragraphs of (do shell script "defaults read /Library/Preferences/com.apple.preferences.accounts")
end try

repeat with i from 1 to count of deletedUsers
	set this_line to item i of deletedUsers
	if this_line contains ":RealName" then
		set o to offset of cut in this_line
		set delName to text (o + 3) thru -2 of this_line
		set next_line to item (i + 1) of deletedUsers
		set o to offset of cut in next_line
		set delName to delName & tab & tab & tab & (text (o + 3) thru -2 of next_line)
		set currentUsers to currentUsers & delName & return
	end if
end repeat

currentUsers

(Jim Underwood) #14

Thanks for solving the mystery, Phi. :smile:

Thanks for that – good to know.

You must have a slow Mac, Phil. On my iMac-27, late 2015 model, SD6 took ONLY 0.09 sec. LOL

All-in-all, pretty damn fast for a script that has two shell script calls. :wink:

Thanks for the optimization – something I would never have had time for. :smile:


(Phil Stokes) #15

Yup, a struggling late 2014 iMac 5K. Intel i5 w/ fusion drive (tho’ at least it’s got the 128GB SSD, instead of the sliver they put in the later models, and a 7200rpm platter).

I might shell out for that iMac Pro if it’ll get me 0.4 or better on this script.

Mind you, my old mid-2009 MBP running 10.6.8 and a 128GB SSD gives 0.6 so maybe not such a good buy for that .2 return. :tongue:


(Shane Stanley) #16

Or he may just have more accounts…


(Phil Stokes) #17

That’s an interesting comment. How does the number of accounts on a mac affect the speed of script execution?

Scratch that. Got it!


(Nigel Garvey) #18
do shell script "echo 'Login User List:' ; dscl . list /Users UniqueID | egrep -v ^'_|daemon|nobody|root' ; defaults read /Library/Preferences/com.apple.preferences.accounts || echo ''  | sed -En '
1 i\\'$'\\n''\\'$'\\n''Deleted User List:
/:RealName/ {
	N
	s/^[^=]+= |.$//g
	s/.\\n[^=]+= /'$'\\t\\t\\t''/p
}'"

:wink:

But the sed’s based on Phil’s AppleScript code. I don’t have any deleted users on which to test it.


(Phil Stokes) #19

It’s faster here, but the output isn’t so neat.


(Nigel Garvey) #20

Mmm. Yes. That output’s total rubbish. Damned if I can see what’s causing it though and I can’t reproduce it. :\