Hey folks! I’m excited to share a feature in development for the next update to FastScripts: Mouse Automation.
The idea here is to basically fill the hole left by GUI Scripting, which offers commands for things like keystrokes, but not for mouse movement, clicking, and dragging.
I’ve spent a good amount of time refining what I think is a pretty good scripting interface to the new support, but of course I’d love to hear feedback from any of you who have thoughts about such things. Here’s a screen capture of the new scripting interface for this functionality:
Something I hope to add before shipping this feature is an additional parameter to both “click mouse” and “move mouse” that would allow the position specified to be interpreted relative to some other point or object (such as a window).
Please let me know if you give it a try and have any feedback. Thanks!
I have quite a few of my own handlers I use for mouse clicking & have the following questions/comments:
Hover Delay/Easing Factor
When using virtual machines or remote desktop applications, synthetic clicks & mouse movements often don’t register if they happen instantaneously.
I use a handler called clickAtGlobalCoordinatesWithHoverDelayAndEasingFactor that allows you to specify a hover delay affecting how long to hover in-place before clicking & an easing factor that allows you to specify how long the mouse movement should take. The latter is particularly important, I think.
Coordinate System
Will your commands use the AppleScript-like coordinate system with {0, 0} as the top left of the screen, or the Core Graphics system? IMHO, the AppleScript coordinates would be better as it will match how AppleScript reports window coordinates & need less math.
Getting the Mouse Position
This would also be a useful command. Not currently possible with AppleScript or ASOC (edit: possible with ASOC, though it uses Core Graphics-like coordinates requiring screen size calculations); I resorted to the very inelegant solution of writing my own executable.
Relative Coordinates
I think that’s a great idea; it’s proved useful for me. I have handlers for clickAtRelativeCoordinatesInWindow for native applications & clickAtScaledCoordinatesInWindowWithOffsets for remote desktop connections, which may be scaled (requiring a scaling factor & horizontal/vertical offsets to account for window chrome). The latter may be niche.
There’s definitely a lot of room for improvements to GUI scripting in general. It’s one of the biggest pain points in AppleScript. The biggest holes, IMO, are:
The natural complement to mouse movements & clicking is being able to identify buttons on the screen. I don’t know if this is possible in ASOC, but I don’t think it wouldn’t be well-suited to a command-line tool because of TCC protections for screen recording. A proper scriptable application such as Fastscripts is probably the way to go.
AppleScript can’t directly report which modifier keys are currently pressed. Something else I had to write an executable for.
keystroke has some very nasty edge cases that cost me a lot of debugging time. I’ve worked around the biggest problem in pure AppleScript, but there’s still room for improvement.
Thanks for the encouragement and for the great feedback.
Delay/easing. Currently I am using a built-in (quite fast, 0.01 seconds) delay between each event that FastScripts queues up as part of the requested action. For example “Click”, “Drag”, “Release” would have a 0.01 second delay between each component. I’m curious to see how the initial effort works but will be happy to tune this as needed, and may even end up exposing scriptable controls over it.
The coordinate system is AppleScript-like for consistency with other AppleScript-facing interfaces.
Great idea - I’ll add “get mouse position” (or similarly named) to the list.
Interesting point about the scaled coordinates. Might leave the scaling as an exercise for the scripter for now - but will gauge interest as things move forward.
It’s great to hear that at least one person will enjoy this feature
Also thanks for the comments about other pain points.
Fully identifying buttons/elements on the screen would definitely be cool, but much more of an undertaking than I’m planning right now. Noted for the future!
Seems like “current keyboard modifiers” would be a good sidekick for “current mouse position”. I’ll look into adding this with this set of updates.
Let me know when you feel like elaborating about the send keys shortcomings.
Understand that for sure - haven’t gotten around to that one myself. I wasn’t thinking anything too fancy (no Accessibility framework), just simple image matching - i.e. getCoordinatesOfImageOn Screen(path_to_PNG). All that would do is let you know you’re clicking what you intend to click.
Very much so – it’s a common intermediate AppleScript question. Don’t know if anyone’s actually released any tool to get the current keyboard modifiers.
For some reason, keystroke sends the numbers as key codes for the number pad area of a full keyboard, rather than the number keys in the row above the QWERTY line.
I was getting intermittently garbled output when using keystroke with a remote desktop application, because as it turned out, the output depended on the state of the Num Lock key in the remote machine, which wasn’t consistent between connections (or visible/controlled by me).
So, it was toss-up whether keystroke "4" would type a 4 key or press a left arrow key. It took me a long time to figure out what was going on.
I wrote a little handler that fixed this, and also allows key commands to be pooled. For example:
-- Type "Testing123", wait 0.2 seconds, press the Escape key, press command-w
my sendKeys({"Testing123", 0.2, 53, {"w", {command down}}})
There are other issues, though, including that caps lock being enabled affects the output, and the fn key isn’t handled. Worse, perhaps, is that physically pressing a modifier key on the keyboard (such as command) will affect whether keystroke "q" types a q or quits the frontmost application.
You’re correct! It’s possible with ASOC. I’ve had the executable around since before ASOC existed.
I do still think it would make sense to bundle this with the other mouse actions in FastScripts, and have it return AppleScript-style coordinates (rather than the inverted y-axis that requires further calculation & getting the screen size).
Yes, that works, but also requires ASOC, and as you point out, AppleScript doesn’t have useful operators for bitmasks - it’s much easier & faster in ObjC or Swift.
This is a really common question for intermediate AppleScript users, has no native AppleScript implementation, and is useful in a lot of contexts (e.g. checking for modifier key release before using the built-in keystroke, or checking that no modifier keys are pressed before launching an application, etc.). For that reason, I think it could be useful for Daniel to include in FastScripts, especially if he adds a keystroke-like command.
Yes, almost anything is possible with ASOC! But the main emphasis of these features in FastScripts is to facilitate easier, more streamlined workflows. Similar to the regular expression support, which can also be done through ASOC but which is made way simpler by FastScripts.
For the modifier flags I envision FastScripts returning a list of enumerators, so testing for command key for example would look something like:
current keyboard modifiers contains command
I can definitely see how this would be a useful tool to have “at hand” for FastScripts users.
Regardless, the issue with both of the above solutions is that they require reaching for ASOC, which can have a 200-400 ms delay on first run as the bridging files are loaded (which is more of a problem with GUI scripting). This delay is experienced every time you run using osascript from the command line or using the native Script Menu, since they spawn a new process each time they are invoked.
As far as I know, FastScripts is the only script runner-type application that preloads the bridging files to prevent this delay.
Exactly – that’s why I think these make sense.
That looks great. keystroke has enumerations for command, option, control, and shift (no caps, fn key, escape key), but I can’t think of any good reason the codes or names would need to match, even if you did end up creating your own version of keystroke.
Oh yeah I’m sure that Daniel’s implementation of pure AppleScript functions in FastScripts will be more user-friendly and convenient than dealing with ASOC.
This version includes the requested additions for “current keyboard modifiers” and “current mouse position”.
It also exposes an optional parameter on click mouse to control the “cadence” of event dispatch. I mentioned earlier that events are dispatched with a bit of default stagger time, 0.01 seconds by default. The “with event cadence” parameter allows you to specify a different cadence either because you know it can be done faster, or (more likely) you discover it needs to happen more slowly.
Mouse Automation is a very exciting addition to FastScripts. I write many AppleScript scripts to automate various tasks. This involves a lot of GUI scripting. I will re-write some of my existing scripts to incorporate FastScript’s mouse automation and get back to you with my comments.
Based on my experience I can tell you that the addition of the ability to specify event cadence is very important. Moving the mouse programmatically frequently occurs faster than the GUI can accommodate.
I have two suggestions for additional features.
The execution of a GUI script can be “broken” if the user manually moves the cursor (i.e. using the mouse or trackpad.) I have attempted to learn how to disable user input during the execution of the script, using AppleScriptObjC, but have yet to succeed. This would make a nice addition to FastScript’s mouse automation; or if anyone knows how to do so, I would appreciate the information.
Going along with the above would be the option of hiding the cursor during execution of a GUI script. It might also make the script’s actions seem more “natural,” i.e. more like a built in function. I have attempted to use NSCursor’s hide: method. However, the cursor is unhidden when (I believe) the script is run by another application, in my case, FastScripts.
I have not tried making the script into an application in order to keep the cursor hidden, but it would be a nuisance to need to make every script an application.
Thank you, Daniel, for continuing to support and develop FastScripts. I have used it for such a long time that I don’t even recall when I began to do so! It is incredibly useful. And you nearly provide it at no cost. My cost per usage amount is sensibly zero. If only all the applications I use were of its quality.
@dmbrown Great to hear that you will also appreciate the enhancements! Thanks for the kind words about FsstScripts, it’s been a real joy for me to be able to maintain and extend it over the many years (decades!).
Interesting suggestions for additional enhancements, I’m not sure if I can “block out” the user during the event automation, but I’ll look into that possibility! I will also look into the possibility of hiding the cursor. Could be difficult since other apps can also show it proactively, but since FastScripts (or more accurately, its script runner) will be actively running throughout the process, maybe it can aggressively re-hide it and hope for the best.
Daniel - I’ve been working with FastScripts’s Mouse Automation. The cadence is an issue, even using “with event cadence.” For example, have a Script Debugger or Script Editor document open and arrange all your windows such that when the cursor is at {100, 100}, it is over the Desktop. Now execute the (nice and elegant) script:
tell application “FastScripts” to click mouse {100, 100} with event cadence X
where the value for X can be anything you want.
Clicking at {100, 100} should bring Finder to the front, but it does not. The following works:
tell application “FastScripts”
move mouse to {100, 100}
delay 0.1
click mouse
end tell
From what I can tell, it appears that “with event cadence” is only applicable to dragging. I think it would be good if “with event cadence” was applied between the two parts of “click mouse”, the move to the location and the subsequent click. This would eliminate manually adding the delay.
As you know, GUI scripts are frequently peppered with short delays. I have been writing GUI scripts for a long time. When I get a new computer, some of the scripts need to have their delay’s increased, presumably a result of the new computer being faster than the last. I think it would be useful to have the ability to set the default cadence in FastScripts’s Settings. This would allow users to set a default that is compatible with their computer and usage.
Even more flexibility would be to also have the ability to set the cadence as needed, such as
tell application “FastScripts”
set default cadence to 0.1
…
set default cadence to 0.5
…
end tell
Thanks for the feedback - I’ll have to look at how I can improve the cadence behavior. Don’t let me ship with this broken! If i neglect to fix it perfectly please let me know I did have the idea to offer a script-level setting like “default event cadence” … that would give FastScripts the ability to infer how much to slow down in a variety of circumstances.
I think that’s a great idea, provided you can make the default stay contained within a given script (otherwise, I can see it causing issues if some scripts explicitly set it but others don’t, and end up inheriting different default values). It would be very efficient for anything involving virtual machines / remote desktop connections.
I agree completely. I’ve done a fair amount of GUI scripting myself and the delays required when controlling the mouse programmatically are often longer than you’d expect. Some applications need the mouse to hover in place for x number of milliseconds before clicking, others need the mouse movement itself to take x number of milliseconds to move to the point you want to click.
This update includes a new script property: “user interaction disabled”. If you set this on “me” in your script:
set user interaction disabled of me to true
FastScripts will block all input events until the script finishes running, or until you set the property back to false.
This is a little risky since a bug in the script, or in FastScripts, could lead to input being blocked to the extent that a user can’t cancel the script or do anything else with their Mac. I’ll have to think about how to mitigate that, but I thought it would be worth sharing with the “bleeding edge” testers here for now.
Note: I wasn’t able to prevent “Mouse moves” from happening. But at least the user can’t interfere by clicking or keyboard pressing while the interaction is disabled.
As far as the “click mouse” situation described by @dmbrown above - I think the problem there is that I’m not doing an implicit ‘move mouse’ before the first click. I am injecting a “cadence” delay between each event, but in the case of “click mouse” it’s currently just clicking at the designated spot without moving there first. Adding an implicit move should introduce the desired cadence delays, I hope!
I did have the idea to offer a script-level setting like “default event cadence” … that would give FastScripts the ability to infer how much to slow down in a variety of circumstances.