FastScripts Feature Preview: Mouse Automation

@dmbrown Great to hear that you will also appreciate the enhancements! Thanks for the kind words about FsstScripts, it’s been a real joy for me to be able to maintain and extend it over the many years (decades!).

Interesting suggestions for additional enhancements, I’m not sure if I can “block out” the user during the event automation, but I’ll look into that possibility! I will also look into the possibility of hiding the cursor. Could be difficult since other apps can also show it proactively, but since FastScripts (or more accurately, its script runner) will be actively running throughout the process, maybe it can aggressively re-hide it and hope for the best.

Daniel

Daniel - I’ve been working with FastScripts’s Mouse Automation. The cadence is an issue, even using “with event cadence.” For example, have a Script Debugger or Script Editor document open and arrange all your windows such that when the cursor is at {100, 100}, it is over the Desktop. Now execute the (nice and elegant) script:

tell application “FastScripts” to click mouse {100, 100} with event cadence X

where the value for X can be anything you want.
Clicking at {100, 100} should bring Finder to the front, but it does not. The following works:

tell application “FastScripts”
move mouse to {100, 100}
delay 0.1
click mouse
end tell

From what I can tell, it appears that “with event cadence” is only applicable to dragging. I think it would be good if “with event cadence” was applied between the two parts of “click mouse”, the move to the location and the subsequent click. This would eliminate manually adding the delay.

As you know, GUI scripts are frequently peppered with short delays. I have been writing GUI scripts for a long time. When I get a new computer, some of the scripts need to have their delay’s increased, presumably a result of the new computer being faster than the last. I think it would be useful to have the ability to set the default cadence in FastScripts’s Settings. This would allow users to set a default that is compatible with their computer and usage.

Even more flexibility would be to also have the ability to set the cadence as needed, such as

tell application “FastScripts”
set default cadence to 0.1

set default cadence to 0.5

end tell

Regards,
David Brown

Thanks for the feedback - I’ll have to look at how I can improve the cadence behavior. Don’t let me ship with this broken! If i neglect to fix it perfectly please let me know :wink: I did have the idea to offer a script-level setting like “default event cadence” … that would give FastScripts the ability to infer how much to slow down in a variety of circumstances.

I think that’s a great idea, provided you can make the default stay contained within a given script (otherwise, I can see it causing issues if some scripts explicitly set it but others don’t, and end up inheriting different default values). It would be very efficient for anything involving virtual machines / remote desktop connections.

I agree completely. I’ve done a fair amount of GUI scripting myself and the delays required when controlling the mouse programmatically are often longer than you’d expect. Some applications need the mouse to hover in place for x number of milliseconds before clicking, others need the mouse movement itself to take x number of milliseconds to move to the point you want to click.

Another beta release to play with, for those who are enjoying the process:

https://redsweater.com/fastscripts/FastScripts3.3b9.zip

This update includes a new script property: “user interaction disabled”. If you set this on “me” in your script:

set user interaction disabled of me to true

FastScripts will block all input events until the script finishes running, or until you set the property back to false.

This is a little risky since a bug in the script, or in FastScripts, could lead to input being blocked to the extent that a user can’t cancel the script or do anything else with their Mac. I’ll have to think about how to mitigate that, but I thought it would be worth sharing with the “bleeding edge” testers here for now.

Note: I wasn’t able to prevent “Mouse moves” from happening. But at least the user can’t interfere by clicking or keyboard pressing while the interaction is disabled.

Daniel

1 Like

As far as the “click mouse” situation described by @dmbrown above - I think the problem there is that I’m not doing an implicit ‘move mouse’ before the first click. I am injecting a “cadence” delay between each event, but in the case of “click mouse” it’s currently just clicking at the designated spot without moving there first. Adding an implicit move should introduce the desired cadence delays, I hope!

1 Like

Daniel,

When looking into disabling user input, I thought of the same risks. I intended to implement the disabling in such a way that prior to doing so, a separate background application would be started. Its only purpose was a “fail-safe” timer; in 30 seconds it would check if input was disabled and, if so, re-enable it. The GUI scripts I use would not need input disabled any longer than that. Perhaps that gives you some ideas.

I never figured out how to disabled user input, so I can not tell you how the envisioned “fail-safe” performed. I am interested to know how you did it, if you are comfortable divulging such information.

Thanks, you’ve given me an idea that I think is good: I will change the property to a command and offer an optional parameter “with timeout” or similar, so that the process for disabling user input will be something like:

disable user interaction with timeout 30

Which would achieve the goal you described of having user interaction automatically restored after 30 seconds.

As for how I achieved the disabling of interaction, I’m using a feature of the OS called “event taps” which are designed to both intercept and suppress events at the “HID” level - that is, where the events enter the system, more or less.

Stay tuned for more updates soon!

Daniel

1 Like

That looks good. This looks like it will work even if e.g. Script Debugger is running the script during development/debugging.

With the previous implementation (using a script property), I’m assuming that FastScripts had to be the application running the script?


When it comes to the “default cadence”, I think you’ll have the same issue? If it’s a script property, then the default cadence will only work when FastScripts is running it, not during development/debugging (of which there’s a lot with GUI scripting) in Script Debugger. That’ll be more of a problem, since it’ll affect timing in the script.

Daniel,

Will the “with timeout” command launch a separate application as the “timer”? The idea being (perhaps obvious) that even if the the script editor, a script application or FastScripts itself hangs or crashes, the independent timer application will live on.

I appreciate the information about event traps. I was looking into using NSEvents somehow, but never did get very far.

The way I’m approaching it now, the user interaction is disabled by the script runner process itself, which means that it actually won’t work when run from Script Debugger or another launcher besides FastScripts. I’ll have to think about that a bit - it would obviously be nice for the functionality to work when run from outside of FastScripts, but bundling the behavior up with the script execution itself has some advantages.

Whatever I end up doing, I don’t think there needs to be a separate app to “monitor” the suppressing process. The way it works, events can only be suppressed if the event filter is actively running, so I could build into the suppression code a timeout value that it would check on its own and uninstall itself when the time is right.

Here’s a link to Apple’s documentation on event taps if you’re curious: CGEventTapCreate | Apple Developer Documentation

Daniel

I think that makes perfect sense for disabling user interaction – it doesn’t matter (and may be useful) if it doesn’t work from within Script Debugger.


Just to echo what I said above, though, I think the opposite is true for how you implement the “default cadence” functionality.

Cheers!

Tonight’s beta release expands on the “disable user interaction” ideas, and I think what I’ve come up with works pretty well:

https://redsweater.com/fastscripts/FastScripts3.3b10.zip

You can now issue a command like this in a script:

tell app "FastScripts"
    disable user interaction with timeout 30
end

And it will cause all mouse and keyboard input from the user to be suppressed. Synthesized keyboard and mouse events should still work, so let me know if you run into any instance where they don’t.

If you try to run a script like above in Script Debugger or any other non-FastScripts launcher, it will generate an error the first time you run it, just to let you know it’s not working, and then be quiet (until FastScripts is quit and launched again).

Also note the “timeout” is the worst case scenario. FastScripts will restore user interaction when the script ends execution, whether the timeout has been reached or not.

Regarding the “disable user interaction” feature, another way to achieve this could be to display a transparent window that covers all screens and captures every mouse or keyboard event. This window could be closed by a specific keyboard shortcut when needed.
The problem here is the window layer…
Is this feasible or is it a stupid idea?

Hi Daniel – I’m not sure I like that behaviour. I have some complex scripts where the UI interaction is towards the end of the script, and it would be frustrating if the whole script ground to a halt just because this is the first run since FastScripts was launched. Many GUI scripts require frequent editing / debugging so I can see that happening often enough to get frustrating.


On a semi-related note, some other software like VMware Fusion uses the button combo “Command-Control” to re-enable user interaction. That could be an alternative/addition to the timeout feature.

I believe the transparent window would do an effective job of trapping mouse clicks, but it would be less effective at capturing keyboard interaction. This is because keystrokes are handed to the “focused” window/control, and nothing about having a transparent window on the screen would negate that (unless you made the transparent window focused - but then it would stymie all the desired automation scripting).

To be clear, the error is only generated when the script is run outside of FastScripts, so unless it’s a script you plan to always run outside of FastScripts, you would probably run into it only while authoring/testing the script. If you do plan to always run it outside of FastScripts, then the command will be ineffective so it might as well be removed from the script? That said, I’m still open to the possibility of removing this error. I just don’t want people to be surprised or confused when the command doesn’t work when invoked outside of FastScripts. Maybe I could put up a notification or something, instead.

Interesting point about the alternative “unlock” gesture. I suspect that in practice being able to just ask FastScripts to lock out the user for short periods of time will do the trick, but I’ll continue considering other options.

Daniel

1 Like

Thanks Daniel. Yes, I understand – personally, I find that any scripts where I’m forced to rely on GUI scripting need frequent editing. Things break very easily. So while I normally run them from FastScripts, I often have to edit them with Script Debugger. IMHO, the fact the command would do nothing in this context would be useful rather than harmful – it’s a plus if the GUI doesn’t lock up while debugging. Hopefully any end-user who has the wherewithal to edit the script would know that clicking things in the GUI would change the script’s behaviour (and if this were a concern, a manual check for current application could be added by the script author).

I was going to suggest displaying a notification instead as well, but that may break scripts that rely on parsing notifications or otherwise be cause for complaint. I certainly understand that removing the error may lead to more support requests to you for the feature “not working” even if you did include a prominent disclaimer.

Personally, I think having to account for the script erroring out in Script Debugger would add complexity to a script – it would be a shame to have to always remember to wrap the command in a try block to catch this behaviour for when the script is run from Script Debugger.

In any case, if you’re looking for a poll, my vote would be against – I’d be interested to know if anyone had a strong case for. I primarily write scripts for my own use, though, so I understand that this may be less of an issue for someone who develops scripts primarily for distribution (though, I don’t know how common that is with GUI scripting because it’s so fragile).

I have been working with the disable user interaction command and do not find its warning a problem. It only occurs the first time the command is used outside of FastScripts, with “first time” meaning the first time the command is used since FastScripts was launched. It sounds like tree_frog was interpreting “first time” to apply to each script, which would be a nuisance. However, I used the command in Script Debugger and received the warning. I then changed the script, saved it, then reopened it. The command generated no warning. I also created a new script using the command and, again, no warning.

That’s right - it’s going to issue the warning once for launch of FastScripts, period. I think I’ll leave it in for now and if it becomes a practical problem maybe I’ll remove it or add a “without warning” parameter to the command. I might also offer a “secret” preference to disable the warning altogether.