FastScripts Feature Preview: Mouse Automation

Daniel,

When looking into disabling user input, I thought of the same risks. I intended to implement the disabling in such a way that prior to doing so, a separate background application would be started. Its only purpose was a “fail-safe” timer; in 30 seconds it would check if input was disabled and, if so, re-enable it. The GUI scripts I use would not need input disabled any longer than that. Perhaps that gives you some ideas.

I never figured out how to disabled user input, so I can not tell you how the envisioned “fail-safe” performed. I am interested to know how you did it, if you are comfortable divulging such information.

Thanks, you’ve given me an idea that I think is good: I will change the property to a command and offer an optional parameter “with timeout” or similar, so that the process for disabling user input will be something like:

disable user interaction with timeout 30

Which would achieve the goal you described of having user interaction automatically restored after 30 seconds.

As for how I achieved the disabling of interaction, I’m using a feature of the OS called “event taps” which are designed to both intercept and suppress events at the “HID” level - that is, where the events enter the system, more or less.

Stay tuned for more updates soon!

Daniel

1 Like

That looks good. This looks like it will work even if e.g. Script Debugger is running the script during development/debugging.

With the previous implementation (using a script property), I’m assuming that FastScripts had to be the application running the script?


When it comes to the “default cadence”, I think you’ll have the same issue? If it’s a script property, then the default cadence will only work when FastScripts is running it, not during development/debugging (of which there’s a lot with GUI scripting) in Script Debugger. That’ll be more of a problem, since it’ll affect timing in the script.

Daniel,

Will the “with timeout” command launch a separate application as the “timer”? The idea being (perhaps obvious) that even if the the script editor, a script application or FastScripts itself hangs or crashes, the independent timer application will live on.

I appreciate the information about event traps. I was looking into using NSEvents somehow, but never did get very far.

The way I’m approaching it now, the user interaction is disabled by the script runner process itself, which means that it actually won’t work when run from Script Debugger or another launcher besides FastScripts. I’ll have to think about that a bit - it would obviously be nice for the functionality to work when run from outside of FastScripts, but bundling the behavior up with the script execution itself has some advantages.

Whatever I end up doing, I don’t think there needs to be a separate app to “monitor” the suppressing process. The way it works, events can only be suppressed if the event filter is actively running, so I could build into the suppression code a timeout value that it would check on its own and uninstall itself when the time is right.

Here’s a link to Apple’s documentation on event taps if you’re curious: CGEventTapCreate | Apple Developer Documentation

Daniel

I think that makes perfect sense for disabling user interaction – it doesn’t matter (and may be useful) if it doesn’t work from within Script Debugger.


Just to echo what I said above, though, I think the opposite is true for how you implement the “default cadence” functionality.

Cheers!

Tonight’s beta release expands on the “disable user interaction” ideas, and I think what I’ve come up with works pretty well:

https://redsweater.com/fastscripts/FastScripts3.3b10.zip

You can now issue a command like this in a script:

tell app "FastScripts"
    disable user interaction with timeout 30
end

And it will cause all mouse and keyboard input from the user to be suppressed. Synthesized keyboard and mouse events should still work, so let me know if you run into any instance where they don’t.

If you try to run a script like above in Script Debugger or any other non-FastScripts launcher, it will generate an error the first time you run it, just to let you know it’s not working, and then be quiet (until FastScripts is quit and launched again).

Also note the “timeout” is the worst case scenario. FastScripts will restore user interaction when the script ends execution, whether the timeout has been reached or not.

Regarding the “disable user interaction” feature, another way to achieve this could be to display a transparent window that covers all screens and captures every mouse or keyboard event. This window could be closed by a specific keyboard shortcut when needed.
The problem here is the window layer…
Is this feasible or is it a stupid idea?

Hi Daniel – I’m not sure I like that behaviour. I have some complex scripts where the UI interaction is towards the end of the script, and it would be frustrating if the whole script ground to a halt just because this is the first run since FastScripts was launched. Many GUI scripts require frequent editing / debugging so I can see that happening often enough to get frustrating.


On a semi-related note, some other software like VMware Fusion uses the button combo “Command-Control” to re-enable user interaction. That could be an alternative/addition to the timeout feature.

I believe the transparent window would do an effective job of trapping mouse clicks, but it would be less effective at capturing keyboard interaction. This is because keystrokes are handed to the “focused” window/control, and nothing about having a transparent window on the screen would negate that (unless you made the transparent window focused - but then it would stymie all the desired automation scripting).

To be clear, the error is only generated when the script is run outside of FastScripts, so unless it’s a script you plan to always run outside of FastScripts, you would probably run into it only while authoring/testing the script. If you do plan to always run it outside of FastScripts, then the command will be ineffective so it might as well be removed from the script? That said, I’m still open to the possibility of removing this error. I just don’t want people to be surprised or confused when the command doesn’t work when invoked outside of FastScripts. Maybe I could put up a notification or something, instead.

Interesting point about the alternative “unlock” gesture. I suspect that in practice being able to just ask FastScripts to lock out the user for short periods of time will do the trick, but I’ll continue considering other options.

Daniel

1 Like

Thanks Daniel. Yes, I understand – personally, I find that any scripts where I’m forced to rely on GUI scripting need frequent editing. Things break very easily. So while I normally run them from FastScripts, I often have to edit them with Script Debugger. IMHO, the fact the command would do nothing in this context would be useful rather than harmful – it’s a plus if the GUI doesn’t lock up while debugging. Hopefully any end-user who has the wherewithal to edit the script would know that clicking things in the GUI would change the script’s behaviour (and if this were a concern, a manual check for current application could be added by the script author).

I was going to suggest displaying a notification instead as well, but that may break scripts that rely on parsing notifications or otherwise be cause for complaint. I certainly understand that removing the error may lead to more support requests to you for the feature “not working” even if you did include a prominent disclaimer.

Personally, I think having to account for the script erroring out in Script Debugger would add complexity to a script – it would be a shame to have to always remember to wrap the command in a try block to catch this behaviour for when the script is run from Script Debugger.

In any case, if you’re looking for a poll, my vote would be against – I’d be interested to know if anyone had a strong case for. I primarily write scripts for my own use, though, so I understand that this may be less of an issue for someone who develops scripts primarily for distribution (though, I don’t know how common that is with GUI scripting because it’s so fragile).

I have been working with the disable user interaction command and do not find its warning a problem. It only occurs the first time the command is used outside of FastScripts, with “first time” meaning the first time the command is used since FastScripts was launched. It sounds like tree_frog was interpreting “first time” to apply to each script, which would be a nuisance. However, I used the command in Script Debugger and received the warning. I then changed the script, saved it, then reopened it. The command generated no warning. I also created a new script using the command and, again, no warning.

That’s right - it’s going to issue the warning once for launch of FastScripts, period. I think I’ll leave it in for now and if it becomes a practical problem maybe I’ll remove it or add a “without warning” parameter to the command. I might also offer a “secret” preference to disable the warning altogether.

Thanks David – yes, I fully understand this is the behaviour. My opinion is as I described above – it could unexpectedly derail a long-running script towards the end of execution if e.g. the machine has been restarted.

but probably not of benefit, though?

I think that would be an excellent solution.

Another beta update, 3.3b11:

https://redsweater.com/fastscripts/FastScripts3.3b11.zip

This update includes the following changes:

  • “click mouse” now infers a mouse move event before clicking if the mouse isn’t already at the click point
  • New “default event cadence” app property
    • Sets for duration of script if script is run from FastScripts
    • Sets indefinitely as shared default for scripts run outside of FastScripts
  • New “secret” user default to suppress first-run warning about suppressing events
    defaults write com.red-sweater.fastscripts3 TrustMeIKnowAboutEventSuppression 1

I think we’re getting close to a usable solution here, at least for starters. I am thinking I’m going to postpone more advanced features like relative positioning (to windows/etc) and focus on getting this first set of changes out to the world first.

Daniel

2 Likes

:joy:

Thanks! Love it.

2 Likes

Greetings Daniel. I have been using FastScripts 3.3b11’s mouse automation. It is working much better in terms of the cadence. In other words, with commands such as click mouse {100, 100}, the click is registered. I have not used it as extensively as I would like due to other projects, but I will give you additional feedback as soon as I can.

However, I encountered a problem. FastScripts’s invoke command no longer works. In fast, the statement ‘tell application “FastScripts” to set dmb1 to script items’ results in an error. I use that function in other scripts, so I reinstalled version Version 3.2.8b2 (1770) for everyday use.

Let me know if any other information would be useful.

Mac Studio M2 Ultra
macOS 14.3

Thanks for reporting this. Seems like a high level problem with the way I added support for the “default event cadence” property. I’ll get it fixed in the next update.

Daniel

@dmbrown Hopefully this update will give you back your “invoke script” etc commands while also having the mouse automation: https://redsweater.com/fastscripts/FastScripts3.3b12.zip

1 Like