I’m trying to automate filling out a complex web form on Google Chrome and it’s just not working. Neither SD Explorer nor GUI Browser can show me the fields and buttons I need to access.
(Does Bill Cheeseman come by here?)
When I open the page in Safari, I can use either to locate the field, but when I build a command it errors out with a: “System Events got an error: Can’t get splitter group 1 of menu bar 1 of application process “Safari”. Invalid index.”
Well the Safari issue is a bug. Not sure if it’s UI Browser or the System or Safari, but what happens is that when I use UI Browser to identify an interface item it points to the wrong window. In some cases the last window I looked at, even if that window is closed. By fixing that with each command I’m back in business.
Ed, generally speaking, neither AppleScript nor System Events has any details about the contents of a web page. IMO, the best way to automate your task is using JavaScript in Browser, which, of course, can be executed from AppleScript.
I do a lot of webscraping, most of which are posted in the Keyboard Maestro Forum. But if you will post the URL, a screenshot showing the fields you want to access, I’ll take a look to see if I can help.
BTW, Keyboard Maestro makes this very easy, as KM has a number of native Actions to get/set web page fields for Safari and Chrome. For more info, see Browser Form Actions (KM Wiki).
I’m filling out forms, not web scraping.
There’s actually a lot of information about and details about the contents of a web page in System Events.
Open the system events dictionary, open explorer, then drill down through your browser’s application process to the window and tab with your page and you might be surprised.
I’m a little worried that a simple update to their process could throw the whole thing out of whack. From what I’ve seen, javaScript would be just a vulnerable.
Here’s what a handler looks like to change the value of one form:
on SetValueOfSlugField(newValue)
tell application "System Events"
tell application process "Safari"
tell window 1
tell text field 1 of group 3 of group 4 of group 1 of group 2 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1
select it
keystroke newValue
delay 1
keystroke tab & tab
end tell
end tell
end tell
end tell
end SetValueOfSlugField
I understand. By “webscraping” I simply meant accessing/setting data on a web page. Do it all the time.
But if the simplest thing changes, your UI structure is broken.
I’m just guessing, but I’d guess that << 1% of all web page form automation is done using AppleScript System Events. JavaScript is the language of the web browser, used by millions.
It probably will impact UI scripting. JavaScript can be fairly safe from changes, as long as the id/class of the object doesn’t change (which is actually fairly rare IME). I use Xpath and querySelector all the time to do this.
But if you want to stick with your AppleScript hammer, that’s fine with me. 
Good luck.
Those commands being sent to a text field UI element give me nightmares. Did you check to see what properties are available for the element that might yield a more robust method of achieving the same outcome ?
Generally speaking, a text field UI element has a value property, which equates to the text content within the field, and is both gettable and settable. That in itself negates the need for the first three out of the four commands, unless keystroke tab & tab simply inserts two tab characters at the end of the text, in which case that’s also now joyfully superfluous.
However, if it functions to move the focus from that text field to the next, then to the next, I suspect applying similar functional changes to how those fields are edited will remove a lot of the fragility in the larger AppleScript, especially if, as you say, it is attempting to manipulate “a complex web form”, which sounds like there are many points where something could falter and potentially stop the rest of the script from working.
One other change to the way you implement your AppleScript code that would make it more resistant to possible future alterations in the UI element hierarchy, is to remove your code’s dependency upon knowing the contents of that hierarchy.
So, this line’s entire purpose is to target text field 1:
tell text field 1 of ¬
group 3 of ¬
group 4 of ¬
group 1 of ¬
group 2 of ¬
UI element 1 of ¬
scroll area 1 of ¬
group 1 of ¬
group 1 of ¬
tab group 1 of ¬
splitter group 1
(* stuff *)
end tell
and it’s a very reasonable line of code to do it, until one UI element within that very long list no longer exists, or a new one gets added, etc.
Instead, starting with window 1 as your “root” UI element object, you can construct an iterative or recursive descent through the UI tree, examining as it goes every UI element of every branch, and terminating once it finds the element that matches criteria specific to the one you want; or perhaps build a list of multiple elements that you know in advance will be relevant, meaning this search only has to happen once.
Criteria for matching can be anything accessible by AppleScript, but will be most typically a class match, a property value match, or an attribute value match (although sometimes the existence of an attribute is a good discriminator).
One thing I’ve found out is that with this web page if I simply set the value of the field the page does not remember it. With one tab, it would sometimes fail, but with two tabs it always works. (adding a delay after the first tab works too, but a second tab is quicker).
That’s not practical here. The hierarchy is so complex and arbitrary that a script inspecting the value of every field would take way too long. Plus, these are empty forms that are being filled out.
The only way I’m able to get the values is using UI Browser, and I’m writing the scripts in a way that if the hierarchy changes it would be a simple matter to replace the reverence to each element using UI Browser.
You could see if setting the value of attribute "AXValue" allows for retention of the data.
Sounds like you’re doing a whole lot of assuming without any actual testing. What web page is this ?
No it’s not.
OK.
Seriously, though, why don’t you use JavaScript ?
What web page are you working with ?
I don’t know javascript. I’m dyed in the wool AppleScript.
Yes the only I am able to get values is using UI Browser. And, yes, I have been pounding away and testing. You may have other ways to get access to fields in Chrome but UI Browser is all I got.
The page is using SNAP to build web pages. It’s not a public page. I’m trying to automate completing templates.
You’ve learned to use ASObjC, right?
JavaScript is way easier to learn, and there are millions of examples, and lots of help (stackoverflow.com).
Actually, you have the main tool also – it’s built into the Chrome (and Safari) browser. Just right-click on any field, and choose “Inspect”.
Ed, take a look at this quick video demo:
If you want to try learning this, I’ll be glad to help.
OK, In my form that gets me this:
<input name="headline" data-attrpath="headlines,basic" maxlength="200" data-char-limit="200" placeholder="Headline for the P2P content item" value="Monday's TV highlights: 'Salvation' on CBS">
I’m not not sure what would be the next step? What would a javascript that entered date into that field look like?
Also, in addition to setting values of fields and clicks a button the page also has pull down menus and pull down menu fields (not sure what they’re called) where you start typing and a list of names appears in a menu as you type which start with the letters you type, as you type them.
Thanks for taking the time with this. My UI scripting solution is kind of clunky and fragile, to the point where I don’t think I could trust it to work for other users. (Although it will save me a lot of time.)
Ed, here’s the complete script, AppleScript with embedded JavaScript.
This works for me using your one-line HTML in a simple HTML form page I have.
It could fail for you IF there are more than one <input name="headline"> fields on the page.
But let’s start here.
It should return “OK” if everything worked, and, of course, you should see the field with the changed value on the web page.
The AppleScript and JavaScript are setup to work with any form on that page that can be uniquely identified by the HTML name attribute.
set formFieldName to "headline"
set newFieldValue to "New Value from Script"
set jsStr to "
'use strict';
(function myMain() { // this will auto-run when script is executed
//debugger;
var scriptResults = 'TBD';
var fldName = '" & formFieldName & "';
var oField = document.querySelector('input[name=' + fldName + ']');
if (oField) {
oField.value = '" & newFieldValue & "';
scriptResults = 'OK';
} else { scriptResults = '[ERROR] Form Field with this Name NOT Found: ' + fldName;
}
return scriptResults;
} // END of function myMain()
)();
"
set scriptResults to my doJavaScript("Chrome", jsStr)
return scriptResults
--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
on doJavaScript(pBrowserName, pScriptStr) -- @JavaScript @Web @Browser @KMLib
--–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
(* VER: 1.1 2017-11-25
PURPOSE: Run JavaScript in the Designated Browser (Safari OR Chrome)
PARAMETERS:
• pBrowserName | text | Name of brower in which to run JavaScript
• IF this parameter is empty "", then the FrontMost App will
be determined and used, IF it is a browser (else an error)
• pScriptStr | text | JavaScript text to be run
RETURNS: Results of the JavaScript
AUTHOR: JMichaelTX
## REQUIRES: Either Google Chrome or Safari
----------------------------------------------------------------
*)
local jsScriptResults, frontAppName
if (pBrowserName = "") then -- get FrontMost App Name
tell application "System Events"
set pBrowserName to item 1 of (get name of processes whose frontmost is true)
end tell
end if
try
set jsScriptResults to "TBD"
if pBrowserName = "Safari" then
tell application "Safari"
set jsScriptResults to do JavaScript pScriptStr in front document
end tell
else if pBrowserName contains "Chrome" then
tell application "Google Chrome"
set jsScriptResults to execute (active tab of window 1) javascript pScriptStr
end tell
else
error ("ERROR: Invalid Broswer Name: " & pBrowserName ¬
& return & "Script MUST BE run with the FrontMost app as either \"Safari\" OR \"Chrome\"")
end if
on error e
error "Error in handler doJavaScript()" & return & return & e
end try
return jsScriptResults
end doJavaScript
--~~~~~~~~~~~~~~~~ END of Handler doJavaScript ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Questions?
Yes (and this is actually cooler than I thought, thanks!)
Here’s another field, but it’s not a field:
<textarea name="description" data-attrpath="description,basic" placeholder="Description for Facebook, Google, & Story Summary" class="settings_textarea" data-char-limit="300"></textarea>
Here’s the error message: [ERROR] Form Field with this Name NOT Found: description
So just looking at your script I can’t tell how to change it from addressing a form to addressing a text area. Maybe change input to text area? I’ll tinker a bit
OK, that worked. Now I need to click buttons.
And I have no idea what to do with this. It’s a text field, but it’s not
<body contenteditable="true" class="cke_editable cke_editable_themed cke_contents_ltr cke_show_borders" spellcheck="true"><p>TV Ratings Story</p></body>
“TV Ratings Story” is the value from the template.
And this button…
<button id="save" class="save-btn btn btn-red">Save & <br class="tiny-mobile">Continue</button>
I’m also noticing an issue similar to an issue with UI Scripting. When I run the script all the values that it sets are set and visible, but when I click the save button (which I hope to figure out how to automate as well) I get an error message that those values are not set (every form must have a headline and slug).
In GUI Scripting I could get around this by hitting tab while the field was select, and a second time.
Not sure how to make that work here.
Ed, sorry I couldn’t get back to you sooner.
To fully address your requirements, I need a good representative HTML page. Could you possibly save the web page to file, zip, and post or send to me? If it has sensitive info, you can carefully edit the file to replace with similar info.
Also, please post everything you want to do on this page. If you have this in a script, then post it. I have found that I can best help people when I know the full requirements up front.
So working on this off and on I’ve made good progress but have hit a few roadblocks.
First roadblock is that in the UI some of the actions result in progress windows opening and you have to wait until they clear to proceed. But in scripting, the buttons or fields are still available to JavaScript. If I add a delay, I have to set it for the longest possible delay but that can be upward of 30 seconds. Usually it takes less than 5.
So the first question is, in JavaScript, is there a way to tell if an element (button, field, window) exists, without clicking it on it or changing its value?
The second roadblock is sometimes the behavior of JavaScript doesn’t match the UI. I some cases where clicking a button in the UI opens a new window or tab in the browser, it doesn’t do that when done through JavaScript, although it’s clear that the button had been clicked. Similarly in some cases entering a value in a field will show the value in the field, but when the page is closed it behaves as if there is nothing there. Clicking the field in the UI and then tabbing out or using an arrow key fixes this; in other cases a when the value is set in the field it does not appear in the UI, but when page is closed it’s clear that the value was correctly set.
So I’m wondering if there’s not a JVS, AsOBJC or AppleScript method to force a browser (chrome or Safari) to refresh?
I just checked StakOverflow and there is this interesting bit about asking HTML to not keep a cache: