Need a reference for a simple real-time or same-day close fund prices


#1

Ever since Yahoo! killed their stock quotes and Numbers did as well, I’m looking for a one-shot URL to get a set of 3 mutual fund end-of-session but same-day quotes. I could trigger this from a launchd plist.

Something of the nature of http :// www. somestockURL.com/?symb=AAAA,BBBB,CCCC that would return the 3 values.

Any suggestions? The ones I have found are way too complicated.


(cobbap) #2

Numbers killed their stock quotes? In Numbers – =STOCK(“aapl”) returns today’s Apple closing stock price.


#3

I get $178.39, which is YESTERDAY’s closing price. Today’s close is $178.12 . This is expected, as it was announced in the forums and explained in the new version of Numbers.

Anyone else have a suggestion for getting 3 fund values? I have several lines of code that seemed to almost work before Yahoo! killed their stock API. Looking for a replacement. The ones I have found want to send me reams of info - all I want is the closing price.


(Ed Stockly) #4

Try this: https://www.marketwatch.com/watchlist

You need to create an account, not sure if they charge.


(Vince Angeloni) #5

You need to learn to do a little web scraping for this. MacScripter’s ‘unscripted’ section has a tutorial on one way to do it (see URL below). I posted a comment several years ago on this article that referenced using a Ruby gem web-scraper which seemed to make it a bit easier to do, at least for me. ymmv.

Macscripter HTML parsing (web scraping)


(Jim Underwood) #6

I agree with that, but I have found that using JavaScript injection in the Browser (Safari or Chrome) is easier and more effective.

There are a variety of JavaScript tools to find/select the HTML of interest.

  • There are the traditional getElementsBy
  • I prefer document.querySelector
  • and use of XPath.
  • and more.

A huge advantage is that you can use Chrome “Inspect” tool on the text/object of interest on the web page, and then use the JavaScript Console to develop/test the JavaScript statements you need to use. The Chrome JavaScript Console provides great support, including very intelligent, easy-to-use autocompletion and debugging.


(Vince Angeloni) #7

Heh - several years = 2011! Time flies…
I didn’t really ever do anything with this, except play around with one Ruby script for grins, but I think what I am suggesting in my comment on the Macscripter article is very similar to what Jim is suggesting. I also noticed that the link I included in that comment is way dead…now that seems to be a jewelry blog or something! Fortunately, I did save the tutorial from the link in a PDF showing how to use Nokogiri to web scrape, if you want to look at it: Web Scraping with Nokogiri


#8

Re: Marketwatch, they’re great but I don’t see an API on their main page. Anyone know what it is?

For example,

set fundPrices to do shell script (“curl www.marketwatch.com/?fund=TIQRX,TLIRX,QREARX”) //rough idea

or similar.


(Ed Stockly) #9

Here’s a script that Shane and Jim posted here a while back that I’ve been using for downloading web pages to scrape. This version get text from market watch that you can parse to get the closing value of the specific stock or index.

set pageURLStr to "https://www.marketwatch.com/investing/stock/aapl"

(*
  PURPOSE: Get Web Page HTML using ASObjC
              (as an alternate to curl)
              
  REF:  Script posted by @ShaneStanley to ASUL, 2017-03-31
        https://lists.apple.com/archives/applescript-users/2017/Mar/msg00421.html 
*)
use AppleScript version "2.4" -- Yosemite (10.10) or later
use framework "Foundation"
use scripting additions

--- SET URL OF WEB PAGE ---

set pageURLnsStr to current application's NSString's stringWithString:pageURLStr
set nsPageURL to current application's NSURL's URLWithString:pageURLnsStr

--- GET WEB PAGE HTML ---

set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
if nsPageHTML = missing value then error (theError's localizedDescription() as text)

-- convert to XML

set {theXMLDoc, theError} to current application's NSXMLDocument's alloc()'s initWithData:nsPageHTML options:(current application's NSXMLDocumentTidyHTML) |error|:(reference)
if theXMLDoc = missing value then error (theError's localizedDescription() as text)

--- SEARCHING & EXTRACTING INFO FROM WEB PAGE ---
--    • As an alternate to JavaScript in the Browser, use ASObjC XML methods
--    • For an example, see https://lists.apple.com/archives/applescript-users/2017/Mar/msg00421.html 

---------------------------------------------------
-- CONVERT TO NORMAL TEXT
--  • There are several options to choose from
--     SEE: Writing XML From NSXML Objects
--          https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/NSXML_Concepts/Articles/WritingXML.html
--------------------------------------------------

-- 1. SIMPLE
set htmlSimpleStr to theXMLDoc's XMLString() as text

-- 2. TIDY
set htmlTidyStr to (theXMLDoc's XMLStringWithOptions:(current application's NSXMLDocumentTidyHTML)) as text

-- (I don't see any difference between #1 and #2)

-- 3. PRETTY PRINT (produces a very readable XML/HTML output)
set htmlPPStr to (theXMLDoc's XMLStringWithOptions:(current application's NSXMLNodePrettyPrint)) as text



(Shane Stanley) #10

Or skipping the text and using XPath to get the price:

set thePrice to (theXMLDoc's nodesForXPath:"//meta[@name='price']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text

(Jim Underwood) #11

Thanks Shane for a great example.

To Those Interested in Learning about XPath:

Defining a XPath can be challenging, but in this case it is straight-forward.

All of the HTML meta tags are in the <header>. . .</header> block. So with the web page open in Chrome, press ⌘⌥I to open the Chrome Dev Tools showing the HTML elements of the page. Expand the <header> block to view its elements.

You can either visually scan for the meta tag of interest, or use Find (⌘F), in this case for “price”. It is the second occurance found:

image

While “name” and “content” are the most common key/value pairs, sometimes other attributes are used. So you may need to adjust accordingly.

For more info, see XPath in Javascript: Introduction

While there is very little on the web about using XPath with ASObjC, there is a lot about using it with JavaScript. So you may need to search for JavaScript examples, then translate them to ASObjC, if that’s your preferred tool.


(Shane Stanley) #12

That’s true, but XPath is XPath wherever you use it. I don’t use it often enough to remember, so I generally start here:

https://www.w3schools.com/xml/xpath_intro.asp

The nodesForXPath:error: method simply returns an array of matching NSXMLNodes. In this case firstObject() gets the first (and only in this case) node in the array, and stringValue() is how you extract the string from an attribute node.


#13

Anybody know if MarketWatch has an API where I can just ask for closing change values? I’m currently just typing the thing into Numbers. Any other site that has a stock API? Do I really have to screen-scrape for this? Maybe I should open up the macOS Stock widget and see what it does. My funds update at 6:15 PM, so I just go to Dashboard and see the 3 closing changes and switch to Numbers and type them in. Surely if someone writes a Stock app they don’t screen-scrape, do they?


(Ed Stockly) #14

The version that Shane wrote doesn’t screen scrape. It reads the HTML served on the site and parses that to extract your data.

If they had an API, it would presumably just read the same XML and do something similar.

If you have a list of closing prices your need, paste the stock names and I can see they work.

Here is the most simplified version. The HTML never displays in a browser.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use framework "Foundation"
use scripting additions

--- SET URL OF WEB PAGE ---
set pageURLStr to "https://www.marketwatch.com/investing/stock/aapl"

set pageURLnsStr to current application's NSString's stringWithString:pageURLStr
set nsPageURL to current application's NSURL's URLWithString:pageURLnsStr

--- GET WEB PAGE HTML ---

set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
if nsPageHTML = missing value then error (theError's localizedDescription() as text)

-- convert to XML

set {theXMLDoc, theError} to current application's NSXMLDocument's alloc()'s initWithData:nsPageHTML options:(current application's NSXMLDocumentTidyHTML) |error|:(reference)
if theXMLDoc = missing value then error (theError's localizedDescription() as text)

-- extract the price
set thePrice to (theXMLDoc's nodesForXPath:"//meta[@name='price']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text


#15

Thanks so much for clarifying. That works just fine, Ed. I got the closing price. Do you know what the name is for the change from previous close? I might as well just get that because then the subtraction is already done.

The funds are QREARX, TIQRX, and TLIRX. For today’s changes you should get -0.05, -0.02, and 0.01. For prices you should get 400.69, 20.35, and 11.70.


(Jim Underwood) #16

You can find those by viewing the web page, right-click on the item of interest, and select “inspect”.

That works in both Safari and Chrome, but I much prefer the Chrome Dev Tools.


(Ed Stockly) #17

I opened the page in Safari, then opened the app explorer in SD, navigated to the document> source

Set the view mode to “source” and found these:

                    <meta name=\"name\" content=\"Apple Inc.\">
                    <meta name=\"tickerSymbol\" content=\"AAPL\">
                    <meta name=\"chartingSymbol\" content=\"STOCK/US/XNAS/AAPL\">
                    <meta name=\"instrumentType\" content=\"Stock\">
                    <meta name=\"exchange\" content=\"U.S.: Nasdaq\">
                    <meta name=\"exchangeCountry\" content=\"US\">
                    <meta property=\"og:title\" content=\"Apple Inc.\">
                    <meta name=\"exchangeIso\" content=\"XNAS\">
                    <meta name=\"exchangeTimezone\" content=\"(UTC-05:00) Eastern Time (US &amp; Canada)\">
                    <meta name=\"price\" content=\"171.27\">
                    <meta name=\"priceChange\" content=\"-3.97\">
                    <meta name=\"priceChangePercent\" content=\"-2.27%\">
                    <meta name=\"quoteTime\" content=\"Mar 21, 2018 4:00 p.m.\">
                    <meta name=\"priceCurrency\" content=\"USD\">
                    <meta name=\"keywords\" content=\"AAPL, AAPL analyst estimates, AAPL earnings estimates, AAPL share estimates, AAPL analyst recommendations, AAPL ratings, Apple Inc. analyst estimates, Apple Inc. earnings estimates, Apple Inc. analyst recommendations, Apple Inc. analyst ratings\">
                    <meta name=\"description\" content=\"Apple Inc. stock price, stock quotes and financial overviews from MarketWatch.\">
                    <meta name=\"url\" content=\"https://www.marketwatch.com/investing/stock/aapl\">
                    <meta property=\"og:url\" content=\"https://www.marketwatch.com/investing/stock/aapl\">```

From this list I picked: Name, price, priceChange, priceChangePercent...

(Ed Stockly) #18
use AppleScript version "2.4" -- Yosemite (10.10) or later
use framework "Foundation"
use scripting additions

set urlRoot to "https://www.marketwatch.com/investing/stock/"

set companyIDs to {"QREARX", "TIQRX", "TLIRX"}

set allPrices to {}
repeat with thisCompany in companyIDs
--- SET URL OF WEB PAGE ---
	set pageURLStr to urlRoot & thisCompany as text
	set the end of allPrices to GetChangeInPrice(pageURLStr)
end repeat

set AppleScript's text item delimiters to {return & return}

return allPrices as text
on GetChangeInPrice(pageURLStr)
	set pageURLnsStr to current application's NSString's stringWithString:pageURLStr
	set nsPageURL to current application's NSURL's URLWithString:pageURLnsStr
	
	--- GET WEB PAGE HTML ---
	
	set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
	if nsPageHTML = missing value then error (theError's localizedDescription() as text)
	
	-- convert to XML
	
	set {theXMLDoc, theError} to current application's NSXMLDocument's alloc()'s initWithData:nsPageHTML options:(current application's NSXMLDocumentTidyHTML) |error|:(reference)
	if theXMLDoc = missing value then error (theError's localizedDescription() as text)
	
	-- extract the information
	--"name\" 
	--"price\" 
	--"priceChange\" 
	--"priceChangePercent\" 
	
	set thePrice to (theXMLDoc's nodesForXPath:"//meta[@name='price']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text
	set theName to (theXMLDoc's nodesForXPath:"//meta[@name='name']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text
	set thePriceChange to (theXMLDoc's nodesForXPath:"//meta[@name='priceChange']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text
	set thePriceChangePercent to (theXMLDoc's nodesForXPath:"//meta[@name='priceChangePercent']/attribute::content" |error|:(missing value))'s firstObject()'s stringValue() as text
	
	set theinfo to {theName, thePrice, thePriceChange, thePriceChangePercent}
	
	
	set AppleScript's text item delimiters to {tab}
	
	return theinfo as text
end GetChangeInPrice

#19

Thanks so much. I’ll incorporate it, now that I understand it. My confusion arose because with the Yahoo! API it seemed that it returned an array with one web call. I did have to massage it a bit. But now I have this new source of quotes. Thanks again.


(Ed Stockly) #20

How can I do this from the source property of a Safari document? If I try to read the HTLM directly the web site generates the sign-on text, even when I’m signed on. But I can see the info I need in the pages source in Safari.

So how do I convert the text from the source that contains XML to an XMLDoc, where I can use nodesForXPath?