How to Detect Whether a PDF has been OCR'd


(Christopher Stone) #1

Hey Folks,

Is there a quick (speedy) and easy way to detect whether or not a PDF has been OCR’d?

The only way I know of presently is to try to extract the text from it, and this can be a trifle slow with large files.

The only way I now of to mitigate that is to extract a page from the PDF and then extract the text from it.

I have some command line PDF tools that can test for things like the fonts in a PDF, so I’m wondering if AppleScriptObjC has access to this sort of thing.

TIA.

-Chris


(Christopher Stone) #2

Here are links to the executables and a small example.

http://www.xpdfreader.com/download.html

I’ve used Xpdf tools since about 2012 – mostly pdftotext which has a -layout switch that attempts to preserve the document layout unlike other text extraction methods.

I really like this critter, but it can have problems with accented characters – and I’d like to find a solution that didn’t have that problem.

I realize there’s a means to extract text using AppleScriptObjC, but it does nothing to preserve the layout.

Sometimes that’s not a problem, but at other times the layout is key to being able to successfully parse the text.

-Chris


(Jean Christophe Helary) #3

Not really helping, but OCRd PDFs are not the only PDFs that don’t let you extract text. As Illustrator specialists will tell you here, as soon as you create the outline of a document, the resulting PDF won’t let you extract text because text has just become an image.


(Jim Underwood) #4

So, Chris provided a link to the CLI tool that will do the job.

But I’m still interested in know if ASObjC can detect whether a PDF has been OCR’d quickly and efficiently, other than getting the text of the PDF.

So can ASObjC get the font list, or other property to do this job?

Why? I’d rather not have to use a 3rd party tool that end-users will have to download and install properly.


(Christopher Stone) #5

Hey JM,

Here’s the clunky method. It extracts page one of the PDF to a temp file and then attempts to extract text from it.

On my system it takes 1 minute 15 seconds to scan 502 PDF files with an aggregate file size of 962.3 MB.

That’s not super fast, but it’s not horribly slow either.

I haven’t compared this to the command line utilities yet, but I probably will.

-Chris

----------------------------------------------------------------
# Auth: Christopher Stone
# dCre: 2018/12/06 17:16
# dMod: 2018/12/06 18:39
# Appl: AppleScriptObjC, Finder
# Task: Determine if Selected PDF Files have been OCR'd.
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @ASObjC, @Finder, @Determine, @Selected, @PDF, @Files, @OCR'd
----------------------------------------------------------------
use AppleScript version "2.3.1"
use scripting additions
use framework "Foundation"
use framework "Quartz" -- for PDF stuff
----------------------------------------------------------------

set tempFilePath to (POSIX path of (path to temporary items folder from user domain)) & "temp.pdf"

# Selected files in the Finder are the target.
tell application "Finder"
   set finderSelectionList to selection as alias list
   if length of finderSelectionList = 0 then error "No files were selected in the Finder!"
end tell

# Tranform alias list to POSIX Paths.
repeat with i in finderSelectionList
   set contents of i to POSIX path of (contents of i)
end repeat

set nonOcrList to {}

repeat with pdfFilePath in finderSelectionList
   # Extract page 1 of the selected PDF file to a temp file.
   set pdfTempFilePath to (my extractPages:1 thruTo:1 ofPDFDocAt:pdfFilePath usingTempFile:tempFilePath)
   set pdfText to (its pdf2Text:pdfTempFilePath)
   if pdfText = "" then set end of nonOcrList to contents of pdfFilePath
end repeat

nonOcrList

----------------------------------------------------------------
--» HANDLERS
----------------------------------------------------------------
on extractPages:firstPage thruTo:lastPage ofPDFDocAt:posixPath usingTempFile:tempFilePath
   --  make URL of the first PDF
   set inNSURL to current application's class "NSURL"'s fileURLWithPath:posixPath
   -- make PDF document from the URL
   set theDoc to current application's PDFDocument's alloc()'s initWithURL:inNSURL
   -- count the pages
   set pageCount to theDoc's pageCount()
   -- delete pages at end
   if lastPage < pageCount then
      repeat with i from pageCount to (lastPage + 1) by -1
         (theDoc's removePageAtIndex:(i - 1)) -- zero-based indexes
      end repeat
   end if
   -- delete pages at start
   if firstPage > 1 then
      repeat with i from (firstPage - 1) to 1 by -1
         (theDoc's removePageAtIndex:(i - 1)) -- zero-based indexes
      end repeat
   end if
   
   # Write to temporary file.
   theDoc's writeToFile:tempFilePath
   
   return tempFilePath
   
end extractPages:thruTo:ofPDFDocAt:usingTempFile:
----------------------------------------------------------------
on pdf2Text:thePath
   set theText to current application's NSMutableString's |string|()
   set anNSURL to current application's |NSURL|'s fileURLWithPath:thePath
   set theDoc to current application's PDFDocument's alloc()'s initWithURL:anNSURL
   set theCount to theDoc's pageCount() as integer
   
   repeat with i from 1 to theCount
      set thePage to (theDoc's pageAtIndex:(i - 1))
      (theText's appendString:(thePage's |string|()))
   end repeat
   
   return theText as text
   
end pdf2Text:
----------------------------------------------------------------

(Shane Stanley) #6

Why not just check the first page in situ? Something like:

if (theDoc's pageAtIndex:0)'s |string|()'s |length|() < 1 then
...

(Shane Stanley) #7

Not directly. You could do something like get the contents of pages as NSAttributedStrings using attributedString rather than string, and then check the fonts used.


(Christopher Stone) #8

Hey Shane,

Because I didn’t know how.  :sunglasses:

But that’s certainly more efficient than what I was doing.

What if the given page has no text? You can still get at least one character (\uFFFC), so length of string can be problematic.

Probably if the string is longer than 3-4 characters it’s safe, but I haven’t tested nearly enough to know for certain.

I suppose the only bombproof way would be to page through the document testing for a string of word characters using regex and exit if you find same.

Is there a better test available than regex? One that can test for a string in any language?

-Chris


(Christopher Stone) #9

Hey Folks,

I’m not entirely satisfied with the text-length test in the appended script, but when run it finds all the non-ocr’d pdfs in my 500 file test set in only 25 seconds (less than half the time of first script).

My first script had a number of false negatives due to \uFFFC characters showing up in NON ocr’d files.

I’d still like a better test than text-length, but at least this script is much less clumsy than the first one.

Thanks Shane!

-Chris

----------------------------------------------------------------
# Auth: Christopher Stone
# dCre: 2018/12/06 17:16
# dMod: 2018/12/07 02:00
# Appl: AppleScriptObjC, Finder
# Task: Determine if Selected PDF Files have been OCR'd.
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @ASObjC, @Finder, @Determine, @Selected, @PDF, @Files, @OCR'd
# Vers: 1.01
----------------------------------------------------------------
use AppleScript version "2.4"
use framework "Foundation"
use framework "Quartz"
use scripting additions
----------------------------------------------------------------

tell application "Finder"
    set finderSelectionList to selection as alias list
    if length of finderSelectionList = 0 then error "No files were selected in the Finder!"
end tell
repeat with theFile in finderSelectionList
    set (contents of theFile) to POSIX path of (contents of theFile)
end repeat

set nonOcrList to {}

repeat with thePath in finderSelectionList
    if (its pdfHasBeenOcrd:thePath) = false then
        set end of nonOcrList to contents of thePath
    end if
end repeat

return nonOcrList

----------------------------------------------------------------
--» HANDLERS
----------------------------------------------------------------
on pdfHasBeenOcrd:thePath
    set theText to current application's NSMutableString's |string|()
    set anNSURL to current application's |NSURL|'s fileURLWithPath:thePath
    set theDoc to current application's PDFDocument's alloc()'s initWithURL:anNSURL
    set theCount to theDoc's pageCount() as integer
    
    set OcrFlag to false
    
    repeat with i from 1 to theCount
        set thePage to (theDoc's pageAtIndex:(i - 1))'s |string|() as text
        
        # Test for Text Content (I'm not satisfied with this yet). •••••
        if (length of thePage) > 20 then
            set OcrFlag to true
            exit repeat
        end if
        
    end repeat
    
    return OcrFlag
    
end pdfHasBeenOcrd:
----------------------------------------------------------------

(Shane Stanley) #10

Shouldn’t that be < rather than >?


(Christopher Stone) #11

Nyet.

I’m setting the threshold to greater than 20 characters.

For now.

I expect I can set it to less, but this is working so far.

-Chris


(Shane Stanley) #12

But you’re setting OcrFlag to true in that case — shouldn’t it be the other way around?


(Christopher Stone) #13

No.

The PDF is OCR’d if there are more than 20 characters.

-Chris


(Jonas Whale) #14

In my case, I had to determine if some scanned documents had been “ocr’d”.
I found that every page that only contains the scanned image has a length of 2.
So the length should be the double of the page count (-1 because of zero indexation in objC).

Here is my script:

use AppleScript version "2.4"
use framework "Foundation"
use framework "AppKit"
use framework "Quartz"
use scripting additions

tell application "Finder" to set finderSel to selection as alias list
if finderSel = {} then return beep

set withoutText to {}
repeat with theFile in finderSel
	set theURL to (current application's NSURL's fileURLWithPath:(POSIX path of theFile))
	if (theURL's pathExtension()'s isEqualToString:"pdf") then
		set theDoc to (current application's PDFDocument's alloc()'s initWithURL:theURL)
		set textLength to theDoc's |string|()'s |length|()
		set pageCount to theDoc's pageCount()
		if (pageCount * 2) - 1 = textLength then set end of withoutText to contents of theURL as text
	end if
end repeat

return withoutText

(Jim Underwood) #15

@ionah, thanks for sharing.

Unfortunately, your script takes 10X as long a Chris’ (@ccstone) last script.
Script Geek Results: 2.374 vs 0.239 min (avg of 5 runs) for 15 files

Well, that’s too bad. I’m thinking that would take longer than Chris’ method – would you agree?


(Shane Stanley) #16

Considerably. It’s a trade-off of (potential) accuracy vs speed.


(Jim Underwood) #17

Chris, thanks for a great script, but you’ve got a slow Mac. :wink:
That’s 0.05 sec/PDF

When I run your script, but also using Shane’s MetaLib lib to search a folder recursively that contains PDFs and other stuff, it took only 0.012 sec/PDF:

Total Time: 2.28 sec
Total PDFs: 184
= 0.012 sec/PDF

PDFs that Need OCR: 23

Here’s the script I used:

property ptyScriptName : "Detect Whether a PDF has been OCRd -- Chris"
property ptyScriptVer : "2.0"
property ptyScriptDate : "2018-12-07"
property ptyScriptAuthor : "Christopher Stone" -- mod by JMichaelTX

(*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PURPOSE:
  • Detect Whether a PDF has been OCRd -- Chris
  
RETURNS:  List of PDFs (Posix Path) that Need to be OCR'd

REQUIRED:
  1.  macOS 10.11.6+

TAGS:  @Lang.AS @Lang.ASObjC @CAT.PDF @type.Example @Auth.Chis

REF:  The following were used in some way in the writing of this script.

  1.  2018-12-07, Christopher Stone, Microsoft Outlook
      Re: How to Detect Whether a PDF has been OCR'd - AppleScript - Late Night Software Ltd.
      geode@thestoneforge.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*)
----------------------------------------------------------------
# Auth: Christopher Stone with Mod by JMichaelTX
# dCre: 2018/12/06 17:16
# dMod: 2018/12/07 02:00
# Appl: AppleScriptObjC, Finder
# Task: Determine if Selected PDF Files have been OCR'd.
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @ASObjC, @Finder, @Determine, @Selected, @PDF, @Files, @OCR'd
# Vers: 1.01
----------------------------------------------------------------
use AppleScript version "2.4"
use framework "Foundation"
use framework "Quartz"
use scripting additions

use script "Metadata Lib"
----------------------------------------------------------------

###set theFolder to choose folder with prompt "Choose Folder for Spotlight Search" ###  path to desktop

set theFolder to "/Users/Shared/Dropbox/Mac Only/Scan Inbox"

--- Search Folder & Sub-Folders to Find All PDF Files ---

set pdfFileList to perform search in folders {theFolder} predicate string "kMDItemContentType == %@" search arguments {"com.adobe.pdf"}

set numPDFs to count of pdfFileList

set nonOcrList to {}

repeat with thePath in pdfFileList
  set pdfOCRdBool to (its pdfHasBeenOcrd:thePath) ### Use Chris' handler
  ##  set pdfOCRdBool to my hasPdfBeenOCRd2(thePath) ### Use MY Shell Script handler
  
  if (pdfOCRdBool) = false then
    set end of nonOcrList to contents of thePath
  end if
end repeat

set numPDFsToOCR to count of nonOcrList

return nonOcrList


----------------------------------------------------------------
--» HANDLERS
----------------------------------------------------------------

on hasPdfBeenOCRd2(pPosixPath)
  
  set PDFPath to pPosixPath
  set cmdStr to "/usr/local/bin/pdffonts " & quoted form of PDFPath
  set fontListStr to do shell script cmdStr
  set numLines to count (paragraphs of fontListStr)
  
  if (numLines > 2) then
    set OCRdBool to true
  else
    set OCRdBool to false
  end if
  
  return OCRdBool
  
end hasPdfBeenOCRd2

on pdfHasBeenOcrd:thePath
  set theText to current application's NSMutableString's |string|()
  set anNSURL to current application's |NSURL|'s fileURLWithPath:thePath
  set theDoc to current application's PDFDocument's alloc()'s initWithURL:anNSURL
  set theCount to theDoc's pageCount() as integer
  
  set OcrFlag to false
  
  repeat with i from 1 to theCount
    set thePage to (theDoc's pageAtIndex:(i - 1))'s |string|() as text
    
    # Test for Text Content (I'm not satisfied with this yet). •••••
    if (length of thePage) > 20 then
      set OcrFlag to true
      exit repeat
    end if
    
  end repeat
  
  return OcrFlag
  
end pdfHasBeenOcrd:
----------------------------------------------------------------

(Jonas Whale) #18

@JMichaelTX, I think you’re misunderstanding my point.

I tried to bring solution to @ccstone’s concern:

With my script, if your PDFs are scanned docs with a unique area per page, you are sure about the result.
With this advantage, speed is not relevant…


#19

I find DEVONthink a helpful context for this kind of thing:

tell application "DEVONthink Pro"
    kind of item 1 of (selection as list)
    --> 'PDF+Text'  or just 'PDF'
end tell

(Ed Stockly) #20

I once had to do something very similar. The source of the text in the PDF wasn’t OCR, but we had to look at PDFs and tell if it had been created from an empty template or if it had been created from a partially completed or fully completed document, the difference being the number and lengths of text blocks.

So this is what the pure appleScript quick and dirty temporary solution that we started using on Mac OS 8, and kept using until a few years ago that never once gave us a bad result.

I actually didn’t have a copy of the script here, but it was pretty trivial to rewrite it and test it using SD.
You can drag and drop pdf files into an SD window and execute the open handler, or open a PDF file as text in TextWrangle and execute the run handler.

The first part of the result is the length of text runs. For our purposes if there were less than 20 text blocks it was generated from a blank template; if there were hundreds of text blocks it was generated from a document in progress; if any of the text block lengths were over 200 it was an unfinished document.

--get text from TextWrangler (or replace with BBEdit or any text editor that can read a PDF file

tell application "TextWrangler" to set myText to text of window 1
set PDFTextInfo to my GetPDFTexts(myText)
display dialog PDFTextInfo as text
--Or Drag&Drop pdf files
on open pdfFiles
   set allPDFInfo to {}
   repeat with thisFile in pdfFiles
      set pdfText to read thisFile
      
      set AppleScript's text item delimiters to {":"}
      set fileName to the last text item of (thisFile as text)
      set PDFTextInfo to my GetPDFTexts(pdfText)
      
      set PDFTextInfo to {fileName} & PDFTextInfo
      set the end of allPDFInfo to PDFTextInfo as text
   end repeat
   display dialog allPDFInfo as text
end open

on GetPDFTexts(myText)
   local allTextLengths, foundTypeBlocks, errorsFound, myText
   set myText to paragraphs of myText
   
   set AppleScript's text item delimiters to {return}
   
   set myText to myText as text
   set AppleScript's text item delimiters to {return & "<< /Type /Font /Subtype /"}
      set typeObjects to the the rest of text items of myText
   set allTextLengths to {"Text Lengths: "}
   set foundTypeBlocks to {"", "Type Blocks:"}
   set errorsFound to {"", "Errors:"}
   repeat with thisTypeBlock in typeObjects
      set AppleScript's text item delimiters to {" >>" & return}
      set thisTypeBlock to text item 1 of thisTypeBlock
            try

      set AppleScript's text item delimiters to {"/FirstChar ", "/LastChar"}
      set charRangeItems to text items of thisTypeBlock
      set startChar to item 2 of charRangeItems
      set endChar to word 1 of item 3 of charRangeItems
         set textLength to (endChar as number) - (startChar as number)
         set the end of allTextLengths to textLength
         
      on error errMsg number errNum
         set errorString to ("Error Number: " & errNum as text) & return & return & "Error Message: " & return & return & "\"" & errMsg & "\""
         
         set the end of errorsFound to errorString & return & tab & thisTypeBlock
      end try
      
      set the end of foundTypeBlocks to thisTypeBlock
   end repeat
   set AppleScript's text item delimiters to {return}

   return {allTextLengths, foundTypeBlocks, errorsFound} as text
end GetPDFTexts
--made a small edit, moving the "try" up a few lines