How to Detect Invisible Unicode Characters in AppleScript?

AppleScript
#1

A command (not from reading a file) in my AppleScript gets a string like this:

ERR-The book Gen cannot be found.

There are many signs that are not properly displayed here (in the editor preview):

When I set the clipboard to the string and convert it into unicode character values, it shows as this:

E\x00R\x00R\x00-\x00T\x00h\x00e\x00 \x00b\x00o\x00o\x00k\x00 \x00\x1c G\x00e\x00n\x00\x1d \x00c\x00a\x00n\x00n\x00o\x00t\x00 \x00b\x00e\x00 \x00f\x00o\x00u\x00n\x00d\x00.\x00

In other words, there are many \x00 signs in the string.
When I paste it to wwwDOTregextesterDOTcom I can search for it. Search for `\u0000’ will also find it.

However, when I use

if myStr contains "\\u0000" then 
--Edit: Someone pointed out to me that this is looking for string '\u0000' 
--         but not the unicode character, 
--but I don't know how to look for the unicode character \u0000

or

if myStr contains "\\x00" then

It does not work. It cannot detect either “\u0000” or “\x00”.

The result in Script Debugger:
(On the left, all results show only “E”. The string is cut off at the first “\x00”).

image

Some one pointed out to me that it is because utf-16 being parsed as utf-8. But I don’t know how to fix it.

Any help appreciated!

#2

(Sorry, new users can only include one media in the post.)

Search result in wwwDOTregextesterDOTcom:

image
image786×196 8.41 KB

#3

Search result in wwwDOTregextesterDOTcom with “\u0000”:

image

#4

I have solved it by using a handler:

on decodeCharacterHexString(theCharacters)
    copy theCharacters to {theIdentifyingCharacter, theMultiplierCharacter, theRemainderCharacter}
    set theHexList to "123456789ABCDEF"
    if theMultiplierCharacter is in "ABCDEF" then
        set theMultiplierAmount to offset of theMultiplierCharacter in theHexList
    else
        set theMultiplierAmount to theMultiplierCharacter as integer
    end if
    if theRemainderCharacter is in "ABCDEF" then
        set theRemainderAmount to offset of theRemainderCharacter in theHexList
    else
        set theRemainderAmount to theRemainderCharacter as integer
    end if
    set theASCIINumber to (theMultiplierAmount * 16) + theRemainderAmount
    return (ASCII character theASCIINumber)
end decodeCharacterHexString

I can then call it:

if myStr contains (decodeCharacterHexString("%00")) then
#5

An interesting problem:

If I add

use AppleScript version "2.4"

I will encounter this error:

image
image847×309 70 KB

Changing it to a different version won’t help.

use AppleScript version "2.7"

If I remove it, I won’t have the error and the script seems to be working just fine.

#6

Hey @martin.z,

When posting problems like this please try to post the actual test string.

Sometimes you can do that using the preformatted-text button in the forum editor, sometimes you need to provide a zipped example file.

Testing beats guessing every time.

-Chris

#7

Thank you very much, Chris.

I did post the actual string in the OP. The forum automatically filters those "\x00"s.
From (in the preview):
image
to

ERR-The book Gen cannot be found.

The only way to preserve them is to first convert them into unicode characters:

E\x00R\x00R\x00-\x00T\x00h\x00e\x00 \x00b\x00o\x00o\x00k\x00 \x00\x1c G\x00e\x00n\x00\x1d \x00c\x00a\x00n\x00n\x00o\x00t\x00 \x00b\x00e\x00 \x00f\x00o\x00u\x00n\x00d\x00.\x00

which I also posted in the OP.

Thank you very much for your tips about the Character ID elsewhere. I was able to replace the deprecated handler with a simple ascii character 0.

This is the right way:

if myStr contains character id 0 then
#8

Right.

So a zipped RTF file or AppleScript would probably be the best way to post a testable string.

-Chris

#9

That’s a good idea. Never thought of that option before. Will do next time.

Thanks!

1 Like