KnowBrainer Speech Recognition
Decrease font size
Increase font size
Topic Title: How can I access specific links or fields in Chrome or Firefox with Advanced Scripting?
Topic Summary: Want to access controls on a website using custom commands
Created On: 04/08/2020 02:38 AM
Status: Post and Reply
Linear : Threading : Single : Branch
 04/08/2020 02:38 AM
User is offline View Users Profile Print this message


bk82
Top-Tier Member

Posts: 241
Joined: 08/04/2007

I'd like to make Advanced Scripting macros that access a specific field on a website in Chrome (and maybe Firefox). 

Like "patient search" and then it selects a search box on the site. Then I could enter my query.

Since all the fields and elements on the site have identifiers, 

The "click" commands don't work in Chrome or Firefox.

Is there a way to do this with Advanced Scripting?



-------------------------

DNS 13 Pro, Dragon Medical Practice Edition 2, German, English, French - Dragon Dictate 4, dns.comfort


Mics: Jabra Chromo NC 2400 USB. SpeechMike Premium 3510


 


 


 


www.in2comp.ch 



 04/08/2020 10:29 AM
User is offline View Users Profile Print this message

Author Icon
Edgar
Top-Tier Member

Posts: 987
Joined: 04/03/2009

There are at least three ways (using Advanced Scripting) that this MIGHT work.

The first way is what I think of as "brute force". Using some combination of mouse positioning, mouse button 
clicking and "SendKeys "{Tab}", 1" 
you can probably figure out some way of activating any given text field or button on a website. 
The most obvious solution here is to start by issuing the SendKeys command which activates the Address Bar 
(in Chrome it is "SendKeys "%d", 1"). 
Next, press the tab key continuously
 - slowly - carefully counting each keypress until the field or button is active
 (my example will assume that you needed to press 10 times). 
You are now ready to create your Advanced Script; one of these two versions will probably work:

Sub Main
   SendKeys "%d"' activate Chrome's Address Bar
   SendKeys "{Tab 10}", 1
End Sub

Sub Main
   SendKeys "%d"' activate Chrome's Address Bar
   For increment = 1 To 10
      SendKeys "{Tab}", 1
      Wait 0.1' this might not be necessary or it might not be long enough
   Next increment
End Sub

Another "brute force" method assumes that the text field or button you wish to activate/press 
is ALWAYS in the same location on your monitor 
(this assumes that your monitor resolution and Chrome's zoom level is always the same).
Sub Main
   SetMousePosition 0, XXX, XXX' where the two XXXs are actual screen-relative position values
   Wait 0.1' this might not be necessary or it might not be long enough
   ButtonClick 1, 1' you might need "ButtonClick 1" depending on your version of Dragon
End Sub

The second way only works if the actual HTML code for the site allows this. Here's an example from the 
Audacity website's front page https://forum.audacityteam.org/; 
it has a button/link labeled "Mark topics read"; note that it also has a "accesskey" which in this case is "m".
Mark topics read
Inspecting the actual HTML code for your site might reveal similar "accesskey"s. In the "Mark topics read" 
example the script would look like:
Sub Main
   SendKeys "%m", 1
End Sub

The third way involves using a third-party scripting solution (like AutoIT3 or AutoHotKey). 
These have more robust methods of interacting with your computer. 
You can write a script in AutoIT3, compile it into an executable, 
then call it from a Dragon/KnowBrainer Advanced Script:
Sub Main
   AppBringUp "U:\AutoITScripts\GreenFishUntitled.exe"
End Sub

If this is not enough information to get you going you might need to share a link to the website with us.

 



-------------------------

-Edgar
DPI 15.3, 64-bit Windows 10 Pro, OpenOffice & Office 365, Norton Security, Shure X2U XLR to USB mic adapter with Audio Technica DB135 vocal mic, Asus X299-Deluxe Prime, Intel Core i9-7940X (14 core, 4.3 GHz overclocked to 4.9 GHz), G.SKILL TridentZ Series 64GB (4 x 16GB) DDR4 3333 (PC4 26600) F4-3333C16Q-64GTZ, NVIDIA GIGABYTE GeForce GTX 1060 GV-N1060G1 GAMING-6GD REV 2.0 6GB graphics card with 3 1920x1080 monitors



 04/08/2020 10:32 AM
User is offline View Users Profile Print this message

Author Icon
Edgar
Top-Tier Member

Posts: 987
Joined: 04/03/2009

Help, I pasted the above from Notepad (with wordwrap on) and the formatting went wonky!

-------------------------

-Edgar
DPI 15.3, 64-bit Windows 10 Pro, OpenOffice & Office 365, Norton Security, Shure X2U XLR to USB mic adapter with Audio Technica DB135 vocal mic, Asus X299-Deluxe Prime, Intel Core i9-7940X (14 core, 4.3 GHz overclocked to 4.9 GHz), G.SKILL TridentZ Series 64GB (4 x 16GB) DDR4 3333 (PC4 26600) F4-3333C16Q-64GTZ, NVIDIA GIGABYTE GeForce GTX 1060 GV-N1060G1 GAMING-6GD REV 2.0 6GB graphics card with 3 1920x1080 monitors

 04/08/2020 10:59 AM
User is offline View Users Profile Print this message


Alan Cantor
Top-Tier Member

Posts: 3940
Joined: 12/08/2007

When programmatic access to web page elements is not possible (or beyond my ability), I find I can do more with Firefox than with Chrome. Firefox has a few settings that other browsers lack, and I have found ways to leverage these settings to improve the speed and reliability of my Dragon scripts.

Some of my web application scripts for Firefox are entirely Advanced Scripting or DVC. But I sometimes get better results by using Macro Express to do the heavy lifting, and triggering the scripts via Dragon. (Other third-party scripting tools can be used equally well, e.g., AutoHotkey.)

Here is a summary of scripting techniques I have used with web-based apps:


1. Search for a specific control, etc. [Very reliable, but not always possible.]

2. Search for a specific pixel colour and then move the cursor in relation to a found pixel.

3. Search for a specific mouse pointer and then move the cursor in relation to that position.

4. Search for specific text and then move the cursor, via mouse or keyboard, in relation to that position.

5. Click on a "neutral" spot on the screen that resets the tab order, and then navigate, by keyboard and/or by mouse, to the desired location.

6. Go to the address line and output a URL.

When programmatic access isn't possible, scripts that rely on the above techniques can be quite complex. I've built scripts that perform over 1000 operations to click on a control. These scripts mostly do calculations (which computers are good at), so they run in the blink of an eye and are very reliable.

 

The alternative to these techniques is to move the mouse pointer to a specific coordinate and click. But these scripts are inherently unreliable. Scripts that "point and click" might fail when the browser zoom level changes, on a different monitor, with a different screen resolution setting, etc.



Statistics
31892 users are registered to the KnowBrainer Speech Recognition forum.
There are currently 0 users logged in.
The most users ever online was 12124 on 09/09/2020 at 04:59 AM.
There are currently 590 guests browsing this forum, which makes a total of 590 users using this forum.

FuseTalk Standard Edition v4.0 - © 1999-2021 FuseTalk™ Inc. All rights reserved.