I have been a programmer for 25 years. Tying the macros to mouse positions just doesn't sit well for me, for reasons that I hope others understand without explanation. Is it possible (without twisting oneself or one's code in knots) to identify the location of the mouse click in Internet Explorer using the DOM element instead of by the mouse position?

I am not certain, but here is an example I would start from:

http://www.knowbrainer.com/pubforum/index.cfm?page=viewForumTopic&topicId=9938&pageNo=1#77D150FD-FC1E-436D-1A0C2880BAFB199Bthread49861

Rüdiger

 

-------------------------

Well, it's past the point where we can make any changes in the code, but we can still make changes to the Easter Egg!

Instead of using the DOM API to find clickable points of components within Internet Explorer you should use MSAA (Microsoft Active Accessibility) or alternatively UiAutomation.  Both these APIs will give you clickable points of every single component within Internet Explorer both for the HTML page and otherwise (e.g. menus, address bar components etc).  The newer API is UIAutomation and it can be used with managed or unmanaged code, just Google it and there are lots of examples and even a Microsoft forum on these Apis alone.

Lindsay

-------------------------

www.pcbyvoice.com
www.pcbyvoice.co.uk

I would deeply appreciate seeing some sample code demonstrating the use of UIAutomation for Internet Explorer. My organisation uses a Internet Explorer interface for our mainframe system. It would be incredibly helpful to be able to programmatically get a hold of any element on the page whether it be a link a button or whatever. The developers have been so clever in their code that Dragon simply can't find some elements.

-------------------------

Steve Morris

 

I posted a project on vworker.com for somebody to write a Visual Basic application that clicks the friends request icon on Facebook. I'm offering $25. So far nobody has taken the project. I guess this is an indication of how hard this task is.

Microsoft has a small application that allows you to move the mouse around on your screen and identify objects that are addressable through UI automation. (Accessible Events Watcher) it is interesting that the Facebook friends request icon is seemingly invisible to this application.

There is a whole Microsoft forum dedicated to UI automation and Active Accessibility (MSAA).  There are some great example projects in C# and Visual Basic.

http://social.msdn.microsoft.com/Forums/en/windowsaccessibilityandautomation/threads

Lindsay

-------------------------

www.pcbyvoice.com
www.pcbyvoice.co.uk

it's a fairly lengthy procedure to add all the supporting code.  You are probably familiar with ShowNumbers that comes with Windows Speech Recognition.  I have developed a similar application for DNS (there is already one available also for DNS called ShowNumbersPlus from Max Roth which uses active accessibility) which uses UI Automation and basically it fits a coloured rectangle around every clickable component on the web page.  See the screenshot below, the coloured squares fade in and out so you can see the clickable item behind.  Basically you just issue the voice command e.g. "left click 64", "double-click 19" etc.  If I could see the friends request icon I could show you an example :-)

However forget about doing it in advanced scripting or a Visual Basic script, you need to use .NET or COM and although advanced scripting allows access to COM components to a degree I wouldn't even attempt it.

Lindsay






-------------------------

www.pcbyvoice.com
www.pcbyvoice.co.uk

To the right of the word Facebook there are three little icons. These are for friend requests, messages, and updates. You have to be signed in to Facebook to see this.

It's just a standard hyperlink control and to click on it I would just say "left click 21", which I have just done and it works fine.  My application below is not downloadable but if you want to see UI Automation in action then just load up Windows speech recognition and use the command "Show Numbers".  If you want to see Microsoft Active Accessibility (MSAA) in action then download ShowNumbersPlus for DNS.  There are stacks of information on these Apis, including examples, within MSDN and the Windows SDK.  It's just a matter of doing some reading and digging, there are plenty of code examples out there.

Edit: you may be thinking to yourself "why can't I just give you a code example in Visual Basic", it's not that simple as it's fairly complicated involving Dragon ActiveX programming, COM Interop to use Dragon API/UI Automation with Visual Basic .NET/C# ( although there is a managed API as well for UIA but it is not as quick), multithreading, WPF programming for the Windows etc etc.  Thousands of lines of code just to do that below.
Lindsay

-------------------------

www.pcbyvoice.com
www.pcbyvoice.co.uk

The tool you want to use to look at these components is called Inspect (.exe) and it comes with the Windows SDK, there is nothing invisible about that control and as I said above its just a standard hyperlink.

Lindsay

-------------------------

www.pcbyvoice.com
www.pcbyvoice.co.uk