Hi all,

Several days ago Rüdiger asked me if I wouldn't help him in the hardware performance test by controlling all the variables except for hardware using his performance testing utility.

The way that this was done was to create a digital voice recorder profile and train it using Rüdiger's audio training DS2 file along with using five audio files created by him for test purposes. By doing this the way that it was done, everything was 100% identical except for the differences in hardware configuration. That is, both systems were using the same identical profile and the test audio files.

To keep the explanation of the results simple, suffice it to say that the only variable that varied in any way was the hardware.

Rüdiger's system:

Testing Performance Tool, Copyright R.Wilke 2010

Processor:
IntelR CoreTM2 Duo CPU     T9400  @ 2.53GHz
Architecture: 64-bit
L2-Cache Size: 6144 KB
L3-Cache Size: 0 KB
Physical Memory: 4095 MB

Operating System:
Microsoft Windows 7 Ultimate N  64-Bit
 SP1

Chuck's system:

Testing Performance Tool, Copyright R.Wilke 2010

Processor:
IntelR CoreTM i7 CPU       X 980  @ 3.33GHz
Architecture: 64-bit
L2-Cache Size: 1536 KB
L3-Cache Size: 12288 KB
Physical Memory: 12279 MB

Operating System:
Microsoft Windows 7 Ultimate  64-bit
 SP1

Results:

Speed vs. Accuracy:			50%
file 1	CR	0.36	0.25	0.25
	RW	0.78	0.42	0.40
file 2	CR	0.22	0.22	0.21
	RW	0.56	0.35	0.35
file 3	CR	0.20	0.21	0.19
	RW	0.59	0.33	0.30
file 4	CR	0.22	0.21	0.21
	RW	0.46	0.32	0.38
file 5	CR	0.23	0.21	0.21
	RW	0.48	0.36	0.35
Speed vs. Accuracy:			100%
file 1	CR	0.48	0.40	0.39
	RW	0.95	0.67	0.64
file 2	CR	0.34	0.35	0.31
	RW	0.66	0.57	0.51
file 3	CR	0.31	0.28	0.27
	RW	0.61	0.49	0.47
file 4	CR	0.32	0.30	0.29
	RW	0.59	0.52	0.50
file 5	CR	0.34	0.36	0.33
	RW	0.63	0.59	0.55

To interpret these results consider that the smaller the number, the faster the performance of the transcription process. That is, 0.33 represents faster performance vs. 0.55.

Rüdiger may wish to provide additional information or further explanation. However, I tried to keep this as simple as possible so that everybody understands how hardware effects performance with DNS.

Also, I wish to thank Rüdiger for providing both the testing utility and the test files. Although the results were a joint effort, Rüdiger did most of the grunt work making it easy to conduct the test. Therefore, all kudos to be directed to him.

Chuck Runquist
Technical Project Manager
VoiceTeach LLC
Home of VoicePower®: Simply powerful, powerfully simple


-------------------------

VoiceComputer: the only global speech interface.

The views, thoughts and opinions expressed in this post are my own and do not reflect those of VoiceTeach LLC.

Chuck Runquist
VoiceComputer technical support

Rudiger,

Ingenious experiment--Kudos!!! Chuck, given that you have about 7 different systems, you should try it now on your lowest performing one, and one with a Sandy Bridge chip and post the results.

Rudiger, it occurrs to me that we frequently see people posting here asking advice about which of two or more systems would be most recommended for use with Dragon. Your tool would be an excellent commercial product for evaluating this objectively, rather that making a best-educated (or ignorant) guess. Also, it would give objective evidence as to comparison of systems to get the best "bang for the buck" to see if differences are real, substantial, or just mostly hype. You should make this into a commercial product for these purposes. Although designed with DNS in mind, your tool gives more fundamental information that would be applicable to anyone, even those not intending to use DNS (but potentially heavy users in other regards), in terms of an objective evaluation tool guiding informed purchase choices of computers.

Great work. Great tool.

MDH

 

-------------------------

Mark,

thanks for the compliments. I think you are basically correct in that doing it this way could even meet industrial standards somehow, however turning it into a commercial product would involve things I wouldn't necessarily like to face. The underlying theoretical concepts are not new, and I don't own the technology being used. What I did though was putting the pieces together and make them work.

Just as an aside, the idea occured to me just recently when thinking of ways to do an immediate comparison between version 11.0 and 11.5 after my first intuitive impression was that there is an improvement in performance in 11.5. In short, there isn't any, so it was merely some sort of placebo effect. Upon further investigating it, I discovered that mrec.dll (the modular recognizer) which is the core modul in here is the same version number and size in version 11.0 and in 11.5.

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

Yeah, like for a car, or plane, or computer--all of which proved useful commercially.

Did you compare DNS v. 10.1 versus 11.5? (especially since v11 is a significant re-write).

MDH

 

 

-------------------------

No I didn't, much closer I compared 11.0 versus 11.5. I did this by creating two identical virtual machines (the second one being a copy of the first), stored them in the same location (well next to each other), and the only difference they had was 11.0 installed on one and 11.5 on the other. I then created a digital recorder profile on each of them, using the same audio recordings on both, and ran the tests.

Like I said, I did this when I first thought that 11.5 is a little snappier than 11.0 in free form dictation, but that was just because I was willing it to be obviously.

However, audio recordings never lie.

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

Chuck,

thanks once again for taking the time to participate in the testings, thus helping to provide evidence as to the usability of the underlying methods and materials, but also for being so kind to post the results upon my request. Much appreciated.

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

Thanks so much, Rudiger and Chuck, for these tests and the significant improvement demonstrated with better equipment.  This makes it easier for me to justify my next purchase of hardware when I upgrade from XP to Win 7.

So how would you all break down and attribute the performance improvement between

(1) the L3 cache being 0 for Rudiger and 12,228 KB for Chuck and

(2) Chuck's memory being three times what Rudiger has?

Inquiring minds want to know.  

 

-------------------------

DNS DPI Pro 15.3, Win10, i7-3770K @ 3.7GHz, 16 Gb, 465 Gb SSD Boot drive, 3-in-1 TableMike - KnowBrainer Pro 2017 - Thanks so much, Lunis.

Since my processor is a dual core, I don't have any L3 cache because they don't have it. All I have is L2 cache for that matter. But you cannot immediately compare the amounts of L2 cache and L3 cache with each other, because the concept is different. However, on a rough estimation (as I could prove by running relevant tests recently) one might say that half the amount of L3 cache equates for double the amount of L2 cache, more or less. In other words, if you have an i5 processor with 3 MB of L3 cache, it will perform about as fast as my system with 6 MB of L2 cache. Therefore, getting such a system now wouldn't mean any improvement over what I currently have.

As far as the amount of memory installed, it doesn't have so much influence, if any, on short term performance, given that you have a certain minimum (2 GB in XP, and 4 GB in Windows 7), when looking at processing single utterances, but it will pay when running many applications with lots of RAM use along with each other.

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

To clarify--only as related to his computer! 

MDH

-------------------------

Rüdiger/Chuck

from a very personal point of view I would love to see this tool developed further and used as some sort of standard although I appreciate what you say Rüdiger in terms of your possible lack of interest in going through all the necessary formalities etc.

For my sins, and as a quadriplegic over many years, I get asked to do expert witnessing fairly regularly where I have to recommend assistive technology equipment (in which DNS and supporting package features prominently) for the disabled and other medical conditions, and also to justify such suggested equipment and software.  It is usually a case of persuading a judge or panel why an insurance company should be forced to pay for a top spec machine or heading in that direction.  I continually have to persuade the authorities that a more powerful PC is of a major benefit in terms of speed and speed/accuracy.  To have such an official benchmarking tool adopted by even Nuance, and why not, would save me a lot of explanation and justification although even in its current form it would be more than useful to prove a point.

So thank you and can you adapt it so I can justify why an RTA victim should have a 12 bedroom luxury house because quite simply DNS performs much better in such an environment.

Lindsay
www.pcbyvoice.com

-------------------------

Lindsay,

interesting idea. I will see what I can do. I guess once you start considering what impact living in a luxury house may have on Dragon's performance, you shouldn't also neglect having a pool like an ocean in the garden as well as being transported in a modified Ferrari.

Anyway, back to being serious, I added the results I received from you to the list. Unfortunately, due to technical problems, you had to cancel out of running the tests very early (was it to do with the electricity in your house once more not working?), but also in all its briefness it clearly demonstrates the processor being used, particularly the processor cache being available, being the determining factor as far as performance on a very systematical scale.

Please keep in mind that Chuck's system has 1.5 MB of L2 Cache and 12 MB of L3 Cache. Your system, referred to as LA in the list, has 1 MB of L2 cache and 8 MB of L3 cache. Thinking of it, this is almost pure algebra. Have a look at it, please:

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

I would like to know whether there was an actual perceived difference in the responsiveness of both systems and whether it really made that much differnce to the user. Does .21 vs .36 really make a differrence in the real world experience?

-------------------------

Dr. Fred,

in order to answer your question in some reasonable manner, I would need an entire chapter to explain, but for the sake of briefness, I will start by referring to some keywords being used in speech recognition research and some articles you or anyone might look up and start from there.

The numbers you are seeing in the list represent the so-called Real Time Factor (which is a technical term in research, cf. http://en.wikipedia.org/wiki/Real_time_factor), what my program does is compute them automatically retrieving the relevant processing data coming straight from the Dragon engine and counting the times retrieved from the operating system. Therefore they are pretty accurate. Since the material being used is recorded speech (audio files) the results are absolutely replicable, and therefore valid.

Now please have a look at the definition of the Real Time Factor:

What this means is, that for instance given the examples that we have tested which were recorded utterances each of them about 10 seconds long (or entire sentences of 20 words on the average for that matter), Chuck's system can handle them in about two third of the time (on the average) that my system will need to do this.

Just to demonstrate, this is a typical result for Chuck's system (file 2, run 2, slider 50%) :

Utterance Length:     9090 msecs
Overall processing time:     2085 msecs
Ratio processing time vs. utterance length:  0.22

Unfortunately I deleted the details in here for my testings (the log files that are created by pressing a button in my program), but if the ratio 0.35 in this run for me according to the list, it must have looked something like this:

Utterance Length: 9090 msecs
Overall processing time: 3250 msecs
Ratio processing time vs. utterance length: 0.35

Now this is just about a third less, or two thirds of it, or however you may like looking at it. You may also argue that this is just a difference of about 1.2 secondes or so, but consider that this is in relation to processing about 10 seconds of speech.

And quite frankly, to me, it will make a difference.

Rüdiger

 

-------------------------

No need to buy if all you want to do is try ...

DragonCapture KB Download (Latest)
DragonCapture Homepage

I have already shared this idea with Rudiger, but I present it here as a possible method of using Rudiger's ingenious Performance Factor Tool to compare computers for use with speech recognition. I will present an example using the data that has been posted with Rudiger's and Chuck's results. We first need to have a "gold standard" with which to compare. This is determined by which computer registers the lowest cumulative best performance factor at the 50% setting of the accuracy versus speed slider. For now, we will assume this to be Chuck's computer, and we will compare Rudiger's.

 Using "best" results:

Chuck's best results added for all 5 files at 50% setting = 1.07, then divided by 5 = 0.21

Rudiger's best results added for all 5 files at 50% setting = 1.72, then divided by 5 = 0.34

Difference = 0.34 minus 0.21 = 0.13

Then 0.13 divided by 0.21 X 100 = 61.9%. This means Rudiger's computer was 61.9% slower than Chuck's at 50% setting.

Then:

Chuck's best results added for all 5 files at 100% setting = 1.59, then divided by 5 = 0.32.

Rudiger's best results added for all 5 files at 100% setting = 2.67, then divided by 5 = 0.53.

Difference = 0.53 minus 0.32 = 0.19

Then 0.19 divided by 0.32 X 100 = 59.4%. This means Rudiger's computer was 59.4% slower than Chuck's at the 100% setting.

Regarding degradation of the performance factor at 100% setting versus the 50% setting:

Chuck's "best" dropped from 0.21 to 0.32 going from the 50% to 100% setting.

Difference = 0.32 minus 0.21 = 0.11

Then 0.11 divided by 0.21 X 100 = 52.4% degradation.

Rudiger's "best" dropped from 0.34 to 0.53 going from the 50% to 100% setting.

Difference = 0.53 minus 0.34 = 0.19

Then 0.19 divided by 0.34 X 100 = 55.9% degradation.

Note that Rudiger's performance factor degradation was surprisingly, on a percentage basis, not a great deal more than Chuck's. (However, Chuck's performance factor at the 100% setting of 0.32 is about the same as Rudiger's (0.34) at the 50% setting.

So, Rudiger's degradation of the performance factor was more than Chuck's, with the difference being 55.9% minus 52.4% = 3.5%

Then according to the comparison method that I suggest:

Computer comparison = Difference from Gold Standard at 50% + Difference in Degradation from Gold Standard

And the lower the number, meaning the less difference, the better. (We assume for now for this comparison that Chuck's computer is the Gold Standard, but this will ultimately be determined by the best performance factor computer at the 50% setting.)

So, applying here:

Rudiger computer comparison:

Rudiger computer score = 61.9 + 3.5 = 65.4

I welcome any comments, including criticisms (be nice , corrections, suggestions. Thanks.

 

MDH

-------------------------