Measuring Document Automation Efficiency
The two most common question when organizations ask when they are seeking document automation technology is “how fast is it?” and “how accurate is it?”. Many don't realize that the two are at opposition to each other most of the time. The more accurate a system the slower it is, and the faster it is the less accurate. But there is one fatal mistake in all these calculations, and that mistake is how efficiency is calculated.
Most companies who trial data capture calculate performance on the slowest step which is optical character recognition OCR. Literally companies will hit the “read” button and immediately start timing until the read is complete. This is what is considered the speed of the document automation system. This is incorrect.
There is no question that OCR can be a tremendous bottleneck in the entire entry process, but poor OCR could create an even greater bottleneck. Imagine an OCR engine that reads a document with 100 characters in 1 second as compared to an engine that reads the same 100 characters in 3 seconds. Your initial thought is that the first engine would be better, but consider that the first engine may be 60% accurate leaving 40 characters to be manually entered, and the other engine 98% accurate leaving 2 characters to be manually entered or correct. If you consider an average entry speed of 1.6 characters per second then it will take the 40 characters an additional 25 seconds to enter for a total entry time of 26 seconds for the faster engine. For the slower engine it will take an additional 1.25 seconds to enter or edit 2 wrong characters thus a total entry time of 4.25 seconds. This means that end-to-end the slower engine is 6 times faster document automation process then the slower engine.
This simple calculation illustrates the folly in assuming that the slower OCR time makes for a slower overall process. Usually focusing on accuracy has the greatest benefit for an organization unless you are improving the speed of a slower engine with hardware, or two engines are to close to see a benefit.
Most companies who trial data capture calculate performance on the slowest step which is optical character recognition OCR. Literally companies will hit the “read” button and immediately start timing until the read is complete. This is what is considered the speed of the document automation system. This is incorrect.
There is no question that OCR can be a tremendous bottleneck in the entire entry process, but poor OCR could create an even greater bottleneck. Imagine an OCR engine that reads a document with 100 characters in 1 second as compared to an engine that reads the same 100 characters in 3 seconds. Your initial thought is that the first engine would be better, but consider that the first engine may be 60% accurate leaving 40 characters to be manually entered, and the other engine 98% accurate leaving 2 characters to be manually entered or correct. If you consider an average entry speed of 1.6 characters per second then it will take the 40 characters an additional 25 seconds to enter for a total entry time of 26 seconds for the faster engine. For the slower engine it will take an additional 1.25 seconds to enter or edit 2 wrong characters thus a total entry time of 4.25 seconds. This means that end-to-end the slower engine is 6 times faster document automation process then the slower engine.
This simple calculation illustrates the folly in assuming that the slower OCR time makes for a slower overall process. Usually focusing on accuracy has the greatest benefit for an organization unless you are improving the speed of a slower engine with hardware, or two engines are to close to see a benefit.
Labels: Accuracy, OCR, performance, speed
2 Comments:
How many organizations submit images for OCR, and have staff wait around doing nothing until the OCR is completed? For these situations, I'd agree that the speed of a single instance of the OCR engine can be a factor, behind OCR accuracy.
However, if OCR is an asynchronious process, then OCR speed should be re-considered as throughput. Adding a 2nd instance of a vendor's OCR engine doubles the throughput, as long as the work can be divided between them.
Since anything except "desktop" solutions provide for multiple instances of OCR, we're now down to the additional hardware and licensing costs for the additional instances needed. With 4 and 8 core WinTel machines, hardware workhorse costs are a very small part of the equation. So, it really comes down to OCR engine licensing costs.
Paul,
First thank you for the comments!
You are right, problem is licenses cost is no small part. A typical licenses of full automated unlimited page count OCR from the top two of the four engines is not less than 10K.
Most organizations with fully automated full-page OCR have an automated business process tied to their enterprise content management system that picks up the images. Services bureaus on the other hand do complain of waiting. There verifiers have one job and one job only checking recognition results so they have to very carefully throttle throughput so as not to be paying for down time. But does not always translate to speed, often these types of companies run hundreds of thousands of pages at night so there is a queue during working hours ready to go.
Post a Comment
<< Home