Comparison of Character Counting Programs

How character counting programs work
There are a number of programs designed to count characters in documents, whether they are MSWord doc files, text files, or other filetypes. Some word processors have a simple character counting function built in. Other programs are designed specifically to count characters, words, or phrases. However, the results seem to vary with almost every program. We ran tests using some of the most popular character counting programs to determine which programs were the most accurate. In determining their accuracy, we needed to examine exactly how characters were stored in the common file formats. This involved both visible characters, and hidden characters and control codes.

Sample files to determine what characters are counted
To help us analyze these character counting programs, we created three test files to use with our test programs. These files were each designed to test different abilities in these character counting software programs. The files are available for download by clicking on the filename in the list below. The files are:

rawtext.txt
This is a plain text file that has been saved in UTF-8 format to accommodate upper ASCII characters. It was created by entering text in a simple text editor and pasting in some special characters from a Word document. It is not very long, but is meant to test simple character counting and also counting of special characters that may be represented internally by more than one character. This analysis is for the UTF-8 format. Files in other formats will have different internal representations but we have found UTF-8 to be the most reliable way to display and edit text files. The actual file details and counts as verified by a binary editor are:

PrincessOfMars.txt
This document was downloaded as an HTML file from the Project Gutenburg website at http://www.gutenberg.org/ebooks/62. The file was then converted to text to ensure that sentences were not truncated on every physical line. It was also saved in UTF-8 format to ensure proper handling of special characters. The resulting file contains a few double and triple byte characters as shown in the chart below. This file was used to demonstrate the character counting programs' performance on a larger file.

rawtextWithHeaders.doc
The last file is a MS Word doc file that was created by opening the rawtext.txt file in Word and then adding a few Word-specific items. I added headers and footers, and a few footnotes and endnotes to see how the various counting software programs handle these items. Not all of the programs can handle Word files directly so we did not test those with this file. Also, the total file size calculation is not really meaningful for Word files since a lot of extra information is added internally in the file by Word. This also makes it impossible to accurately estimate the amount of internal storage for hidden characters. However, the main numbers of total characters should be consistent and accurate. It appears that none of the counting programs included the running headers and footers in their totals. Word reports the characters in footnotes and endnotes separately so we included those numbers on a separate line.