Fetch Technologies Blog

Jerry Thurber, President of Tandem Select, blogs about the use of artificial intelligence in background screening:

http://backgroundcheck.tandemselect.com/bid/53806/Artificial-Intelligence-in-Background-Checking-A-Quick-Introduction

Here’s the full post:

Artificial intelligence (AI) has been a standard tool in many industries for a decade or more. Recently artificial intelligence has begun to gain a foothold in the background checking industry. AI, or machine learning, is a useful way to train machines to do things humans do when that task is not strictly the same all the time. In other words, the machine has to be smart enough to know how to adjust to basic changes in the process. An example of this in background checking occurs when you use AI “agents” (or robots if you prefer) to access and retrieve criminal history data. A growing volume of criminal history data is maintained in secure websites that are made available for background checking and pre employment screening. These sites require that a person log into the site, enter their credentials, then search for a specific name to see if there is a criminal record for that person. The reason why artificial intelligence is so nice with sites like these is because the exact navigation for finding these records differs from site to site and from search to search. For example, if I search for Jane Doe’s criminal record in Colorado I may need to search through three or four pages to cover all the information I need but if I then search for John Doe’s criminal record on that same Colorado criminal history site, I may find something that takes me off in a different direction and requires me to look at information that wasn’t relevant for Jane Doe but is relevant for John Doe’s criminal history. In other words, the search is situational. Jane Doe’s search required different navigation than John Doe’s search. When you use an artificial intelligence tool for a search like this, it can be “trained” to see these situational anomalies and it can adjust to make sure it follows the correct search path.

These trained agents have been found to be very accurate in their search execution. In one assessment done at Fetch Technologies they worked with one of their customers and found that the AI search was 10% more accurate than the human beings conducting those same searches. Machines don’t forget to look at every page or fail to navigate through all the records; they don’t get tired or type the wrong information. Once trained – they do their job completely, every time. The other advantage of AI tools is they don’t have to sleep. They can search records 24 hours a day, seven days a week. At Tandem Select, we have used AI tools to improve our accuracy and our turnaround time. When we can use artificial intelligence for criminal history site searches, we reduce turnaround time from several hours to usually several seconds. At Tandem Select, we think artificial intelligence helps our clients get better, faster, more accurate criminal history results.

Fetch Technologies uses just two measurements to describe performance: accuracy and coverage. Accuracy, which describes how well a web extraction system is performing in extracting data fields from web pages, and coverage, which describes how well a web extraction system is performing in retrieving the targeted web pages, were both covered in part I. In this post, we describe how sampling is used to gather data for accuracy and coverage measurements.

Sampling

In practice, it’s difficult to compute the exact accuracy and coverage because our answer key is normally incomplete. That is, if we want to know precisely what the performance of a system was on a website, we would need to know, for every target page on the entire site, what the extracted values should be. Because this is usually impractical, we normally test the system on a set of sample pages. There are some important things to remember about sample pages. First, the set of sample pages should be randomly selected. That is, there should be no bias in terms of how the pages collected. Second, in order for our measurements to be reflective of the true performance of the system, we need to collect “enough” sample pages. If our sample set is too small, then we are in danger of obtaining a measurement that is too high or too low, due to bad luck. The larger the sample set, the better our chances are of getting an accurate measurement.

How big should our sample set be? Statisticians have been studying this problem for many years. There are formulas available to help determine an appropriate sample size depending how confident we want to be. For instance, suppose we want our accuracy to be at least 80% and we find that on 10 sample pages accuracy is 90%. Unfortunately, we can’t really be very confident that the true accuracy is really over 80%, because 10 sample pages is a very small sample. However, if we use 50 sample pages and measure accuracy at 90% (for instance), then statistical formulas tell us that we can be highly confident that the accuracy is at least 80%, if not higher. In fact, statistics allow us to define what we mean by “highly confident” in a very precise way. Information about statistical sampling can be found on-line – Wikipedia has good articles about sampling – or from statistics textbooks.

Performance Measurements and Statistics

Measuring both accuracy and coverage enables us to gain a clear understanding of any problems the system may be experiencing when errors are encountered. For instance, if both coverage measures look good but the accuracy is low then presumably the system is correctly identifying the target pages, but poorly extracting the data from these pages. On the other hand, if the %extra is high, %missing is low, and accuracy is poor, then presumably the problem is that the system is collecting too many pages due to an overly-general description of the target pages.

For the statistically minded, we note that our coverage measures are directly related to the terms recall and precision used in the scientific community. Specifically, %missing is the same as 1 – Recall, and %extra is the same as 1 – Precision. Also, statisticians often refer to extra pages as false positives, to missing pages as false negatives. Similarly, the correctly retrieve pages are referred to as true positives, and the pages that the system correctly determined not to retrieve are called true negatives.

Figure 4: Relationship of Missing and Extra to False Positives and False Negatives

The approach that we have described in this series of blog posts works well when there are a fixed number of data items being extracted from the target pages. However, there are more complex cases which make evaluation more difficult. For instance, sometimes we want to extract a list of data values from target pages. This makes it hard to measure accuracy, because the number of items in the list may vary from page to page. Fortunately, the coverage measures we introduced previously can be used very broadly.
In particular, we can create an answer key with all the data values that should result from the extraction process, and simply evaluate how many of the data values are missing from the answer key and how many extra data values there are on the answer key. This gives us a basic way to measure performance, but the downside is that we don’t compute a separate accuracy measure, and as a result it can be harder to figure out what went wrong if performance is poor. For instance, if there is a target list that includes the data value “John Smith”, and the system incorrectly extracts the result as “John”, than this will count as both a missing value (because “John Smith” is missing) and an extra value (because “John” is not a target value), which can be confusing.

The basic approach described in these posts has several virtues. First, it is relatively straightforward and practical to implement. Second, when extraction issues occur, they can be relatively easily pinpointed and debugged. Finally, the approach can be described in terms of standard statistical metrics, and if necessary can be extended to handle more complex situations.

Extracting data from websites is a complex process, and evaluating how well such systems perform can be difficult. In this post, we describe the approach Fetch Technologies uses in measuring the performance of an extraction system. Fetch Technologies uses just two measurements to describe performance: accuracy and coverage. Accuracy describes how well a web extraction system performs in extracting data fields from web pages. Coverage describes how well a web extraction system performs in retrieving the targeted web pages. This approach was developed by our scientists after years of experience with both commercial tests and scientific evaluations reported in leading academic journals.

Accuracy

First, let’s consider the basics. Let’s assume that we want to extract several data fields from a set of web pages. A data field is a piece of information that may appear on a page, for instance a price on a page from a retail site, an article title on a page from a news site, or a phone number from an individual’s home page. The data in a field may be a string, or a number, a URL, or any other type of data. A data field is the smallest unit of extracted data. Note that an address might be extracted as four separate data fields, such as street address, city, state and zip, or it might be extracted as a single data field containing the entire string. Figure 1 shows a page from a retail store, TractorSupply.com, showing three target fields: “Product Title”, “Price” and “Product Description”.

Figure 1: Sample Webpage Showing Data Fields

For a fixed set of web pages, it is useful to create an answer key that specifies the target values for the data fields. That is, for every page the answer key indicates the correct value for each data field that should be extracted.

Once we have an answer key, we can characterize the performance of an extractor in terms of its accuracy over a set of pages. Accuracy is simply the number of correct values extracted out of the total number of target values.

For instance, suppose there are 200 pages and 5 data fields, and the extractor gets 990 of the 1000 values correct, and the remaining 10 values wrong. Then the accuracy would be 99%.

Coverage

Accuracy is a valuable metric for describing extraction performance, but it is not the whole story. It is most appropriate for describing how well an extractor works on a given type of page, where each such page is reasonably expected to contain a target value for each field. Often, however, we have to navigate through a site to identify the target pages we want to extract target values from. For example, on a retail site, our target pages might be “product detail pages” each of which contains a target value for the fields “product name”, “price” and “product description”. On some sites the target pages might only be a small minority of the site’s pages. For instance, on TractorSupply.com, we might only be interested in extracting data from pages in the “Lawn and Garden” category.

Coverage can be measured counting the number of missing pages, and the number of extra pages retrieved by the system, as shown in Figure 2.

Figure 2: Conceptual Illustration of Coverage

A system with perfect coverage would identify all the target pages, and only the target pages, as shown in Figure 3.

Figure 3: Conceptual Illustration of Perfect Coverage

These can be expressed as percentages.

For instance, if there are 50 target pages, and the system identifies 51 pages, where 48 are correct and 3 are extra, then we say that 4% are missing (2 missing pages out of 50 target pages) and 6% are extra (3 extra pages out of the 51 pages identified).

In the next post, we’ll describe how sampling is used to gather data for accuracy and coverage measurements.

Interesting new play in the real estate space – powered by Fetch:

Spatial Match gives Realtors® competitive edge
Wednesday, April 6, 2011
By Kim Shindle

A new program for websites gives Realtors® a competitive edge over mega-sites like Zillow and Trulia.

“We recognize that it’s been a challenge for independent Realtors® without mega budgets to compete for consumers’ time when larger companies like Zillow and Trulia have such engaging websites,” said Grant Gould, vice-president of SpatialMatch®.com.

That’s why founders Gould and CEO John Perkins set out to create an information-rich mini-portal to be embedded into Realtors®’ websites to give consumers all the information they have been navigating off-site to get. The result for the Realtor® website is more interaction, more time-on-site, and ultimately more loyalty and deal flow from consumers.

“Most consumers start their search for a home online. When they go from a site like Zillow or Trulia to most Realtors®’ sites, the latter sites lack the deeper localized information that the consumers want prior to making their purchase decisions,” Gould said. “SpatialMatch® was designed to be embedded into the Realtor’s® site and gives consumers a way to research all aspects of areas and neighborhoods including a lifestyle search that helps them create a file of what’s important in their day-to-day life.”

For example, if consumers want to live in a specific school district, they can select and search for homes using that as a choice. If they want to live within 10 miles of their office, they can enter that address and any others as custom addresses and include them in the search. Or if they want to stop at Starbucks every morning and want it within five miles of their home, they can choose that as well. “We have 100 different databases feeding information into the matches and we continue to integrate new data monthly,” Gould explained.

See the rest of the article