OAEI 2012

From Thinclabwiki

Jump to: navigation, search


OAEI 2012 preliminary results of Optima+

Optima+, the latest version of Optima participated in Ontology Alignment Evaluation Initiative 2012. Optima+ participated in the following tracks of the campaign offered in SEALS platform: Benchmark, Conference, Library and Anatomy.

OAEI paper summarizing the specification of Optima+ and the results can be found [here].

Benchmark Track

The Benchmark test library consists of 5 different test suites. Each of the test suits is based on individual ontologies. Each of these test suits consists of number of test cases. Each test case discards a number of information from the ontology to evaluate the change in the behavior of the algorithm. There are six categories of such alterations. These include changing name of entities, suppression or translation of comments, changing hierarchy, suppressing instances, discarding properties with restrictions or suppressing and expanding classes into several classes or vice versa. Suppressing entities and replacing their names with random strings results into scrambled labels of entities. Test cases from 248 till 266 consist of such entities with scrambled labels.

The results for benchmark track are reported in the table below. An explicit split between the scores of test cases with scrambled labels was necessary to establish performance of Optima+ . Average precision for Optima+ is 0.95 while average recall is more than 0.83 for all the test cases in 200 series except those with scrambled labels. For test cases with scrambled labels, average precision is dropped to 0.91 and average recall to 0.36. When labels are scrambled, lexical similarity becomes ineffective. For Optima+ algorithm, structural similarity stems from lexical similarity hence scrambling the labels makes the alignment more challenging for Optima+ . Result is 46% decrease in average F-Measure from 0.85 to 0.46.

Bibliography 2 3 4 Finance
100 Series 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
201-247 0.88 0.85 0.85 1 0.84 0.87 0.97 0.88 0.89 0.93 0.77 0.79 0.96 0.8 0.83
248-266 0.65 0.35 0.43 1 0.36 0.46 0.98 0.38 0.49 0.96 0.34 0.43 0.96 0.38 0.49

This trend of reduction in precision, recall and f-measure can be observed through- out the benchmark track. For all the test suits, test cases with scrambled labels resulted into lower precision, recall and f-measure. Optima+ ’s algorithm is yet to be adopted for aligning ontologies with low or no lexical similarity.

Descriptive results for each test suit can be found at: Bibliography, 2, 3 , 4 , Finance .

Conference Track

For this track, Optima+ achieves recall of 0.68 and precision of 0.62. Both the recall and the precision are improved compared to the performance of Optima in OAEI 2011. Overall there is 81% increase in F-Measure compared to OAEI 2011. This makes Optima+ , the top performer in terms of F-Meaure according to OAEI 2011[3] . Table 2 lists the harmonic mean of precision, recall and f-measure along with total runtime for conference track of Optima in OAEI 2011 and Optima+ in OAEI 2012. Following Table shows comparison between performances of Optima+ in OAEI 2012 and Optima in OAEI 2011 for conference track.

Year 2011 2012
Precision (H-mean) 0.26 0.62
Recall (H-mean) 0.6 0.68
F-Measure (H-mean) 0.36 0.65
Total Runtime 15hr< 1349sec

Much of the improvement for Optima+ in conference track arises from the improved similarity measure and the alignment extraction. Optima+ also utilizes improved design and optimization techniques to reduce the runtime. The runtimes reported in the above table cannot be compared directly as the underlying systems used for evaluations differ. Although the runtime improvement from more than 15 hours to 23 minutes (1349 seconds) is perspicuous. Descriptive results for Conference track can be found here.

Multifarm Track

Since Optima+ focus only on english languge ontologies, it gives low performance in this track as expected. However its interesting to notice that Optima+ yields an average recall of 1 with an average precision of 0.01. Descriptive results for Multifarm track can be found here.

Anatomy Track

Since Optima does not have an explict large ontology matching stratergy, it did not participate in large ontology matching tracks. This year it debutes the large ontology matching tracks. In anatomy track, Optima+ yields 0.854 precision and 0.584 recall in 6460 seconds. There are no descriptive results for this track.

Library Track

Library is another large ontology matching track in OAEI 2012. Optima+ attains a precision of 0.321 and a recall of 0.072 in 37,457 seconds. We hope to improve our large ontology matching performance by creating a special alignment extraction for large ontologies. There are no descriptive results for this track.

Personal tools