Following the standard practice in object detection, we calculate the average precision (AP) for each submission. The metric will be calculated at IoU=0.5. Illegible or non-English text will be treated as “don’t care” objects. The AP at IoU = 0.5 will be taken as the primary challenge metric for ranking the submissions. This metric is equivalent to the “meanAP” metric adopted by PASCAL VOC, since we only have one category.

Compared to Hmean, measuring average precision takes into account the trade-off between precision and recall, thus enabling a more comprehensive view on the performance. AP also exempts the participants from manually adjusting thresholds on their results. Further, we will also provide the individual recall, precision and Hmean values for each submission for a detailed comparison.

Localization results will be saved in one UTF-8 encoded text file per test image. Authors will be asked to submit a single ZIP file of the folder containing all the text files (folder should be named detections). Result files are named after the name of the test images (e.g. Printed_600_hindi_180.txt). The resultant text files will have the following format with CR/LF line endings.

left_1,top_1,right_1,bottom_1,score_1
left_2,top_2,right_2,bottom_2,score_2
...
left_n,top_n,right_n,bottom_n,score_n

Authors will be asked to include a confidence score for every bounding box. As this score will allow us to calculate an Average Precision (AP) score for the submission.

For more details regarding the online submissions, please refer to the main task page.

Online Submission is a feature introduced by the Indian language benchmark system for aiding in reducing the over-fitting of a particular submission to a specific dataset. This feat is accomplished by requesting the author for producing the result of the method individually, i.e. instead of uploading all the results from a test dataset in task user is asked to run a script that downloads an individual test image from dataset, calculates the result for that image according the logic provided by the user and then uploads the result to the benchmark evaluation system.

This process can be easily accomplished using GET/POST requests framework built in the system.
Below is provided a sample (python) code that uses this online framework to upload one online submission.

For each registered user, there is a unique api_key generated by the system that the user has to use in the url of GET/POST requests

Please Login/Register to get the api_key nessacary for online submissions

No Description Available

Please Login/Register to upload new submissions.