Standardization of Criteria across Multiple Evaluators to Detect Objects

Authors

  • Naoya Wakabayashi Graduate School of Information Science and Engineering, University of Ritsumeikan, Japan
  • Hiromitsu Shimakawa Graduate School of Information Science and Engineering, University of Ritsumeikan, Japan

DOI:

https://doi.org/10.14738/tecs.111.13876

Keywords:

Object detection, Inconsistent labels, Supervised machine learning, Criteria standardization, Label synthesis, Reform based on feedback

Abstract

For a typical object detection task with machine learning technique, there has an absolute correct label on which a mechanical model is constructed. However, there are many tasks in which labels vary with evaluators because they have different criteria for discrimination. It happens when the detection criteria are vague and undefined among evaluators. This problem is avoided by constructing an individual model, but it is not recommended from a long-term perspective, because the model depends on a specific evaluator. This paper proposes a method to standardize the evaluator's criteria. The method is verified in the image detection task of asteroid powders in paint material as an example of ambiguity in the criteria. The performance and variance of detection with the proposed method are compared with conventional ones to evaluate whether it allows us to standardize the evaluators’ criteria. It turns out a clear reduction in the variance of evaluators’ detection results without significant degradation of the performance for all evaluators. It is confirmed this method standardizes the criteria across multiple evaluators. In addition, the paper discusses how to obtain manuals formally expressing the standardized criteria with text mining.

References

L. Elisa Celis, and Vijay Keswani, Implicit Diversity in Image Summarization. Proceedings of the ACM on Human-Computer Interaction, 2020. 4(139): p. 1-28.

Thomas Deselaers, et al., Jointly optimising relevance and diversity in image retrieval. CIVR '09: Proceedings of the ACM International Conference on Image and Video Retrieval, 2009. 39: p. 1-8.

Chen Karako, and Putra Manggala, Using Image Fairness Representations in Diversity-Based Re-ranking for Recommendations. UMAP '18: Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization, 2018. p. 23-28.

Zhiming Liu, et al., ML-CookGAN: Multi-Label Generative Adversarial Network for Food Image Generation. ACM Transactions on Multimedia Computing, Communications, and Applications, Just Accepted, 2022.

Lianli Gao, et al., Lab2Pix: Label-Adaptive Generative Adversarial Network for Unsupervised Image Synthesis. MM '20: Proceedings of the 28th ACM International Conference on Multimedia, 2020. p. 3734-3742.

Robert Styrzyński, et al., Development of paints and varnishes in the face of dynamic changes of end users’ needs - challenges for raw material producers. 2019.

Frank J. Maile, et al., Effect pigments - past, present and future. Progress in Organic Coatings, 2005. vol. 54, 3(1): p. 150-163.

Wei Liu, et al., Cold Bonding Method for Metallic Powder Coatings. Materials, 2018. vol. 11, 11(11).

Muhammad Sopyan Yahya, and Isti Surjandari, Classification of Printing Ink Quality based on Pigment, Dispersion Process: A Case Study. Proceedings of the 7th North American International Conference on Industrial Engineering and Operations Management, 2022. p. 942-951.

Naoya Wakabayashi, and Hiromitsu Shimakawa, Choosing Suitable Object Detection Method to Examine Pigment Powders of Similar Shape. 2022 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), 2022.

Joseph Redmon, et al., You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. p. 779-788.

Harold A. Linstone, and Murray Turoff, The Delphi Method: Techniques and Applications. Addison-Wesley, 1975.

Kari K. Lilja, et al., Using the Delphi method. 2011 Proceedings of PICMET '11: Technology Management in the Energy Smart World (PICMET), 2011. p. 1-10.

Tzutalin Darrenl, The labelImg package. [Online]. Available: https://github.com/heartexlabs/labelImg.

Mark Everingham, et al., The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 2010. 88(2), p. 303-338.

Li Liu, et al., Deep Learning for Generic Object Detection: A Survey. 2020. 128(2), p. 261-318.

Cartucho, The mAP package. [Online]. Available: https://github.com/Cartucho/mAP.

Google, Google Colaboratory. [Online]. https://colab.research.google.com.

Naoya Wakabayashi, and Hiromitsu Shimakawa, Detection of Powder Granule Attachment Using Object Detection Methods. FIT2022, In Japanese, 2022.

Ultralytics, The yolov5 package. [Online]. Available: https://github. com/ultralytics/yolov5.

Michael W. Berry, Survey of Text Mining: Clustering, Classification, and Retrieval. Springer, 2004.

Downloads

Published

2023-01-31

How to Cite

Wakabayashi, N., & Shimakawa, H. (2023). Standardization of Criteria across Multiple Evaluators to Detect Objects. Transactions on Engineering and Computing Sciences, 11(1), 22–36. https://doi.org/10.14738/tecs.111.13876