TRANSIT/ieee_template
Template
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Valère Plantevin
2026-04-18 20:00:07 -04:00
commit b37e65cc2a
50 changed files with 11372 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
examples/2023botrosRAL.qmd Normal file
View File

@@ -0,0 +1,128 @@
---
title: "USMicroMagSet: Using Deep Learning Analysis to Benchmark the Performance of Microrobots in Ultrasound Images"
format:
ieee-pdf:
keep-tex: true
ieee-html: default
tex-author-no-affiliation: true
affiliations:
- id: PRISME
name: "INSA Centre Val de Loire, Univ. Orléans, PRISME EA4229"
affiliation-url: https://www.univ-orleans.fr/fr/prisme
city: Bourges
country: France
- id: pascal
name: Clermont Auvergne INP, CNRS, SIGMA Clermont
departement: Institut Pascal
city: Clermont Ferrand
country: France
url: http://www.institutpascal.uca.fr
author:
- name: Karim Botros
orcid: 0000-0003-0245-1852
membership: Student Member, IEEE
affiliations:
- ref: PRISME
note:
- "Manuscript received 31 October 2022; accepted 9 March 2023. Date of publication 5 April 2023; date of current version 21 April 2023."
- "This paper was recommended for publication by Editor Pietro Valdastri upon evaluation of the Associate Editor and Reviewers' comments. This work was supported by the Region Centre Val de Loire Fund with the BUBBLEBOT project."
- name: Mohammad Alkhatib
affiliations:
- ref: pascal
orcid: 0000-0003-0971-3835
membership: Member, IEEE
note: "Karim Botros, David Folio, and Antoine Ferreira are with the INSA Centre Val de Loire, Laboratoire PRISME, 18000, Bourges, France (e-mail: <karim.botros@insa-cvl.fr>; david.folio@insa-cvl.fr; antoine.ferreira@insacvl.fr)."
- name: David Folio
affiliations:
- ref: PRISME
orcid: 0000-0001-9430-6091
membership: Member, IEEE
email: david.folio@insa-cvl.fr
url: https://dfolio.fr/
note: "Mohammad Alkhatib is with the Clermont Auvergne INP, CNRS, SIGMA Clermont, Institut Pascal, F-63000 Clermont Ferrand, France"
- name: Antoine Ferreira
affiliations:
- ref: PRISME
orcid: 0000-0001-6295-3876
membership: Member, IEEE
email: antoine.ferreira@insa-cvl.fr
attributes:
corresponding: true
note: "Digital Object Identifier (DOI): [10.1109/LRA.2023.3264746](https://doi.org/10.1109/LRA.2023.3264746)."
abstract: |
Microscale robots introduce great perspectives into many medical applications such as drug delivery, minimally invasive surgery, and localized biometric diagnostics. Fully automatic microrobots' real-time detection and tracking using medical imagers are actually investigated for future clinical translation. Ultrasound (US) B-mode imaging has been employed to monitor single agents and collective swarms of microrobots _in vitro_ and _ex vivo_ in controlled experimental conditions. However, low contrast and spatial resolution still limit the effective employment of such a method in a medical microrobotic scenario due to uncertainties associated with the position of microrobots. The positioning error arises due to the inaccuracy of the US-based visual feedback, which is provided by the detection and tracking algorithms. The application of deep learning networks is a promising solution to detect and track real-time microrobots in noisy ultrasonic images. However, what is most striking is the performance gap among state-of-the-art microrobots deep learning detection and tracking research. A key factor of that is the unavailability of large-scale datasets and benchmarks. In this paper, we present the first publicly available B-mode ultrasound dataset for microrobots (_USmicroMagSet_) with accurate annotations which contains more than 40000 samples of magnetic microrobots.
In addition, for analyzing the performance of microrobots included in the proposed benchmark dataset, 4 deep learning detectors and 4 deep learning trackers are used.
keywords: [Micro/nano robots, Medical robots and systems, deep learning methods.]
hyphenation: [in-vasive, auto-matic, micro-robot]
pageheader:
left: "IEEE Robotics and Automation Letters. Preprint Version (generated by the authors). Accepted March, 2023"
right: 'K. Botros _et al._: USMicroMagSet'
bibliography: 2023botrosRAL.bib
date: 2023-04-05
pdf: 2023botrosRAL.pdf
funding: Region Centre Val de Loire Fund with the BUBBLEBOT
citation:
container-title: IEEE Robotics and Automation Letters
page: 3254 - 3261
type: article-journal
publisher: IEEE
issued: 2023-06
doi: 10.1109/LRA.2023.3264746
url: https://ieeexplore.ieee.org/document/10093014
volume: 8
issue: 6
---
# Introduction {#sec-intro}
:::{#fig-dataset fig-env="figure*" fig-pos="ht"}
![](Figures/Figure1.png){width="80%"}
Illustration of the three classes of eight swimming magnetic microrobots used for the $USMicroMagset$ dataset. For each microrobot, 5k images were extracted from 5 videos of 10min each. (Row 1) Class SMF: Steering magnetic field with three subclasses (sphere, cube and cylinder) of hard NdFeB magnets. (Row 2) Class RMF: Rotating magnetic field with three subclasses (helix microrobot, soft magnetic sheet microrobot, cube microrobot). (Row 3) Class OMF: Oscillating magnetic field with two subclasses (chain-like swimmer, one-armed flagella swimmer with magnetic head).
:::
[R]{.IEEEPARstart}[ecently]{} untethered mobile magnetic microrobots have been proposed for a wide range of biomedical applications (drug delivery, clot removal, tissue regeneration) @li2017micro due to their small size and ability to operate in confined spaces and hard-to-reach environments @chung2021magnetically. In endovascular navigation, real-time localization and tracking of microrobots are essential for achieving targeted navigation through imaging feedback. To do so, various medical imaging technologies are being used for the detection and tracking of microrobots such as ultrasound (US) imaging, magnetic resonance imaging, computer tomography, infrared fluorescence imaging, X-ray, and positron emission tomography @pane2019imaging. Recently, there has been increasing attention towards the navigation of magnetic microrobots using active mode US imaging endorsed by healthcare applications @scheggi2017magnetic. It can be explained by the fact that ultrasonic imaging is a noninvasive, versatile, low cost and well-established technique that is regularly used in medical settings.
The ultrasound B-mode imaging was successfully used at different scales to detect and to track in real-time different robots _in vitro_ and _in vivo_.
For example at the millimeter-scale, soft grippers @khalil2018mechanical, helical swimmers @hu2018smallscale or soft-bodied robots @ackermann2016detection have been investigated.
Different tracking techniques have been implemented to track the small-scale robots such as modified Markov chain Monte Carlo data association algorithm @zhang2021kalman, Kalman filtering @zhang2021kalman, and conventional neural networks @tiryaki2022deep. At the microscale, various types of microrobots were successfully demonstrated such as paramagnetic particles @khalil2014magneticbase, acoustically actuated microswimmer @chen2019ultrasound, self-propelled microjets @sanchez2014magnetic and bio-inspired magnetosperm @magdanz2020ironsperm in living tissue. However, the lack of B-mode resolution owing to poor microrobot echogenicity in image contrast, as well as the low signal-to-noise ratio (SNR), hinders the real-time tracking of microrobots in endovascular applications. To enhance spatial resolution, ultrasound phase analysis was implemented in @pane2021realtime to derive microrobot features such as size and position over time allowing to perform imaging and tracking of a low contrast microrobot in chicken breast.
Finally, Doppler-based ultrasound appears as a promising tool for tracking microrobots in echogenic and dynamic environments as biological tissues. In [@wang2020realtime; @wang2021ultrasound] a strategy to navigate a nanoparticle microswarm in real-time under ultrasound Doppler imaging guidance for active endovascular delivery was implemented in blood vessels. The Doppler signals near the microswarm in flowing blood environments were observed, and the microswarm was efficiently tracked and navigated in real-time using Doppler feedback. However, taking into the wide diversity of robot geometries and sizes, variability of swimming principles, and robot material echogenicity properties, the proposed state-of-the-art real-time detection and tracking methodologies face shadowing, low contrast, strong attenuation across an image, and fuzziness of vessel boundaries.
Microrobot detection and tracking in vessels is a challenging task, particularly when using US imaging. These challenges in US images include speckle noise, dynamic backgrounds, blur, and US image artifacts. This drives the employment of deep learning-based methods to enhance microrobot detection and tracking in vessels @liu2022capsule.
However, further research on deep learning detection of microrobots in US images is still not fully explored. Furthermore, convolution neural networks (CNN) models and frameworks may also be retrained using a customized dataset, offering deep learning techniques more flexibility than computer vision methods.
Moreover, deep learning-driven techniques yield promising performance for automatic detection and tracking of navigable magnetic microrobots using ultrasonic imaging @botross2022fully. But, there is no well-established benchmarking dataset. Typically, some research teams test and report the performance of their technique on their own private dataset utilizing experimental setups designed expressly for that purpose. Consequently, comparing the performance of multiple approaches or predicting how a given methodology will perform if the experimental setup/conditions change is challenging.
In this work, we propose a public large-scale benchmarking dataset ($USMicroMagset$), summarized in @fig-dataset. The dataset consists of ultrasound B-mode images of magnetic microrobots with different aspect ratios, sizes (1 mm to 300 µm), shapes, soft/rigid structures, and locomotion principles. This survey provides some insights into the challenges of detectability and trackability of a wide variety of magnetic microrobots navigating in microfluidic channels mimicking a vessel network.
In addition, this dataset is used to perform a comprehensive survey of deep learning-based microrobot detection and tracking in ultrasonic B-mode images. Based on these comparative analyses [@srivastava2021comparative; @marvasti-zadeh2022deep], we have extracted the best four detectors and the best four trackers. Then we evaluated these detectors and trackers on our dataset $USMicroMagset$. Finally, using the best detector and tracker algorithms, we evaluated each microrobot under real-time challenging scenarios simulating in vivo imaging problems such as changing ultrasound parameters, changing speed of microrobot motion, partial and full occlusion, and out-of-plane motion.
The paper is organized as follows: @sec-benchmarking describes the datasets and the methods used in the proposed work. Section 3 describes the deep learning detectors and trackers. Section 4 made a comparative assessment of the different detectors and trackers before concluding.
# Microrobot Benchmarking {#sec-benchmarking}
## Classes
In the proposed $USMicroMagset$ dataset, there are three different classes describing three operating principles of hard and soft magnetic microrobots. Our dataset is composed of 40k images subdivided into three classes of magnetic microrobots.
@fig-dataset summarizes the locomotion principles for each class of magnetic microrobot used in the datasets. We recorded 5k images for each microrobot prototype (80\% for training and 20\% for testing). All images have been annotated manually by an experienced expert.
## Microrobot principle
Basically, governing equations of magnetized hard and soft microrobots controlled by magnetic fields are as follows :
\begin{eqnarray}
{\vb{\textbf{F}}_{m}}&= \left(\vb{M}\cdot \nabla\right) \vb{B} \label{eq:Fm} \\
\vb{\textbf{T}_{m}}&= \left(\vb{M}\times\vb{B}\right) \label{eq:Tm}
\end{eqnarray}
where $\vb{\textbf{T}_{m}}$ is the resultant magnetic torque; $\vb{\textbf{F}_{m}}$ is the resultant magnetic force; $\vb{M}$ is the magnetic moment of the object; $\vb{B}$ is the applied magnetic field. Depending on the actuating magnetic fields, three classes have been identified and tested:
### Steering magnetic field (SMF) class
The first class consists of three subclasses of force-driven microrobots (Row\;1 in @fig-dataset) using various shapes of hard NdFeB magnets (spheres, cubes, and cylinders).
The steering motion relies on the use of SMF induced by three-axis Maxwell pair coils.
The resultant steering force $\vb{\textbf{F}_{m}}$ provides a linear steering motion of the microrobot [@kummer2010octomag; @mathieu2010steeringa; @ghanbari2014electromagnetic]
:::{.center data-latex=""}
**Full text are available from [IEEEXplore^®^](https://ieeexplore.ieee.org/document/10093014/)**
:::
::: {.content-visible when-format="pdf"}
# References {-}
:::