Abstract

Structural health monitoring (SHM) data is the essential foundation for any SHM structural integrity assessment, including large civil infrastructure such as the miter gate application in this work. For some applications, the amount of monitoring data is limited due to various reasons such as a lack of sensor deployment investment, sensor reliability, inaccessibility of measurement locations, expensive duty cycles, etc. This limited data could result in uncertainty in structural health assessment. This paper addresses this challenging issue by proposing a data augmentation method based on image translation for Bayesian inference-based damage diagnostics. In particular, we translate the monitoring data of one miter gate to that of another, thereby increasing the volume of monitoring data available for assessing the structural health of a target miter gate. This translation starts with converting the monitoring data of different miter gates into images. After that, Cycle Generative Adversarial Networks (CycleGAN) are employed to accomplish the task of image translation among different miter gates. A verification method is developed to verify the accuracy of the translated images (i.e., synthetic monitoring data). After the accuracy verification, the translated images are used together with the true monitoring data for damage diagnostics. Two types of CycleGAN architectures are investigated and compared using a case study. Results of the case study show that the proposed data augmentation method can effectively improve the accuracy and confidence of damage diagnostics of miter gates. It demonstrates the potential of integrating synthetic data generation with probabilistic model updating in structural health monitoring.

References

1.
Foltz
,
S. D.
,
2017
,
Investigation of Mechanical Breakdowns Leading to Lock Closures Investigation of Mechanical Breakdowns Leading to Lock Closures
,
Construction Engineering Research Laboratory, Engineer Research and Development Center (U.S.)
, Vicksburg, MI.
2.
U.S. Army Corps of Engineers, Engineer Research and Development Center
,
2021
SMART GATE 2007
,” U.S. Army Corps of Engineers, Vicksburg, MS, accessed May 2024, https://www.erdc.usace.army.mil/Media/Fact-Sheets/Fact-Sheet-Article-View/Article/476668/smart-gate/
3.
Feng
,
D.
, and
Feng
,
M. Q.
,
2015
, “
Model Updating of Railway Bridge Using In Situ Dynamic Displacement Measurement Under Trainloads
,”
J. Bridge Eng.
,
20
(
12
), p.
04015019
.10.1061/(ASCE)BE.1943-5592.0000765
4.
Guo
,
T.
,
Li
,
A.
, and
Wang
,
H.
,
2008
, “
Influence of Ambient Temperature on the Fatigue Damage of Welded Bridge Decks
,”
Int. J. Fatigue
,
30
(
6
), pp.
1092
1102
.10.1016/j.ijfatigue.2007.08.004
5.
Randall
,
R. B.
,
2010
,
Vibration-Based Condition Monitoring: Industrial, Aerospace and Automotive Applications
, 1st ed.,
Wiley
,
Hoboken, NJ
.
6.
Catbas
,
F. N.
,
Susoy
,
M.
, and
Frangopol
,
D. M.
,
2008
, “
Structural Health Monitoring and Reliability Estimation: Long Span Truss Bridge Application With Environmental Monitoring Data
,”
Eng. Struct.
,
30
(
9
), pp.
2347
2359
.10.1016/j.engstruct.2008.01.013
7.
Morgan
,
C. J.
,
Sparling
,
B. F.
, and
Wegner
,
L. D.
,
2022
, “
Use of Structural Health Monitoring to Extend the Service Life of the Diefenbaker Bridge
,”
J. Civ. Struct. Health Monit.
,
12
(
4
), pp.
913
929
.10.1007/s13349-022-00585-1
8.
Ramancha
,
M. K.
,
Vega
,
M. A.
,
Conte
,
J. P.
,
Todd
,
M. D.
, and
Hu
,
Z.
,
2022
, “
Bayesian Model Updating With Finite Element Vs Surrogate Models: Application to a Miter Gate Structural System
,”
Eng. Struct.
,
272
, p.
114901
.10.1016/j.engstruct.2022.114901
9.
Orchard
,
M. E.
, and
Vachtsevanos
,
G. J.
,
2007
, “
A Particle Filtering Approach for On-Line Failure Prognosis in a Planetary Carrier Plate
,”
Int. J. Fuzzy Logic Intell. Syst.
,
7
(
4
), pp.
221
227
.10.5391/IJFIS.2007.7.4.221
10.
Daigle
,
M.
, and
Goebel
,
K.
,
2011
, “
Multiple Damage Progression Paths in Model-Based Prognostics
,”
2011 Aerospace Conference
, Big Sky, MT, Mar. 5–12, pp.
1
10
.10.1109/AERO.2011.5747574
11.
An
,
D.
,
Choi
,
J. H.
,
Schmitz
,
T. L.
, and
Kim
,
N. H.
,
2011
, “
In Situ Monitoring and Prediction of Progressive Joint Wear Using Bayesian Statistics
,”
Wear
,
270
(
11–12
), pp.
828
838
.10.1016/j.wear.2011.02.010
12.
Liao
,
L.
, and
Köttig
,
F.
,
2014
, “
Review of Hybrid Prognostics Approaches for Remaining Useful Life Prediction of Engineered Systems, and an Application to Battery Life Prediction
,”
IEEE Trans. Reliab.
,
63
(
1
), pp.
191
207
.10.1109/TR.2014.2299152
13.
Zeng
,
J.
,
Todd
,
M. D.
, and
Hu
,
Z.
,
2023
, “
Probabilistic Damage Detection Using a New Likelihood-Free Bayesian Inference Method
,”
J. Civ. Struct. Health Monit.
,
13
(
2–3
), pp.
319
341
.10.1007/s13349-022-00638-5
14.
Jesus
,
A.
,
Brommer
,
P.
,
Westgate
,
R.
,
Koo
,
K.
,
Brownjohn
,
J.
, and
Laory
,
I.
,
2019
, “
Modular Bayesian Damage Detection for Complex Civil Infrastructure
,”
J. Civil Struct. Health Monit.
,
9
(
2
), pp.
201
215
.10.1007/s13349-018-00321-8
15.
Zeng
,
J.
, and
Kim
,
Y. H.
,
2020
, “
Identification of Structural Stiffness and Mass Using Bayesian Model Updating Approach With Known Added Mass: Numerical Investigation
,”
Int. J. Struct. Stabil. Dyn.
,
20
(
11
), p.
2050123
.10.1142/S0219455420501230
16.
Zyphur
,
M. J.
, and
Oswald
,
F. L.
,
2015
, “
Bayesian Estimation and Inference: A User's Guide
,”
J. Manag.
,
41
(
2
), pp.
390
420
.
17.
Feng
,
D.
, and
Feng
,
M. Q.
,
2018
, “
Computer Vision for SHM of Civil Infrastructure: From Dynamic Response Measurement to Damage detection - A Review
,”
Eng. Struct.
,
156
, pp.
105
117
.10.1016/j.engstruct.2017.11.018
18.
Moaveni
,
B.
,
He
,
X.
,
Conte
,
J. P.
, and
Restrepo
,
J. I.
,
2010
, “
Damage Identification Study of a Seven-Story Full-Scale Building Slice Tested on the UCSD-NEES Shake Table
,”
Struct. Saf.
,
32
(
5
), pp.
347
356
. 10.1016/j.strusafe.2010.03.006
19.
Astroza
,
R.
,
Ebrahimian
,
H.
,
Conte
,
J. P.
,
Restrepo
,
J. I.
, and
Hutchinson
,
T. C.
,
2016
, “
System Identification of a Full-Scale Five-Story Reinforced Concrete Building Tested on the NEES-UCSD Shake Table
,”
Struct. Control Health Monit.
,
23
(
3
), pp.
535
559
.10.1002/stc.1778
20.
Gao
,
X.
,
Duan
,
G.
, and
Lan
,
C.
,
2021
, “
Bayesian Updates for an Extreme Value Distribution Model of Bridge Traffic Load Effect Based on SHM Data
,”
Sustainability
,
13
(
15
), p.
8631
.10.3390/su13158631
21.
Vega
,
M. A.
,
Hu
,
Z.
, and
Todd
,
M. D.
,
2020
, “
Optimal Maintenance Decisions for Deteriorating Quoin Blocks in Miter Gates Subject to Uncertainty in the Condition Rating Protocol
,”
Reliab. Eng. Syst. Saf.
,
204
, p.
107147
.10.1016/j.ress.2020.107147
22.
Yang
,
Y.
,
Chadha
,
M.
,
Hu
,
Z.
, and
Todd
,
M. D.
,
2023
, “
An Optimal Sensor Design Framework Accounting for Sensor Reliability Over the Structural Life Cycle
,”
Mech. Syst. Signal Process.
,
202
, p.
110673
.10.1016/j.ymssp.2023.110673
23.
Yang
,
S.
,
Xiao
,
W.
,
Zhang
,
M.
,
Guo
,
S.
,
Zhao
,
J.
, and
Shen
,
F.
,
2022
, “
Image Data Augmentation for Deep Learning: A Survey
,” preprint
arXiv:2204.08610
.10.48550/arXiv.2204.08610
24.
Iwana
,
B. K.
, and
Uchida
,
S.
,
2021
, “
An Empirical Survey of Data Augmentation for Time Series Classification With Neural Networks
,”
PLoS One
,
16
(
7
), p.
e0254841
.10.1371/journal.pone.0254841
25.
Zhuang
,
F.
,
Qi
,
Z.
,
Duan
,
K.
,
Xi
,
D.
,
Zhu
,
Y.
,
Zhu
,
H.
,
Xiong
,
H.
, and
He
,
Q.
,
2021
, “
A Comprehensive Survey on Transfer Learning
,”
Proc. IEEE
,
109
(
1
), pp.
43
76
.10.1109/JPROC.2020.3004555
26.
Huang
,
X.
,
Xie
,
T.
,
Wang
,
Z.
,
Chen
,
L.
,
Zhou
,
Q.
, and
Hu
,
Z.
,
2022
, “
A Transfer Learning-Based Multi-Fidelity Point-Cloud Neural Network Approach for Melt Pool Modeling in Additive Manufacturing
,”
ASCE-ASME J. Risk Uncertainty Eng. Syst., Part B: Mech. Eng.
,
8
(
1
), p.
011104
.10.1115/1.4051749
27.
Thelen
,
A.
,
Zhang
,
X.
,
Fink
,
O.
,
Lu
,
Y.
,
Ghosh
,
S.
,
Youn
,
B. D.
,
Todd
,
M. D.
,
Mahadevan
,
S.
,
Hu
,
C.
, and
Hu
,
Z.
,
2023
, “
A Comprehensive Review of Digital Twin-Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectives
,”
Struct. Multidisciplinary Optim.
,
66
(1),
p. 1
.10.1007/s00158-022-03410-x
28.
Jiang
,
C.
,
Vega
,
M. A.
,
Ramancha
,
M. K.
,
Todd
,
M. D.
,
Conte
,
J. P.
,
Parno
,
M.
, and
Hu
,
Z.
,
2022
, “
Bayesian Calibration of Multi-Level Model With Unobservable Distributed Response and Application to Miter Gates
,”
Mech. Syst. Signal Process.
,
170
, p.
108852
.10.1016/j.ymssp.2022.108852
29.
Thelen
,
A.
,
Zhang
,
X.
,
Fink
,
O.
,
Lu
,
Y.
,
Ghosh
,
S.
,
Youn
,
B. D.
,
Todd
,
M. D.
,
Mahadevan
,
S.
,
Hu
,
C.
, and
Hu
,
Z.
,
2022
, “
A Comprehensive Review of Digital Twin-Part 1: Modeling and Twinning Enabling Technologies
,”
Struct. Multidiscip. Optim.
,
65
(
12
), p.
354
.10.1007/s00158-022-03425-4
30.
Vega
,
M. A.
,
Hu
,
Z.
,
Fillmore
,
T. B.
,
Smith
,
M. D.
, and
Todd
,
M. D.
,
2021
. “
A novel framework for integration of abstracted inspection data and structural health monitoring for damage prognosis of miter gates
,”
Reliab. Eng. Syst. Saf.
,
211
, p.
107561
10.1016/j.ress.2021.107561.
31.
Orchard
,
M.
,
Kacprzynski
,
G.
,
Goebel
,
K.
,
Saha
,
B.
, and
Vachtsevanos
,
G.
,
2009
,
Advances in Uncertainty Representation and Management for Particle Filtering Applied to Prognostics
,
Springer
,
Netherlands, Dordrecht
, pp.
23
35
.
32.
Zeng
,
J.
,
Todd
,
M. D.
, and
Hu
,
Z.
,
2023
, “
A Recursive Inference Method Based on Invertible Neural Network for Multi-Level Model Updating Using Video Monitoring Data
,”
Mech. Syst. Signal Process.
,
203
, p.
110736
.10.1016/j.ymssp.2023.110736
33.
Wu
,
Z.
,
Fillmore
,
T. B.
,
Vega
,
M. A.
,
Hu
,
Z.
, and
Todd
,
M. D.
,
2022
, “
Diagnostics and Prognostics of Multi-Mode Failure Scenarios in Miter Gates Using Multiple Data Sources and a Dynamic Bayesian Network
,”
Struct. Multidiscip. Optim.
,
65
(
9
), p.
270
.10.1007/s00158-022-03381-z
34.
Yang
,
Y.
,
Chadha
,
M.
,
Hu
,
Z.
, and
Todd
,
M. D.
,
2022
, “
An Optimal Sensor Placement Design Framework for Structural Health Monitoring Using Bayes Risk
,”
Mech. Syst. Signal Process.
,
168
, p.
108618
.10.1016/j.ymssp.2021.108618
35.
Dash
,
A.
,
Ye
,
J.
, and
Wang
,
G.
,
2021
, “
A Review of Generative Adversarial Networks (GANs) and Its Applications in a Wide Variety of Disciplines - From Medical to Remote Sensing
,”
IEEE Access.
, 12, pp.
18330
18357
.10.1109/ACCESS.2023.3346273
36.
Sandfort
,
V.
,
Yan
,
K.
,
Pickhardt
,
P. J.
, and
Summers
,
R. M.
,
2019
, “
Data Augmentation Using Generative Adversarial Networks (CycleGAN) to Improve Generalizability in CT Segmentation Tasks
,”
Sci. Rep.
,
9
(
1
), p.
16884
.10.1038/s41598-019-52737-x
37.
Goodfellow
,
I. J.
,
Pouget-Abadie
,
J.
,
Mirza
,
M.
,
Xu
,
B.
,
Warde-Farley
,
D.
,
Ozair
,
S.
,
Courville
,
A.
, and
Bengio
,
Y.
,
2014
, “
Generative Adversarial Networks
,” preprint
arXiv:1406.2661
.10.48550/arXiv.1406.2661
38.
Kong
,
H.-J.
,
Kim
,
J. Y.
,
Moon
,
H.-M.
,
Park
,
H. C.
,
Kim
,
J.-W.
,
Lim
,
R.
,
Woo
,
J.
,
Fakhri
,
G. E.
,
Kim
,
D. W.
, and
Kim
,
S.
,
2022
, “
Automation of Generative Adversarial Network-Based Synthetic Data-Augmentation for Maximizing the Diagnostic Performance With Paranasal Imaging
,”
Sci. Rep.
,
12
(
1
), p.
18118
.10.1038/s41598-022-22222-z
39.
Zhang
,
T.
, and
Wang
,
Y.
,
2019
, “
Deep Learning Algorithms for Structural Condition Identification With Limited Monitoring Data
,” International Conference on Smart Infrastructure and Construction 2019 (
ICSIC
)
, Churchill College, Cambridge, July 8–10, pp.
421
426
.10.1680/icsic.64669.421
40.
Heesch
,
M.
,
Mendrok
,
K.
, and
Dworakowski
,
Z.
,
2021
, “
Time-Domain Signal Synthesis With Style-Based Generative Adversarial Networks Applied to Guided Waves
,”
Artificial Intelligence and Soft Computing: 20th International Conference
, ICAISC 2021, Virtual Event, June 21–23, pp.
78
88
.10.1007/978-3-030-87986-0_7
41.
Luleci
,
F.
,
Catbas
,
F. N.
, and
Avci
,
O.
,
2023
, “
Generative Adversarial Networks for Labeled Acceleration Data Augmentation for Structural Damage Detection
,”
J. Civ. Struct. Health Monit.
,
13
(
1
), pp.
181
198
.10.1007/s13349-022-00627-8
42.
Zhu
,
J.
,
Park
,
T.
,
Isola
,
P.
, and
Efros
,
A. A.
,
2017
, “
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
,” 2017 IEEE International Conference on Computer Vision (
ICCV
), Venice, Italy, Oct. 22–29, pp.
2242
2251
.10.1109/ICCV.2017.244
43.
Luleci
,
F.
,
Catbas
,
F. N.
, and
Avci
,
O.
,
2022
, “
A Literature Review: Generative Adversarial Networks for Civil Structural Health Monitoring
,”
Front. Built Environ.
,
8
(
1
), p.
1027379
.10.3389/fbuil.2022.1027379
44.
Cohen
,
N.
,
Sood
,
S.
,
Zeng
,
Z.
,
Balch
,
T.
, and
Veloso
,
M.
,
2020
, “
Visual Forecasting of Time Series With Image-to-Image Regression
,” preprint
arXiv:2011.09052
.10.1145/3490354.3494387
45.
Ronneberger
,
O.
,
Fischer
,
P.
, and
Brox
,
T.
,
2015
, “
U-Net: Convolutional Networks for Biomedical Image Segmentation
,” Medical Image Computing and Computer-Assisted Intervention (
MICCAI
), Munich, Germany, Oct. 5–9, pp. 234–241.10.1007/978-3-319-24574-4_28
46.
Kiranyaz
,
S.
,
Malik
,
J.
,
Abdallah
,
H. B.
,
Ince
,
T.
,
Iosifidis
,
A.
, and
Gabbouj
,
M.
,
2021
, “
Self-Organized Operational Neural Networks With Generative Neurons
,”
Neural Networks.
, 140, pp.
294
308
.10.1016/j.neunet.2021.02.028
47.
Brunet
,
D.
,
Vrscay
,
E. R.
, and
Wang
,
Z.
,
2012
, “
On the Mathematical Properties of the Structural Similarity Index
,”
IEEE Trans. Image Process.
,
21
(
4
), pp.
1488
1499
.10.1109/TIP.2011.2173206
48.
Chong
,
M. J.
, and
Forsyth
,
D.
,
2020
, “
Effectively Unbiased FID and Inception Score and Where to Find Them
,”
IEEE/CVF Conference on Computer Vision and Pattern Recognition
(
CVPR
), Seattle, WA, June 13–19, pp.
6070
6079
.10.1109/CVPR42600.2020.00611
49.
Ao
,
D.
,
Hu
,
Z.
, and
Mahadevan
,
S.
,
2017
, “
Dynamics Model Validation Using Time-Domain Metrics
,”
ASME J. Verif., Valid. Uncertainty Quantif.
,
2
(
1
), p.
011004
.10.1115/1.4036182
You do not currently have access to this content.