Skip to main content
Skip to article control options
No AccessFull-Length Papers

Mixed Variable Gaussian Process-Based Surrogate Modeling Techniques: Application to Aerospace Design

Published Online:https://doi.org/10.2514/1.I010965

Within the framework of complex system analyses, such as aircraft and launch vehicles, the presence of computationally intensive models (e.g., finite element models and multidisciplinary analyses) coupled with the dependence on discrete and unordered technological design choices results in challenging modeling problems. In this paper, the use of Gaussian process surrogate modeling of mixed continuous/discrete functions and the associated challenges are extensively discussed. A unifying formalism is proposed in order to facilitate the description and comparison between the existing covariance kernels allowing to adapt Gaussian processes to the presence of discrete unordered variables. Furthermore, the modeling performances of these various kernels are tested and compared on a set of analytical and aerospace-engineering-design-related benchmarks with different characteristics and parameterizations. Eventually, general tendencies and recommendations for such types of modeling problem using Gaussian process are highlighted.

References

  • [1] Queipo N. V., Haftka R. T., Shyy W., Goel T., Vaidyanathan R. and Kevin Tucker P., “Surrogate-Based Analysis and Optimization,” Progress in Aerospace Sciences, Vol. 41, No. 1, 2005, pp. 1–28. https://doi.org/10.1016/j.paerosci.2005.02.001 CrossrefGoogle Scholar

  • [2] Draper N. R. and Smith H., Applied Regression Analysis, Vol. 326, Wiley, New York, 1998, Chap. 9. CrossrefGoogle Scholar

  • [3] Papadrakakis M., Lagaros N. D. and Tsompanakis Y., “Structural Optimization Using Evolution Strategies and Neural Networks,” Computer Methods in Applied Mechanics and Engineering, Vol. 156, Nos. 1–4, 1998, pp. 309–333. https://doi.org/10.1016/S0045-7825(97)00215-6 CrossrefGoogle Scholar

  • [4] Dyn N., Levin D. and Rippa S., “Numerical Procedures for Surface Fitting of Scattered Data by Radial Functions,” SIAM Journal on Scientific and Statistical Computing, Vol. 7, No. 2, 1986, pp. 639–659. https://doi.org/10.1137/0907043 CrossrefGoogle Scholar

  • [5] Fang H. and Horstemeyer M. F., “Global Response Approximation with Radial Basis Functions,” Engineering Optimization, Vol. 38, No. 4, 2006, pp. 407–424. https://doi.org/10.1080/03052150500422294 CrossrefGoogle Scholar

  • [6] Smola A. J. and Schölkopf B., “A Tutorial on Support Vector Regression,” Statistics and Computing, Vol. 14, No. 3, 2004, pp. 199–222. https://doi.org/10.1023/B:STCO.0000035301.49549.88 CrossrefGoogle Scholar

  • [7] Friedman J. H., “Multivariate Adaptive Regression Splines,” Annals of Statistics, Vol. 19, No. 1, 1991, pp. 1–67. CrossrefGoogle Scholar

  • [8] Wang G. G. and Shan S., “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” Journal of Mechanical Design, Vol. 129, No. 4, 2007, p. 370. https://doi.org/10.1115/1.2429697 CrossrefGoogle Scholar

  • [9] Rasmussen C. E. and Williams C. K. I., Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, 2006, pp. 63–71. Google Scholar

  • [10] Bartz-Beielstein T. and Zaefferer M., “Model-Based Methods for Continuous and Discrete Global Optimization,” Applied Soft Computing, Vol. 55, June 2017, pp. 154–167. https://doi.org/10.1016/j.asoc.2017.01.039 CrossrefGoogle Scholar

  • [11] Meckesheimer M., Barton R. R., Simpson T., Limayem F. and Yannou B., “Metamodeling of Combined Discrete/Continuous Responses,” AIAA Journal, Vol. 39, No. 10, 2001, pp. 1950–1959. https://doi.org/10.2514/2.1185 LinkGoogle Scholar

  • [12] Swiler L. P., Hough P. D., Qian P., Xu X., Storlie C. and Lee H., “Surrogate Models for Mixed Discrete-Continuous Variables,” Constraint Programming and Decision Making, Springer, Berlin, 2014, pp. 181–202. CrossrefGoogle Scholar

  • [13] Zhou Q., Qian P. Z. G. and Zhou S., “A Simple Approach to Emulation for Computer Models With Qualitative and Quantitative Factors,” Technometrics, Vol. 53, No. 3, 2011, pp. 266–273. https://doi.org/10.1198/TECH.2011.10025 CrossrefGoogle Scholar

  • [14] Alvarez M. A., Rosasco L. and Lawrence N. D., “Kernels for Vector-Valued Functions: A Review,” Foundations and Trends in Machine Learning, Vol. 4, No. 3, 2012, pp. 195–266. CrossrefGoogle Scholar

  • [15] Zhang Y. and Notz W. I., “Computer Experiments with Qualitative and Quantitative Variables: A Review and Reexamination,” Quality Engineering, Vol. 27, No. 1, 2015, pp. 2–13. https://doi.org/10.1080/08982112.2015.968039 CrossrefGoogle Scholar

  • [16] Roustant O., Padonou E., Deville Y., Clément A., Perrin G., Giorla J. and Wynn H., “Group Kernels for Gaussian Process Metamodels with Categorical Inputs,” arXiv preprint, arXiv:1802.02368, 2018. Google Scholar

  • [17] Oliver M. A. and Webster R., “Kriging: A Method of Interpolation for Geographical Information Systems,” International Journal of Geographical Information Systems, Vol. 4, No. 3, 1990, pp. 313–332. https://doi.org/10.1080/02693799008941549 CrossrefGoogle Scholar

  • [18] Sacks J., Welch W. J., Mitchell T. J. and Wynn H. P., “Design and Analysis of Computer Experiments,” Statistical Science, Vol. 4, No. 4, 1989, pp. 409–423. CrossrefGoogle Scholar

  • [19] Simpson T. W., Peplinski J. D., Koch P. N. and Allen J. K., “Metamodels for Computer-Based Engineering Design: Survey and Recommendations,” Engineering with Computers, Vol. 17, No. 2, 2001, pp. 129–150. https://doi.org/10.1007/PL00007198 CrossrefGoogle Scholar

  • [20] Aronszajn N., “Theory of Reproducing Kernels,” Transactions of the American Mathematical Society, Vol. 68, No. 3, 1950, pp. 337–404. https://doi.org/10.1090/S0002-9947-1950-0051437-7 CrossrefGoogle Scholar

  • [21] Steinwart I. and Christmann A., Support Vector Machines, Springer Science & Business Media, Berlin, 2008, pp. 110–163. Google Scholar

  • [22] Scholkopf B. and Smola A. J., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, 2001, Chap. 3. Google Scholar

  • [23] Santner T. J., Williams B. J. and Notz W. I., The Design and Analysis of Computer Experiments, Springer, New York, 2003, pp. 27–66. CrossrefGoogle Scholar

  • [24] Suits D. B., “Use of Dummy Variables in Regression Equations,” Journal of the American Statistical Association, Vol. 52, No. 280, 1957, pp. 548–551. https://doi.org/10.1080/01621459.1957.10501412 CrossrefGoogle Scholar

  • [25] Halstrup M., “Black-Box Optimization of Mixed Discrete-Continuous Optimization Problems,” Ph.D. Thesis, TU Dortmund, Dortmund, Germany, 2016. Google Scholar

  • [26] Hutter F., “Automated Configuration of Algorithms for Solving Hard Computational Problems,” Ph.D. Thesis, Univ. of British Columbia, Vancouver, 2009. Google Scholar

  • [27] Gower J. C., “A General Coefficient of Similarity and Some of Its Properties,” Biometrics, Vol. 27, No. 4, 1971, p. 857. https://doi.org/10.2307/2528823 CrossrefGoogle Scholar

  • [28] Zhang Y., Tao S., Chen W. and Apley D. W., “A Latent Variable Approach to Gaussian Process Modeling with Qualitative and Quantitative Factors,” Technometrics, Vol. 62, No. 3, 2019, pp. 1–12. Google Scholar

  • [29] Journel A. G. and Huijbregts C. J., Mining Geostatistics, Vol. 600, Academic Press, London, 1978, Chap. 2. Google Scholar

  • [30] Goovaerts P., Geostatistics for Natural Resources Evaluation, Oxford Univ. Press, Oxford, 1997, Chap. 5. Google Scholar

  • [31] Pinheiro J. and Bates D. M., “Unconstrained Parametrizations for Variance-Covariance Matrices,” Statistics and Computing, Vol. 6, No. 3, 1996, pp. 289–296. https://doi.org/10.1007/BF00140873 CrossrefGoogle Scholar

  • [32] Qian P. Z. G., Wu H. and Wu C. F. J., “Gaussian Process Models for Computer Experiments with Qualitative and Quantitative Factors,” Technometrics, Vol. 50, No. 3, 2008, pp. 383–396. https://doi.org/10.1198/004017008000000262 CrossrefGoogle Scholar

  • [33] Byrd R. H., Lu P., Nocedal J. and Zhu C., “A Limited Memory Algorithm for Bound Constrained Optimization,” SIAM Journal on Scientific Computing, Vol. 16, No. 5, 1995, pp. 1190–1208. https://doi.org/10.1137/0916069 CrossrefGoogle Scholar

  • [34] Storn R. and Price K., “Differential Evolution—A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces,” Journal of Global Optimization, Vol. 11, No. 4, 1997, pp. 341–359. https://doi.org/10.1023/A:1008202821328 CrossrefGoogle Scholar

  • [35] Hansen N., “Towards a New Evolutionary Computation: Advances in the Estimation of Distribution Algorithms,” Towards a New Evolutionary Computation, Springer, Berlin, 2006, pp. 75–102. CrossrefGoogle Scholar

  • [36] McKay M. D., Beckman R. and Conover W., “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code,” Technometrics, Vol. 21, No. 2, 1979, p. 239. https://doi.org/10.1080/00401706.2000.10485979 CrossrefGoogle Scholar

  • [37] Deng X., Hung Y. and Lin C. D., “Design for Computer Experiments with Qualitative and Quantitative Factors,” Statistica Sinica, Vol. 25, No. 4, Oct. 2015, pp. 1567–1581. Google Scholar

  • [38] Forrester A., Sobester A. and Keane A., Engineering Design via Surrogate Modelling: A Practical Guide, Wiley, Hoboken, NJ, 2008, p. 196. CrossrefGoogle Scholar

  • [39] Picheny V., Wagner T. and Ginsbourger D., “A Benchmark of Kriging-Based Infill Criteria for Noisy Optimization,” Structural and Multidisciplinary Optimization, Vol. 48, No. 3, 2013, pp. 607–626. https://doi.org/10.1007/s00158-013-0919-4 CrossrefGoogle Scholar

  • [40] Deng X., Lin C. D., Liu K. W. and Rowe R. K., “Additive Gaussian Process for Computer Models with Qualitative and Quantitative Factors,” Technometrics, Vol. 59, No. 3, 2017, pp. 283–292. https://doi.org/10.1080/00401706.2016.1211554 CrossrefGoogle Scholar

  • [41] McBride B. J. and Gordon S., “Computer Program for Calculation of Complex Chemical Equilibrium Compositions and Applications. User Manual and Program Description,” NASA Reference Publication 1311, 1996. Google Scholar

  • [42] Levine I. N., Physical Chemistry, McGraw–Hill, New York, 2009, Chap. 5. Google Scholar

  • [43] Bonnans J.-F., Gilbert J. C. and Lemarechal C., Numerical Optimization: Theoretical and Practical Aspects, Springer, Berlin, 2006, Chap. 4. Google Scholar

  • [44] Ford H. and Alexander J. M., Advanced Mechanics of Materials, Wiley , New York, 1977, Chap. 2. Google Scholar

  • [45] MacNeal R. H. and McCormick C. W., “The NASTRAN Computer Program for Structural Analysis,” Computers and Structures, Vol. 1, No. 3, Oct. 1971, pp. 389–412. CrossrefGoogle Scholar

  • [46] Jones D. R., Schonlau M. and Welch W. J., “Efficient Global Optimization of Expensive Black-Box Functions,” Journal of Global Optimization, Vol. 13, Dec. 1998, pp. 455–492. https://doi.org/10.1023/A:1008306431147 CrossrefGoogle Scholar

  • [47] Ho Y.-C. and Pepyne D. L., “Simple Explanation of the No-Free-Lunch Theorem and Its Implications,” Journal of Optimization Theory and Applications, Vol. 115, No. 3, 2002, pp. 549–570. https://doi.org/10.1023/A:1021251113462 CrossrefGoogle Scholar

  • [48] Pelamatti J., Brevault L., Balesdent M., Talbi E.-G. and Guerin Y., “Efficient Global Optimization of Constrained Mixed Variable Problems,” Journal of Global Optimization, Vol. 73, No. 3, 2019, pp. 583–613. https://doi.org/10.1007/s10898-018-0715-1 CrossrefGoogle Scholar

  • [49] Matthews A. G. D. G., van der Wilk M., Nickson T., Fujii K., Boukouvalas A., León-Villagrá P., Ghahramani Z. and Hensman J., “GPflow: A Gaussian Process Library Using Tensor Flow,” Journal of Machine Learning Research, Vol. 18, No. 40, 2017, pp. 1–6. Google Scholar

  • [50] Abadi M., Agarwal A., Barham P., Brevdo E., Chen Z., Citro C., Corrado G. S., Davis A., Dean J., Devin M., Ghemawat S., Goodfellow I., Harp A., Irving G., Isard M., Jia Y., Jozefowicz R., Kaiser L., Kudlur M., Levenberg J., Mané D., Monga R., Moore S., Murray D., Olah C., Schuster M., Shlens J., Steiner B., Sutskever I., Talwar K., Tucker P., Vanhoucke V., Vasudevan V., Viégas F., Vinyals O., Warden P., Wattenberg M., Wicke M., Yu Y. and Zheng X., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” 2015, https://www.tensorflow.org/. Google Scholar