Abstract

Material selection plays a pivotal role in many industries, from manufacturing to construction. Material selection is usually carried out after several cycles of conceptual design, during which designers iteratively refine the design solution and the intended manufacturing approach. In design research, material selection is frequently modeled as a structured decision-making process, where optimization techniques, whether single-objective or multiobjective, are employed to identify solutions that best meet the design requirements. However, traditional approaches can be limited by their reliance on existing knowledge and data, which may not adequately capture the full range of considerations involved in material selection. In this article, we introduce MSEval, a novel dataset comprised expert material evaluations across a variety of design briefs and criteria. This dataset is designed to serve as a benchmark to facilitate the evaluation and modification of machine learning models in the context of material selection for conceptual design. By focusing on a diverse set of design tasks and criteria, MSEval enables a more nuanced understanding of the material selection and the thought process, providing valuable insights for both human designers and AI systems.

References

1.
Brown
,
T. B.
,
Mann
,
B.
,
Ryder
,
N.
,
Subbiah
,
M.
,
Kaplan
,
J.
,
Dhariwal
,
P.
, and
Neelakantan
,
A.
, et al.,
2020
, “Language Models Are Few-Shot Learners,” arXiv:2005.14165 [cs], http://arxiv.org/abs/2005.14165. Accessed May 27, 2024.
2.
Vaswani
,
A.
,
Shazeer
,
N.
,
Parmar
,
N.
,
Uszkoreit
,
J.
,
Jones
,
L.
,
Gomez
,
A. N.
,
Kaiser
,
L.
, and
Polosukhin
,
I.
,
2023
, “Attention Is All You Need,” arXiv:1706.03762 [cs], http://arxiv.org/abs/1706.03762. Accessed April 3 2024.
3.
Wei
,
J.
,
Tay
,
Y.
,
Bommasani
,
R.
,
Raffel
,
C.
,
Zoph
,
B.
,
Borgeaud
,
S.
, and
Yogatama
,
D.
, et al.,
2022
, “Emergent Abilities of Large Language Models,” preprint arXiv:2206.07682.
4.
OpenAI
,
2023
, “GPT-4 Technical Report,” 2303.08774.
5.
Touvron
,
H.
,
Lavril
,
T.
,
Izacard
,
G.
,
Martinet
,
X.
,
Lachaux
,
M.-A.
,
Lacroix
,
T.
, and
Rozière
,
B.
, et al.,
2023
, “Llama: Open and Efficient Foundation Language Models,” preprint arXiv:2302.13971.
6.
Driess
,
D.
,
Xia
,
F.
,
Sajjadi
,
M. S.
,
Lynch
,
C.
,
Chowdhery
,
A.
,
Ichter
,
B.
, and
Wahid
,
A.
, et al.,
2023
, “PaLM-E: An Embodied Multimodal Language Model,” preprint arXiv:2303.03378.
7.
Ma
,
K.
,
Grandi
,
D.
,
McComb
,
C.
, and
Goucher-Lambert
,
K.
,
2023
, “
Material Selection Using Large Language Models
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Volume 16838,
Boston, MA
,
Aug. 20–23
,
American Society of Mechanical Engineers
, p.
V006T06A021
.
8.
Zhu
,
Q.
, and
Luo
,
J.
,
2023
, “
Generative Transformers for Design Concept Generation
,”
ASME J. Comput. Inf. Sci. Eng.
,
23
(
4
), p.
041003
.
9.
Jiang
,
S.
,
Sarica
,
S.
,
Song
,
B.
,
Hu
,
J.
, and
Luo
,
J.
,
2022
, “
Patent Data for Engineering Design: A Critical Review and Future Directions
,”
J. Comput. Inf. Sci. Eng.
,
22
(
6
), p.
060902
.
10.
Picard
,
C.
,
Edwards
,
K. M.
,
Doris
,
A. C.
,
Man
,
B.
,
Giannone
,
G.
,
Alam
,
M. F.
, and
Ahmed
,
F.
,
2023
, “From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design,” preprint arXiv:2311.12668.
11.
Picard
,
C.
, and
Ahmed
,
F.
,
2024
, “
Untrained and Unmatched: Fast and Accurate Zero-Training Classification for Tabular Engineering Data
,”
J. Mech. Des.
,
146
(
9
), pp.
1
15
.
12.
Meltzer
,
P.
,
Lambourne
,
J. G.
, and
Grandi
,
D.
,
2024
, “
What’s in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models Through User-Provided Names in Computer Aided Design Files
,”
J. Comput. Inf. Sci. Eng.
,
24
(
1
), p.
011002
.
13.
Song
,
B.
,
Zhou
,
R.
, and
Ahmed
,
F.
,
2024
, “
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
,”
J. Comput. Inf. Sci. Eng.
,
24
(
1
), p.
010801
.
14.
Zhong
,
W.
,
Cui
,
R.
,
Guo
,
Y.
,
Liang
,
Y.
,
Lu
,
S.
,
Wang
,
Y.
,
Saied
,
A.
,
Chen
,
W.
, and
Duan
,
N.
,
2023
, “AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models,” arXiv:2304.06364 [cs], http://arxiv.org/abs/2304.06364. Accessed May 27, 2024.
15.
Zarandi
,
M. H. F.
,
Mansour
,
S.
,
Hosseinijou
,
S. A.
, and
Avazbeigi
,
M.
,
2011
, “
A Material Selection Methodology and Expert System for Sustainable Product Design
,”
Int. J. Adv. Manuf. Technol.
,
57
(
9
), pp.
885
903
.
16.
Bi
,
L.
,
Zuo
,
Y.
,
Tao
,
F.
,
Liao
,
T. W.
, and
Liu
,
Z.
,
2017
, “
Energy-Aware Material Selection for Product With Multicomponent Under Cloud Environment
,”
J. Comput. Inf. Sci. Eng.
,
17
(
3
), p.
031007
.
17.
“Materials in Products Selection: Tools for Including User-Interaction in Materials Selection,” https://www.ijdesign.org/index.php/IJDesign/article/view/129. Accessed March 12, 2024.
18.
Ashby
,
M. F.
,
Bréchet
,
Y. J. M.
,
Cebon
,
D.
, and
Salvo
,
L.
,
2004
, “
Selection Strategies for Materials and Processes
,”
Mater. Des.
,
25
(
1
), pp.
51
67
.
19.
Bhat
,
B. N.
, ed.,
2018
,
Aerospace Materials and Applications
,
American Institute of Aeronautics and Astronautics Inc.
,
Reston, VA
.
20.
Ashby
,
M. F.
,
1989
, “
Materials Selection in Conceptual Design
,”
Mater. Sci. Technol.
,
5
(
6
), pp.
517
525
.
21.
Prabhu
,
R.
,
Leguarda
,
R. L.
,
Miller
,
S. R.
,
Simpson
,
T. W.
, and
Meisel
,
N. A.
,
2021
, “
Favoring Complexity: A Mixed Methods Exploration of Factors That Influence Concept Selection When Designing for Additive Manufacturing
,”
J. Mech. Des.
,
143
(
10
), p.
102001
.
22.
Karandikar
,
H. M.
, and
Mistree
,
F.
,
1992
, “
An Approach for Concurrent and Integrated Material Selection and Dimensional Synthesis
,”
J. Mech. Des.
,
114
(
4
), pp.
633
641
.
23.
Ashby
,
M. F.
,
2011
,
Materials Selection in Mechanical Design
, 4th ed.,
Butterworth-Heinemann
,
Burlington, MA
. OCLC: ocn639573741.
24.
Shiau
,
C.-S. N.
, and
Michalek
,
J. J.
,
2009
, “
Optimal Product Design Under Price Competition
,”
J. Mech. Des.
,
131
(
7
), p.
071003
.
25.
Hazelrigg
,
G. A.
,
1997
, “
On Irrationality in Engineering Design
,”
J. Mech. Des.
,
119
(
2
), pp.
194
196
.
26.
Jain
,
Y.
,
Grandi
,
D.
,
Groom
,
A.
,
Cramer
,
B.
, and
McComb
,
C.
,
2024
, “
Conceptual Design Generation Using Large Language Models
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Washington, DC
,
Aug. 25–28
,
American Society of Mechanical Engineers
.
27.
Grandi
,
D.
,
Jain
,
Y.
,
Groom
,
A.
,
Cramer
,
B.
, and
McComb
,
C.
,
2024
, “
Evaluating Large Language Models for Material Selection
,”
J. Comput. Inf. Sci. Eng.
,
25
(
2
), p.
021004
.
28.
Rajpurkar
,
P.
,
Zhang
,
J.
,
Lopyrev
,
K.
, and
Liang
,
P.
,
2016
, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” 1606.05250.
29.
Bowman
,
S. R.
,
Angeli
,
G.
,
Potts
,
C.
, and
Manning
,
C. D.
,
2015
, “
A Large Annotated Corpus for Learning Natural Language Inference
,”
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
,
Lisbon, Portugal
,
Sept. 17–21
,
Association for Computational Linguistics
, pp.
632
642
. https://10.18653/v1/D15-1075
30.
Wang
,
A.
,
Singh
,
A.
,
Michael
,
J.
,
Hill
,
F.
,
Levy
,
O.
, and
Bowman
,
S.
,
2018
, “
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
,”
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
,
Brussels, Belgium
,
Nov. 1
,
Association for Computational Linguistics
, pp.
353
355
.
31.
Wang
,
A.
,
Pruksachatkun
,
Y.
,
Nangia
,
N.
,
Singh
,
A.
,
Michael
,
J.
,
Hill
,
F.
,
Levy
,
O.
, and
Bowman
,
S. R.
,
2020
, “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems,” arXiv:1905.00537 [cs], http://arxiv.org/abs/1905.00537. Accessed June 20, 2024.
32.
Krithara
,
A.
,
Nentidis
,
A.
,
Bougiatiotis
,
K.
, and
Paliouras
,
G.
,
2023
, “
BioASQ-QA: A Manually Curated Corpus for Biomedical Question Answering
,”
Scientific Data
,
10
(
1
), p.
170
.
33.
Jin
,
D.
,
Pan
,
E.
,
Oufattole
,
N.
,
Weng
,
W.-H.
,
Fang
,
H.
, and
Szolovits
,
P.
,
2020
, “What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset From Medical Exams,” arXiv:2009.13081 [cs], http://arxiv.org/abs/2009.13081. Accessed June 20, 2024.
34.
Petrova
,
A.
,
2022
, “chinmusique/outcome-prediction,” original-date: 2020-09-21T11:10:44Z, https://github.com/chinmusique/outcome-prediction. Accessed June 20 2024.
35.
“Materials Data Repository Home.” https://materialsdata.nist.gov/. Accessed June 20, 2024.
36.
“Materials Project - Home,” https://next-gen.materialsproject.org/. Accessed June 20, 2024.
37.
Wilkinson
,
M. D.
,
Dumontier
,
M.
,
Aalbersberg
,
I. J.
,
Appleton
,
G.
,
Axton
,
M.
,
Baak
,
A.
, and
Blomberg
,
N.
, et al.,
2016
, “
The FAIR Guiding Principles for Scientific Data Management and Stewardship
,”
Sci. Data
,
3
(
1
), pp.
1
9
.
38.
Design Research Collective
,
2024
, “Material_Selection_Eval (Revision 30493e2),” doi:10.57967/hf/2318, https://huggingface.co/datasets/cmudrc/Material_Selection_Eval.
39.
Lhoest
,
Q.
,
Villanova del Moral
,
A.
,
von Platen
,
P.
,
Wolf
,
T.
,
Šaško
,
M.
,
Jernite
,
Y.
, and
Thakur
,
A.
, et al.,
2021
, “Datasets: A Community Library for Natural Language Processing,” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.: 175–184, https://aclanthology.org/2021.emnlp-demo.21. Accessed June 20, 2024.
You do not currently have access to this content.