(Photo by Tara Winstead at Pexels.com)
PLEASE JOIN US this July 12th for a special HYBRID tutorial on how to develop an open source Generative Artificial Intelligence (GenAI) Large Language Model (LLM) for systems engineering. This tutorial will be conducted in person, and simultaneously available online via Zoom.
Large Language Models are a type of artificial intelligence model trained on massive amounts of data to understand and generate human-like language (examples include ChatGPT, Gemini and Llama). These models can also be customized or fine-tuned to support systems engineering tasks and problem-solving. This tutorial will show students how to customize an open source LLM for SE.
This tutorial will be taught by Dr. Ray Madachy of the Naval Post Graduate School (NPS), and graduate students Mr. Ryan Bell and Mr. Ryan Longshore.
MAIN INFO
Date: Saturday, July 12, 2025, 9:30am – 3pm (Pacific Time)
Location: SAIC office, 1615 Murray Canyon Rd, 2nd floor, San Diego, CA 92108 (google maps)
Cost: In-person prices are $50 INCOSE members, $75 non-INCOSE members, and $25 students. Remote attendance price is $25. This price includes morning coffee, soft drinks and lunch (sandwiches etc.) for in-person attendees (please contact us if you have special dietary preferences).
Parking: There is free parking on the street adjacent the office building.
Remote Participation: We will use Zoom for remote students. After you register, Zoom access details will be sent to you via email prior to the class.
Foreign Nationals – Foreign Nationals may attend, but must register by at least June 20th and let us know in the registration Notes textbox so that we may get SAIC’s approval. Please email info@sdincose.org pertinent information such as your legal status in the USA and which country you hold citizenship in. Thank you!
SYNOPSIS
Join us for an immersive tutorial where we leverage the power of custom Large Language Models (LLMs) to tackle the complexities of systems engineering. This hands-on session is designed for systems engineers who are eager to explore how LLMs can revolutionize their workflows, enhance model accuracy, and address domain-specific challenges in managing socio-technical systems.
Participants will utilize Llama factory to design and fine-tune their own LLMs using a specialized systems engineering dataset. You’ll not only gain valuable experience in building these models but also work collaboratively to create a robust evaluation dataset. Together, we will put these models to the test using our custom evaluation dataset and SysEngBench, a benchmark specifically crafted for the systems engineering domain!
WHAT YOU WILL LEARN
By the end of the workshop, you will understand how to:
- Tailor LLM performance
- Strike a balance between accuracy and computational efficiency, and
- Learn how to create AI solutions to elevate the practice of systems engineering
The goal is to walk away with practical, cutting-edge skills and a strong foundation to make impactful contributions to the future of SE.
TUTORIAL AGENDA
0930-1030: Introduction to Custom Generative Pre-trained Transformer (GPT)s
- Overview of LLMs in Systems Engineering
- Setting up the environment and tools
- Case Studies
1030-1045 BREAK
1045-1200: RAG Pipeline with Documents
- Hands-on walk through of using a model with non-public documents
- Participants work on using RAG with a language model with guidance
1200-1300 LUNCH
1300-1400: Creating Evaluation Datasets
- Collaborative session to develop evaluation datasets
- Types of evaluation tasks: multiple-choice, fill-in-the-blank, textual response prompts
1400-1415 BREAK
1415-1530: Model Evaluation with Benchmarking
- Evaluating models with benchmarks
- Case Study: Evaluate Llama 3.2 1B on SysEngBench
- (optional) Case Study: Evaluate Llama 3.2 1B on Hour 3’s dataset
- Wrap-up and Q&A
WHAT YOU WILL NEED
This tutorial will use Google Colab Jupyter Notebook. It would be advantageous for everyone who attends in-person to have a laptop. This is recommend for remote students as well.
Please note that to use Google Colab that you may need a gmail email account. If you do not have a gmail, you can create one for free.
We will also include all of the setup instructions in the notebook, so those in the session can follow along real time.
INSTRUCTORS
Raymond Madachy
Raymond Madachy, Ph.D., is a Professor in the Systems Engineering Department at the Naval Postgraduate School. His research interests include systems engineering tool environments for digital engineering, modeling and simulation of systems and software engineering processes, generative AI, and system cost modeling. He has developed widely used cost estimation tools for systems and software engineering, and created the Systems Engineering Library (se-lib). His books include Software Process Dynamics, What Every Engineer Should Know about Modeling and Simulation, What Every Engineer Should Know about Python; and co-authored Software Cost Estimation with COCOMO II and Software Cost Estimation Metrics Manual for Defense Systems.
Ryan Bell
Ryan Bell is an 9 year experienced engineer in the defense industry. In his current role at Naval Information Warfare Center Atlantic (NIWC LANT), Ryan provides modeling and simulation expertise to a variety of programs for the Navy and USMC. He specializes in simulating communication systems in complex environments and is an advocate for the use of digital engineering early in the systems engineering lifecycle. Ryan earned a BS in Electrical Engineering from Clemson University, a MS in Electrical Engineering from Clemson University with a focus on Electronics, and is currently pursuing his PhD in Systems Engineering at the Naval Postgraduate School. He is a South Carolina registered Professional Engineer (PE), published author, and teacher.
Ryan Longshore
Ryan Longshore is a 20-year veteran of the defense and electric utility industries. At Naval Information Warfare Center Atlantic, he leads teams developing and integrating new technologies into Navy command centers. He is involved in the Navy’s digital engineering transformation with a focus on model-based systems and model-based engineering. Ryan holds a BS in Electrical Engineering (Clemson), a MS in Systems Engineering (SMU), and is pursuing a PhD is Systems Engineering at the Naval Postgraduate School. He is a South Carolina Registered Professional Engineer, an INCOSE CSEP, and holds the OMG SysML Model Builder Fundamental Certification.