FSW 2021: cFS Test Framework (CTF) - Aly Shehata

Summary

The CFS Test Framework (CTF) is a powerful Python-based tool designed for automated testing and verification of flight software in Core Flight Systems (CFS). It enables users to execute tests through customizable JSON scripts, interacting with external systems via a plugin architecture. Key features include core functionality management, plugin extensibility, and a graphical editor for test script creation. CTF supports test execution, telemetry verification, and command validation, offering an efficient solution for flight software testing. It is open-source, continuously evolving, and actively used by NASA for CFS validation and certification.

Takeaways

• 😀 CTF (CFS Test Framework) is a tool for automated testing and verification of CFS-based flight software systems.
• 😀 CTF supports test scripts written in JSON, which include test metadata and instructions for execution.
• 😀 The framework interfaces with external systems such as CFS cores, executing tests and capturing the results.
• 😀 CTF is written in Python and uses a plugin system, allowing customization and extension for specific needs.
• 😀 The core components of CTF include a plugin manager, default plugins (e.g., CFS, CCSDS, SSH), and a graphical user interface (CTF Editor).
• 😀 CTF utilizes INI-format configuration files to define test setup, including build directories, target IPs, and plugin-specific settings.
• 😀 Key CTF plugins include the CFS plugin (for commanding and telemetry verification), the CCSDS plugin (for header and message management), and the SSH plugin (for remote system communication).
• 😀 Test scripts consist of multiple test cases and instructions, such as sending CFS commands, checking telemetry, and verifying event messages.
• 😀 CTF's graphical user interface (CTF Editor) enables easy editing, running, and monitoring of test scripts with drag-and-drop functionality.
• 😀 CTF is an open-source tool available on GitHub, with active development and ongoing support for users across various programs, including Gateway.

Q & A

What is the purpose of the CFS Test Framework (CTF)?

The CFS Test Framework (CTF) is a tool designed to enable automated testing and verification of CFS-based flight software systems. It executes test scripts, interfaces with external systems, and captures/logs test results.

How does CTF interact with test scripts?

CTF loads test scripts written in JSON format, which contain test metadata and instructions. It then executes these scripts against a selected test configuration while interacting with the relevant external systems.

What programming language is CTF written in and why is this significant?

CTF is written in Python, and it utilizes a Python-based plugin system. This allows developers to extend its functionality by creating custom plugins for specific testing needs.

What is the role of the CTF plugin manager?

The CTF plugin manager is responsible for loading and managing plugins. It ensures that each plugin interacts with the core CTF framework and other plugins, providing core functionality for the framework.

Can users create their own plugins in CTF?

Yes, users can create their own plugins or use existing ones. The plugins have a well-defined interface for interaction with the core CTF framework, making it easy to extend or customize CTF functionality.

What are some of the default plugins provided by CTF?

CTF provides several default plugins, including the CFS plugin (for commanding and telemetry verification), the CCSDS plugin (for managing CCSDS header types and message structures), and an SSH plugin (for remote system communication).

How does the configuration file in CTF work?

The configuration file in CTF is written in INI format and can specify parameters for CTF core functionality, plugins, and the environment in which CTF operates. It can be customized via command-line arguments for different test configurations.

What types of instructions can be executed in CTF test scripts?

CTF test scripts can include instructions such as registering CFS instances, building CFS flight software, sending commands, checking telemetry values, and archiving CFS artifacts. Each instruction is mapped to a specific plugin for execution.

What is the function of the CTF editor?

The CTF editor provides a graphical user interface (GUI) for editing and executing test scripts. It offers a modern user experience, allowing users to drag and drop test instructions and receive real-time status updates for each instruction.

How does CTF handle test script failures?

If a test script instruction fails, such as a mismatch between expected and actual telemetry values, the CTF editor will indicate the failure. The failure propagates up to the test case and test script levels, and the logs will display the actual vs expected values.

How can CTF be installed and set up?

CTF can be cloned from NASA's GitHub repository, and its environment can be set up using the provided setup script. Dependencies are managed through an Anaconda environment, and a sample CFS workspace is provided for testing and evaluation.

What is the process for running a test script in CTF?

Once the CTF environment is set up, users can run test scripts by specifying the script's path. CTF loads the script, executes the instructions (e.g., building flight software, sending commands), logs the results, and shuts down the system when the test is completed.