fprime-scales-ref F' project

Watch our video demo on YouTube!

Check out our docs page!

Development Environment

May or may not be required, but this is what we found best to use for development:

• Ubuntu 22.04 host machine
• python3.11
• git lfs (install here for amd64 and here for arm64)
• IMX8X BSP set up on your host machine (instructions here)

How to Clone

Use the commands below in terminal to clone and set up the repository. Make sure to source the fprime-venv before you continue developing! Make sure you have git lfs installed before proceeding.

git clone https://github.com/BroncoSpace-Lab/fprime-scales-ref.git
cd fprime-scales-ref
make setup
source fprime-venv/bin/activate

Necessary Changes

Some lines need to be commented in lib/fprime/cmake/API.cmake in order to use fprime-python. Comment out lines 545 and 562.

After this, you should be good to go!

Hardware Setup

These directions are currently only for the FlatSat, not the custom boards the SCALES team has developed.

Make sure your hardware is configured as follows:

How to Build JetsonDeployment

First time setup only: Set up the Arena SDK for the Ethernet camera. Run this command on the Jetson.

make arena-init

You must generate build JetsonDeployment on the Jetson, we have not set up cross-compilation for aarch64-linux yet.

JetsonDeployment Build Configuration Details

Your settings.ini should look like this:

[fprime]
project_root: .
framework_path:     ./lib/fprime
; uncomment this line for JetsonDeployment
library_locations:  ./lib/fprime-python:./lib/fprime-scales
; uncomment this line for ImxDeployment
; library_locations:  ./lib/fprime-scales:

default_cmake_options:  FPRIME_ENABLE_FRAMEWORK_UTS=OFF
                        FPRIME_ENABLE_AUTOCODER_UTS=OFF

Your project.cmake should look like this:

# This CMake file is intended to register project-wide objects.
# This allows for reuse between deployments, or other projects.

add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/Components")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/ImxDeployment/")
add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/JetsonDeployment/")
add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/lib/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/lib/fprime-scales/scales/scalesSvc")

Your CMakeLists.txt in the root project directory should have line 17 containing register_fprime_target("${CMAKE_SOURCE_DIR}/lib/fprime-python/cmake/target/pybind.cmake") uncommented.

Your Components/CMakeLists.txt should look like this:

# Include project-wide components here

# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/StandardBlankComponent/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/PythonComponent/")
add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/MLComponent/")
add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/RunLucidCamera/")

After all of this, on the Jetson you should be able to generate and build the JetsonDeployment.

To generate:

fprime-util generate aarch64-linux -f

To build and set up the build environment:

make build-jetson

On the Jetson, should be able to generate and build the JetsonDeployment with the commands below:

fprime-util generate aarch64-linux -f
make build-jetson

The make build-jetson command will fprime-util build aarch64-linux and create a linked folder for the camera images. This command also runs a script to create the build environment for fprime-python on the Jetson.

ImxDeployment

To correctly generate and build for the IMX, you need to have the build environment on your machine. Refer to this guide we made on our docs for how to set up the IMX SDK.

ImxDeployment Build Configuration Details

For Successful Build

Your settings.ini should look like this:

[fprime]
project_root: .
framework_path:     ./lib/fprime
; uncomment this line for JetsonDeployment
; library_locations:  ./lib/fprime-python:./lib/fprime-scales
; uncomment this line for ImxDeployment
library_locations:  ./lib/fprime-scales

default_cmake_options:  FPRIME_ENABLE_FRAMEWORK_UTS=OFF
                        FPRIME_ENABLE_AUTOCODER_UTS=OFF

Your project.cmake should look like this:

# This CMake file is intended to register project-wide objects.
# This allows for reuse between deployments, or other projects.

# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/Components")
add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/ImxDeployment/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/JetsonDeployment/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/lib/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/lib/fprime-scales/scales/scalesSvc")

Your CMakeLists.txt in the root project directory should have line 17 containing register_fprime_target("${CMAKE_SOURCE_DIR}/lib/fprime-python/cmake/target/pybind.cmake") commented.

Your Components/CMakeLists.txt should look like this:

# Include project-wide components here

# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/StandardBlankComponent/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/PythonComponent/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/MLComponent/")
# add_fprime_subdirectory("${CMAKE_CURRENT_LIST_DIR}/RunLucidCamera/")

After all of this, you should be able to generate and build the ImxDeployment on your host machine.

To generate:

fprime-util generate imx8x -f

To build:

fprime-util build imx8x

Generate and build the ImxDeployment on your host machine with the commands below:

fprime-util generate imx8x -f && fprime-util build imx8x -j20

To Run the SCALES Demo

IMX Setup

These steps are only required if there are changes made to ImxDeployment. Otherwise, the binary on the IMX should be fine.

1. Follow the instructions above to build ImxDeployment on the host machine. Use the following command to ssh into the IMX.

   ssh root@<ip of imx> -o HostKeyAlgorithms=+ssh-rsa -o PubKeyAcceptedAlgorithms=+ssh-rsa
   

2. Make sure you are able to ping both the host machine and the Jetson from the IMX. Copy the ImxDeployment binary from the host machine to the IMX. (Run this command on the host machine.)

   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/build-artifacts/imx8x/ImxDeployment/bin/ImxDeployment root@<ip of imx>:~/.
   

3. Copy the binary files for the sequences to the IMX.

   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/save-png.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/batch-send-img.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/snap-n-save.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/Zip-n-send.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/test-resnet.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/demo.bin root@<ip of imx>:~/.
   scp -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes=+ssh-rsa ~/fprime-scales-ref/Sequences/run-ml.bin root@<ip of imx>:~/.
   

Jetson Setup

1. On the Jetson, follow the above directions to generate and build JetsonDeployment.

2. Change the IP of the IMX in Jetsondeployment/Top/JetsonDeploymentTopology.cpp to match the IP of the IMX.

   // line 32
   const char* REMOTE_HUIP_ADDRESS = "10.3.2.2"; // ip of JPL IMX
   // const char* REMOTE_HUIP_ADDRESS = "192.168.0.66"; // ip of CPP IMX
   const U32 REMOTE_HUPORT = 50500;
   

3. Rebuild JetsonDeployment.

   make build-jetson
   

4. For first time setup only: Make a folder with a symbolic link to where the camera images are saved. This is done to assure the paths for commands in the fprime-gds are not too long.

   sudo ln -s ~/fprime-scales-ref/build-python-fprime-aarch64-linux/Images/ ./Images
   sudo ln -s ~/fprime-scales-ref/build-pyhon-fprime-aarch64-linux/Images/ ./Images
   

   The Images folder will be created in your root directory.

Host Setup

1. Open another terminal on the host machine and enter the directory for the repo and source your environment.

   cd fprime-scales-ref
   source fprime-venv/bin/activate
   

2. Copy the ImxDeployment dictionary to the GDS-Dictionary folder on the host machine. Run this command on the host machine.

   cp ~/fprime-scales-ref/build-artifacts/imx8x/ImxDeployment/dict/ImxDeploymentTopologyAppDictionary.xml ~/fprime-scales-ref/GDS-Dictionary/.
   

3. Copy the JetsonDeployment dictionary from the Jetson to the host machine. Run this command on the host machine.

   scp <jetson name>@<jetson IP>:~/fprime-scales-ref/build-artifacts/aarch64-linux/JetsonDeployment/dict/JetsonDeploymentTopologyAppDictionary.xml ~/fprime-scales-ref/GDS-Dictionary/.
   

4. Combine the GDS dictionaries with the merger.py script. Run this command on the host machine.

   cd GDS-Dictionary
   python merger.py JetsonDeploymentTopologyAppDictionary.xml ImxDeploymentTopologyAppDictionary.xml GDSDictionary.xml
   

You are now ready to run the demo!

Running the Demo

1. After you finished setting up the demo in the previous section, on the host machine, navigate to the GDS-Dictionary folder and run the fprime-gds.

   fprime-gds -n --dictionary GDSDictionary.xml --ip-client --ip-address <ip of imx>
   

2. On the IMX, run the ImxDeployment binary. You should see a green dot on the fprime-gds and "Accepted client" in the IMX terminal.

   ./ImxDeployment -a 0.0.0.0 -p 50000
   

3. On the Jetson, navigate to the build-python-fprime-aarch64-linux directory to run the fprime-gds using python.

   cd build-python-fprime-aarch64-linux
   python -c "import python_extension; python_extension.main()"
   

   This command opens the python environment and connects to the IMX's fprime-gds using the hub pattern. If you want to exit the python environment, the command is exit().

4. On the host machine, use the fprime-gds to run the jetson_cmdDisp.CMD_NO_OP to test the connection with the Jetson. Do the same for the IMX with the imx_cmdDisp.CMD_NO_OP. You should be able to see that both events completed in the "Events" tab of the gds.

5. Once the camera is connected (flashing green light on camera), run the jetson_lucidCamera.SETUP_CAMERA command to verify the connection via fprime.

6. To take a picture with the camera, run the imx_cmdSeq.CD_RUN command in the fprime-gds with argument demo.bin. This will take a pictire with the camera, downlink it to the IMX, and then downlink it again to the Host Machine. You can download the image from the Downlink tab in the GDS. This sequence will also run a resnet ML model to identify what is in the image. The output will be displayed in the Events tab of the GDS. Images are deleted from the Jetson after the demo.bin sequence concludes. Repeat this step if you wish to take more images.

   

   This sequence will trigger the Images from the Jetson to be downlinked to the IMX, and then again downlinked from the IMX to the Host Machine. Check the Downlink tab in the GDS to see the images.

   

   Click the Download button in the Downlink tab of the fprime-gds to download the zipped Image folder to the host machine. You can then unzip the folder and view the images from the Jetson!

Alternative Commands

1. If you would like to send a batch of images from the Jetson to the Host Machine, run a sequence on the IMX using the imx_cmdSeq.CS_RUN command on the fprime-gds with fileName argument send.bin. The Command String is as follows:

   imx_cmdSeq.CS_RUN, "send.bin", BLOCK
   

   

   This sequence will trigger the Images from the Jetson to be zipped into a smaller file to be downlinked to the IMX, and then again downlinked from the IMX to the Host Machine.

   

   Click the Download button in the Downlink tab of the fprime-gds to download the zipped Image folder to the host machine. You can then unzip the folder and view the images from the Jetson!

2. To run ML on the images, run the mlManager.SET_ML_PATH command with argument resent_inference. Then, set the inference path to where the images are stored with the mlManager.SET_INFERENCE_PATH command with argement ../Images. Finally, run the ML model with command mlManager.MULTI_INFERENCE. You should see the results of the ML model both in the Jetson's terminal and in the Jetson's fprime-gds Events log.

That's how to run the SCALES demo!

Watch our video demo on YouTube! Some minor changes have been implemented since the creation of this video, but the core process remains the same.

To Run Scales-ML

Scales-ML

1. Follow the setup described in previous sections for the IMX, Jetson, and Host Machine.

2. In the fprime-gds, run the imx_cmdSeq.CS_RUN command with argument test-resnet.bin. This sequence will:

   • Set the ML path to a resnet model
   • Set the inference path to a folder called test-imagery with example images
   • Execute the MULTI_INFERENCE command to inference on all images in that folder.

Troubleshooting

When trying to run the SCALES demo, you may encounter a few issues.

Hanging/Crashing During Downlink

This is because there is an exisiting file on the IMX names image.png from a previous, incomplete run of the demo. Just delete the image.png file from the IMX with rm image.png and try running the demo again.

This also may be due to an issue with the Images/ folder on the Jetson. Return to step 4 in Jetson Setup to make sure you have the Images/ folder set up correctly.

fprime-gds Crashes on Jetson When Trying to Connect

Instead of doing the shortened command to connect to the gds from the Jetson, try entering the python environment first and then running import python_extension and python_extension.main() one at a time.

Inferencing Error

When trying to run the MULTI_INFERENCE command on the Jetson, you may experience an error similar to:

'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/resnet-18/resolve/main/preprocessor_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff3bf23450>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 6d6a5cec-e762-484c-a97a-1e1d9748bcba)')' thrown while requesting HEAD https://huggingface.co/microsoft/resnet-18/resolve/main/preprocessor_config.json

Make sure the Jetson is connected to WiFi and try again. This is a new issue we have encountered that we are still trying to find the root cause of, but a WiFi connection fixes the issue.

---

This project was auto-generated by the F' utility tool.

F´ (F Prime) is a component-driven framework that enables rapid development and deployment of spaceflight and other embedded software applications. Please Visit the F´ Website: https://fprime.jpl.nasa.gov.