Unity Drone Simulator is an application which allows you to simulate the 'Eagle Concept', streaming out the Camera feed of the 'Eagle' Drone and receiving inputs to move the 'Eagle' Drone. This project is meant to test the capabilities of a Computer Vision Model and PID Controller Script.
Unity's High Definition Render Pipeline (HDRP) is used in this project to enhance the graphical fidelity, enabling the Simulator to be as similar to reality, thus testing the Computer Vision in near-realistic environments.
An 'Eagle' Drone is flying above the 'Target' Drone, with its camera facing vertically down. The height between the 2 drones is the Separation Distance of the drones. The Eagle Drone's height is fixed at 55m, while adjusting the Separation Distance will move the Target Drone's vertical position. For example, with a Separation Distance of 20m, the Target Drone will be placed at 35m.
In this application, the user is looking from the Eagle Drone's Camera's Point of View, and is able to move the Target Drone with Keyboard Controls (WASD).
Follow the below steps to use the Application together with a Computer Vision model and Controller Script.
- Download the latest release.
Alternatively, clone this repository and open the Unity Project (Unity 2020.3.30f1). You can run it in Unity Editor or build the Application yourself.
- Launch the Application, as well as a RTSP Server, e.g. rtsp-simple-server.
⚠️ Warning!⚠️ Ensure that a server is running before starting the RTSP stream, else the Application will crash.
- Configure Settings. Important Settings to note are:
- RTSP output path
- ZeroMQ listener path
Set the ZeroMQ listener path to be the output path of the Controller Script
- Click on Start to load into the Scene. RTSP and ZeroMQ will automatically start up.
To confirm, the rtsp-simple-server will indicate that a session is publishing to the path provided in the settings.
Start up the Computer Vision and Controller Script.
- If running on another device, ensure the devices are able to connect to each other, e.g. change firewall settings.
- For the Computer Vision, consume the RTSP stream via the RTSP output path as set and shown in the Application's Settings.
To confirm, the rtsp-simple-server will indicate that a session is reading from the path provided in the settings.
Move the Target Drone with WASD, and observe that the Computer Vision is able to detect the drone. The Controller Script should then receive the detections, and subsequently send commands to move the Eagle Drone. The Eagle Drone, also the Application's POV, should be moving and following the Target Drone.
Press Escape or click on the gear icon in the top left corner to open up the Menu while running. The Input Replay Menu, Waypoint Menu and Circle Menu will allow you to automatically move the Target Drone. For more details, refer to the Runtime Section.
Entering the Application will bring up the Main Menu. A Quick Glance sidebar is also displayed, allowing quick and easy access to critical settings. The Main Menu contains the following:
- Start
Loads Scene and automatically starts RTSP and ZeroMQ.
- Settings
Opens up Settings Page (see below).
- Fullscreen
Toggles Fullscreen (based on Windowed Mode's Aspect Ratio).
- Quit
Closes the application (stops running if in Editor).
The Settings Page will enable you to modify the below:
The application is able to stream out the Eagle Drone's camera feed via RTSP using FFmpegOut. For more information on FFmpegOut, do refer to the FFmpegOut README or the GitHub repository.
⚠️ Warning!⚠️ Ensure that a server is running before starting the RTSP stream. Consider using rtsp-simple-server to run the server, located at /rtsp simple server. Refer to the rtsp-simple-server README for information on setting up.
Settings Guide
Adjust the Port Number and URL for the RTSP output path. By default, Port Number is 8554 and URL is /drone. The application will display a list of IP Address your device is connected to. This RTSP stream can then be accessed from another device by connecting to a listed IP Address.
Ensure that the two devices are able to connect to each other, e.g. change firewall settings.
An additional Delay can be introduced into the RTSP stream by adjusting the slider in the Settings page. Select how much to delay the stream in milliseconds.
Note that this will not account for the existing latency, meaning that with a base latency of
500ms, adding1000msdelay will result in a combined delay of1500ms.
Click to toggle the Timestamp, which will show the time elapsed since the application was opened.
Note that this Timestamp, located in the bottom left of the screen, will be visible in the RTSP stream. This is useful for measuring latency between the application and the receiving client.
If necessary, adjust the other parameters for the stream:
- Width (default 1920),
- Height (default 1080),
- and Frame Rate (default 30).
The application is able to receive inputs via ZeroMQ, to move the Eagle Drone. The Controller Script should send in inputs as Vx,Vy.
ℹ️
VxandVyshould be within the range from -1 to 1, which will then be modified by the Drone's set Max Speed.
Settings Guide
Enter the Path for the application to connect and listen to.
A Delay can also be introduced, meaning that inputs will only be executed a set fixed time after receiving from the Controller Script.
Similar to the RTSP delay, this will not account for the existing latency.
Modify the Settings for both the Eagle Drone and the Target Drone.
Settings Guide
As mentioned in Eagle Concept, Separation Distance will adjust the height of the Target Drone from the Eagle Drone, from a range of 1m to 50m.
The Max Speed of the Eagle Drone and the Target Drone can be adjusted. This will determine how fast the drones will travel. For example, with Vx = 0.5 and Max Speed of 10m/s, the drone will move at 5m/s.
The 3D Model for the Target Drone can also be changed. This repository comes with the models of DJI Mavic and DJI Phantom. New models can also be imported by placing them into the Assets/Resources/Drone Models folder, as well as the StreamingAssets/Drone Models folder.
During runtime, models can be imported through the StreamingAssets/Drone Models folder. Supported formats are:
The Refresh Button next to the Model Dropdown will reload all the files and import the models. For more info on the Runtime Importers, refer to the Assets README on Runtime Importers.
The Target Drone's Material Colour can also be edited using the Flexible Color Picker. For more info on the Flexible Color Picker, refer to the Assets README on the Flexible Color Picker or the FlexibleColorPickerDoc.pdf.
Upon pressing Start, the RTSP Stream and ZeroMQ Client will both be automatically started. The Target Drone can then be controlled with Keyboard Controls (WASD).
⚠️ Warning!⚠️ Attempting to Start without a RTSP server will cause the Application to crash.
Press Escape on your keyboard or click on the gear icon in the top left corner to open the Menu. The following options are available in the Menu:
- Input Replay Menu
- Waypoint Menu
- Circle Menu
- Return to Main Menu
Stops the RTSP Stream and ZeroMQ and will return to the Main Menu starting screen, where Settings can be changed.
- Reset Drone Positions
Moves the Eagle and Target Drones back to the starting position.
- Quit
Closes the application (stops running if in Editor).
The Input Replay Menu, Waypoint Menu, and Circle Menu all allow the Target Drone to be moved automatically, not requiring manual input from the Keyboard.
An Input Replay is a recording of the Target Drone's Velocity normalised by Max Speed, automatically moving the Target Drone without requiring manual input. The Application will check if the Target Drone is within the boundaries of the Camera Screen. If the Target Drone remains in view of the Camera for the entire duration of the Input Replay, it is deemed as a Pass, else it will be considered a Fail.
Both Recording and Replaying are possible. Input Replays are stored in the StreamingAssets/inputreplay folder. Input Replays can be manually imported into the folder. However, it is recommended to create an Input Replay via the Recording function.
To save a Recording, input a custom file name or leave blank to auto-generate a new file name. Inputting an existing file name will overwrite the existing Input Replay. Press the 'Save' button to start recording, and press the 'Save' button again to stop recording.
All Input Replays found are listed in the Input Replay Menu. The 'Play' button next to each Input Replay will Load the respective Input Replay. Loading another Replay while one is running will end the existing Replay and start the new Replay.
The 'Load All' button will successively load each Input Replay, displaying the output of each Replay whether it is a Pass or Fail.
A Waypoint is a more human-readable format to give directions to the Target Drone. The Waypoint will provide the x and y coordinates, as well as the speed for the Target Drone to move.
Both Recording and Replaying are possible. Waypoints are stored in the StreamingAssets/waypoint folder. Waypoints can be created via the Recording function. However, it is recommended to manually create a Waypoint, to ensure precision.
To save a Recording, input a custom file name or leave blank to auto-generate a new file name. Inputting an existing file name will overwrite the existing Waypoint. Press the 'Start' button to start recording, and press the 'Stop' button to stop recording.
All Waypoints found are listed in the Waypoint Menu. The 'Play' button next to each Waypoint will Load the respective Waypoint. Loading another Waypoint while one is running will end the existing Waypoint and start the new Waypoint.
Enabling the Circle function will allow the Target Drone to orbit around the initial position. The desired Radius and Speed, as well as Direction (Clockwise or Counter-Clockwise) can be changed in the Menu.
ℹ Note that the Circling function is disabled while an Input Replay or a Waypoint is being Loaded.
Below is a comparison among the 3 methods of automated movement:
| Input Replay | Waypoint | Circle |
|---|---|---|
| Saves Velocity | Saves Location and Speed | Cannot Save |
| Precise Recording | Inaccurate Recording | Cannot Save |
| Less readable | More Readable | N/A |
| Changes Velocity | Changes Velocity | Changes Position |
| Stops if exits screen | Loads entire file | Infinite duration |
| Affected by Max Speed | Independent, set in file | Set Speed in Menu |
Additional Asset Packages were used to assist in the creation of this project. More details can be found on the Assets README.