Vivado is one of our main tools when working with FPGAs. However when implementing for big FPGAs such as Kintex/Virtex Ultrascale devices, the processing can take forever when running with a high resource usage. The goal of this article is to show a way to tackle this problem and generate more bitstreams faster. By launching Vivado multiple times in the Cloud on instances of our choice (need 128 CPUs and 2TB of RAM, go ahead, just kidding, but this is possible).
With the possibility to launch multiple instances of Vivado on very capable machines, we can upgrade our bitstream generating capacity by an order of magnitude. Or simply work from a tiny laptop and do the implementation in the cloud. There are many advantages and very few drawbacks.
This guide focuses more on the graphical workflow with Vivado, since it seems very popular and not everyone is familiar with the batch mode (tcl / makefile based workflow). But users with experience in such workflows should have no trouble launching instances, working with the command line over SSH and launching their runs remotely.
Amazon Web Services EC2
Amazon Web Services is one of the oldest Cloud services and is quite popular. This is the first cloud service based company/product I worked with so, if you have any advice or remarks on my workflow, feel free to comment.
Preparing a machine
For this, we will need a GNU/Linux machine with Vivado and a way to forward the graphical interface. Vivado can be run via command line in batch mode but, sometimes, you need the graphical interface, e.g., to manually place floorplanning P-blocks. Therefore, we need a way to do remote desktop. For our remote desktop solution we want :
- Low latency
- A Free and open source solution, FOSS
- Not to be required to open extra ports (As port 22 is open we should tunnel over SSH)
- Ease of installation and configuration
Happily for us there exists such a solution ! X2Go, this software allows us to do remote desktop, forward X over SSH, etc.
As for the OS, I chose to use the Ubuntu based on the Amazon “Ubuntu Server 16.04 LTS (HVM), SSD Volume Type – ami-0370f4064dbc392b9” AMI (Amazon Machine Image). Since I have had good experiences running Vivado on Ubuntu 16.04.
The main idea for the next steps will be to prepare a new AMI based on the AMI above but with a desktop environment and remote desktop solution. This will be our base image. Next we want to prepare a disk with the installation of Vivado. The idea behind having another disk with the installation of Vivado is to create a snapshot of this disk (e.g., for each version of Vivado). So we can launch a machine with a base AMI and load the snapshot of whatever version of Vivado we need. E.g.,
- Lubuntu + remote desktop AMI with a Vivado 2017.4 disk attached
- Lubuntu + remote desktop AMI with a Vivado 2018.2 disk attached
- Xubuntu + remote desktop AMI with a Vivado 2017.4 disk attached
- Fedora + … you get the idea
So we only need to keep a single AMI up to date with the OS and the different versions of Vivado can be stored as snapshots and loaded when needed.
This guide supposes you are, at least, somewhat familiar with the amazon cloud computing services and is by no means a complete tutorial, but rather guidelines to create a working environment for Vivado in the cloud quickly and effectively. This guide also supposes you already have an Amazon AWS Account.
Installation of the machine
Now that we know what we want, we can start preparing the machine. For this, I suggest launching a free-tier eligible machine (slow but free) to do the installation and configuration for creating our AMI and later, when we want to do some real synthesis or place and route, launch the AMI on a more powerful machine.
Start by launching a new instance from the EC2 dashboard.
For the AMI, let’s choose the “Ubuntu Server 16.04 LTS (HVM), SSD Volume Type – ami-0370f4064dbc392b9” AMI
As for the instance, let’s use the free tier eligible t2.micro
Review the instance and launch. The 8GB EB disk should be enough. Feel free to adapt this to your needs. We only use this disk for the operating system, Vivado and projects will be loaded on attached disks.
Once the instance is launched, connect it through SSH.
ssh -i "path/to/certificate.pem" email@example.com
Install Desktop Environment
Once connected install the desktop environment, I chose LXDE (Lubuntu) because it’s a lightweight desktop environment. But you can choose any desktop environment you are familiar with. Note that unity has compatibility issues with X2Go. Check the compatibility list for desktop bindings.
sudo apt-get update sudo apt-get install -y lubuntu-desktop
Install X2Go Server
sudo add-apt-repository ppa:x2go/stable sudo apt-get update sudo apt-get install -y x2goserver x2goserver-xsession sudo apt-get install -y x2golxdebindings
When adding the repository, you are greeted with installation instructions for the server as well as the client. They are the same as the code snipped above. Let’s install the server. After running the four commands above the machine is ready to go !
Install the X2Go client on your machine
Follow the instructions at Installing the Qt-based X2Go Client.
Clients are available for MS Windows, Mac OS X, and GNU/Linux. Since I am running on a Ubuntu linux machine, I simply follow the same instructions to install the client on my host machine.
sudo add-apt-repository ppa:x2go/stable sudo apt-get update sudo apt-get install -y x2goclient
This allows me to run the X2Go client. The X2Go client was also tested on Mac OS X and runs fine.
Testing the machine
Now we want to test if we have a remote desktop access and how it behaves (is it fast enough etc.). Let’s connect to the machine. For this, we use the exact same address and credentials we used to connect via SSH.
Create a new session and fill the fields shown below
You can change the resolution in the “Input/Output” tab and remove sound support and client side printing support in the “Media” tab. The resolution can also be changed later once connected. Once configured, you can simply launch the session by clicking on it. It should open a new window with the desktop.
It may be a bit sluggish depending on your connection, distance to the compute server, and remember this is running on a single core 1GB RAM free-tier instance. Running on faster instances will change the experience. Since we are located in Europe changing the servers from USA based to Europe based changed our experience a lot. (You can choose where you launch your instances in the Amazon dashboard).
Since we are logged over SSH and use a certificate the user account has no password set yet. You can set the password with the following command. Note : you may need to rerun this when creating new instances later.
sudo passwd ubuntu
This will be needed for all the graphical applications requesting super user access. But on the command line, you can use sudo directly without needing the password since you are connected over SSH with a certificate.
Once the password is set, let’s update the software on the machine before creating the image. Either use a terminal or use the graphical software update tool.
sudo apt-get update sudo apt-get upgrade
This also let’s us play a bit with the machine and test how it reacts over remote desktop. Once we are done let’s log out. This will close the remote access connection.
Generating our AMI (Amazon Machine Image)
Now we have our base image ready containing the operating system and a way to do remote desktop. To create the AMI, let’s first power off the machine (this is recommended but it should be possible to create an AMI of a running machine).
Once the machine stopped, create the Image.
Set the Image name, description and create it
This will generate an image request. Once the image created, it should appear in the snapshots and the AMIs.
This concludes the preparation and generation of our AMI.
Volumes can be used as disks and attached to instances on the fly. I will use an EBS (Elastic Block Storage) Volume for the installation of Vivado and create a snapshot in order to make it available as a copy for every instance I want to launch.
Create a new Volume
Under the Volumes tab of the amazon EC2 dashboard, click on Create Volume.
Then choose your Volume Type (This is a tradeoff between speed and cost). I use General Purpose SSD for now, but since we will make a snapshot, the snapshot can then be loaded on any other volume type later.
For the size, I use 20GB but feel free to allocate more if you need the full Vivado Installation with HLx and whatnot. Since I only need Vivado to speed up synthesis and implementation for a specific device, 20GB is enough for me.
One important setting is the “Availability Zone”. Be sure to use the same zone as your instances otherwise you will not be able to attach it to your instances. (You can check the zone of your instances under the instance tab).
The volume will now be created and shown in the Volumes tab.
Install Vivado on the new Volume
We will now install Vivado on the Volume. For this, we can continue with the instance created above or launch a new instance with our AMI. I choose to launch a new instance to test the AMI we created and, at the same time, will choose a faster instance in order to speed-up the installation.
Launch a new instance
and chose the AMI created earlier
Then choose your instance type, I like the c5.2xlarge for Vivado, it is quite fast, 8 CPUs and 16GB of RAM. (less than 0.5$ per hour).
Review and launch, you can directly attach the Volume we created, but let’s attach it on a running machine to see how it works.
You can already connect to the Instance with the X2Go client, just update the host address in the X2Go client settings. Let’s do this now since we will need the remote desktop to install Vivado.
Note : you can install Vivado with command line tools too, using the batch install mode (see UG973 “Batch Mode Installation Flow” available since version 2015.1). If you use a command line only Vivado workflow, you don’t need X2Go simply do everything through SSH.
Let’s attach the Volume to the running instance. Under Volumes, choose Attach Volume for the selected volume (the 20GB one for Vivado).
Attach it to the running instance. You can find the instance ID under instances but it will list your instances. If you only have this one running, you can directly select it. For the device, use a /dev entry. The one suggested is fine.
Set the partition table and format the disk
You can use lsblk in a terminal to see that the disk is attached, but not automatically mounted.
I like to use gparted to partition and format the disk but you can use fdisk and mkfs if you prefer. So I install gparted and launch it. Once it is launched, I select the disk and create a partition table.
sudo apt-get update sudo apt-get install -y gparted sudo gparted
msdos type is fine and apply. Then I create a new partition of 20GB, ext4 type is fine.
And I apply the modifications to the disk by using the green tick symbol. You can now mount it, either with the terminal or the file explorer. Note : Since we launched a new machine, we need to set the user password with
sudo passwd ubuntu
otherwise, there will be no password set and graphical prompts for super user rights will reject any password.
Once the disk is mounted (I used the file explorer), change the owner of the disk from root to the user (ubuntu). This can be done with the following command :
sudo chown $USER:`id -g $USER` /mount/point/of/the/disk
Now our disk is ready to use.
Download and Install Vivado
Dowload the version of Vivado you need. I used the web installer (the ~100MB file).
Once you have the file, make it executable with the following command and launch it.
chmod +x Xilinx_Vivado_SDK_Web_2017.4_1216_1_Lin64.bin ./Xilinx_Vivado_SDK_Web_2017.4_1216_1_Lin64.bin
This will launch the GUI installer. Use it to install Vivado to the 20GB disk.
The installation is quite fast since Amazon provides a good network connection and the instance I selected to install (c5.2xlarge) is quite fast. The download speed was about 15MB/sec and this installation took only a minute (Vivado only).
Test the installed version of Vivado
Let’s test if the software runs. Since I chose to not include Vivado on the desktop and launch menus, I have to start it from the terminal. You can also navigate through the installation folder and, in Xilinx/Vivado/2017.4/bin, double click and launch the vivado executable (on Lubuntu at least).
And voilà ! It launches. Let’s log out and stop the machine in order to make a snapshot of this disk, making it available to other instances on launch.
Make a snapshot of the disk with Vivado on it
From the “Instances” tab, stop the running instance. From the “Volumes” tab, detach the volume (don’t forge to stop the instance first).
Once detached use the “Create Snapshot” to make a snapshot of this disk.
Now we have a snapshot of the Volume with Vivado installed. Now we can directly launch instances with our AMI (Lubuntu) and the snapshot of the Vivado installation.
Note : Once you have the snapshots, you can delete the Volumes when no longer needed to save space (money).
Launching an instance
For this example, I will use another AMI I created with the same steps as above but with Xubuntu-desktop instead of Lubuntu-desktop to show the versatility of the solution.
For this, I chose my Xubuntu AMI and used a c5.2xlarge instance. But before launching, let’s attach a disk with the Vivado installation directly from the snapshot. Go to “Step 4 : Add Storage” by pressing next a few times.
Create a new Volume and use the snapshot we created by entering the name (it will find the snapshot as you can see below).
You can set the delete on termination option since we have the snapshot to make new volumes. Review and launch the instance. Once launched, connect to it with X2Go. You should see the disk with Vivado (You may need to mount it). From here, you can simply launch Vivado and begin working with your projects. Note : The first time you launch Vivado from the mounted disk may take a while, but then the data should be cached and run faster for subsequent usages.
Here we can see Vivado 2017.4 running on my Xubuntu AMI. As an example project I used the PYNQ project base overlay.
The synthesis and implementation times were almost as fast as running locally on an i7-4790 @ 4GHz with 32GB of RAM. The out-of-context was faster than on my machine. These results can be explained since my machine has a faster clock 4GHz vs 3GHz on the amazon cores, but the amazon instance gives us access to 8 processors therefore being faster on the out-of-context module runs (done in parallel). I ran everything with 8 jobs in Vivado.
When running on a server in the same region as your location, running the remote desktop through X2Go feels almost like using a virtual machine, it is a bit less responsive sometimes and the image is not as good since it is compressed but the usability is good enough.
The reason I wanted to run Vivado in the cloud was running as many instances as I need in parallel. When working on projects that take more than 5 hours to synthesize and implement (Kintex Ultrascale +90% resource usage), you don’t have the possibility to build many bitfiles in a single day. Now with this solution, I can simply launch 10 instances and launch runs on each of them so I can generate a lot more bitstreams in a day. When you have an instance running, you can simply run other like it with the “Launch more like this” option.
Sharing your project with the instances
The two main ways to share your project with the instances I see are :
- git – simply clone your repository
- scp – secure copy over SSH, transfer your project directly.
I think using a git repository for the sources and a .tcl file to generate the project is the easiest.
Licensing may be a major concern for many user. Let’s discuss the different type of licenses and their implications.
The free license works out of the box.
FlexLM Server Based
Connect to your company network over a VPN and everything should work (unless your compay does whitelisting on MAC or hostname, but you can change those, see below for MAC). I connect through a VPN and have access to the licenses.
However if you have a limited number of seats associated with the license, your company may not like you launching 10 instances all eating up license seats. A possible solution would be to launch a bigger amazon instance with many cores and RAM and launch multiple Vivado runs on the same machine. Feel free to explore and share your solutions.
MAC based .lic file
If you use a MAC based .lic file you may need to generate a new file for the MAC address of the instance, but this will not be a good solution since the MAC address can change during reboot. Also, if you start a new instance, it will not be the same MAC address. However there is a solution, use ENI Elastic Network Interfaces. More info here. This will allow you to set a fixed MAC address.
I have no experience with ENI’s so you need to explore on your own. But it is possible to assign a selectable, fixed MAC address to a machine’s virtual network interface.
Cloud Computing is an amazing technology and service ! Letting me borrow compute power when I need it in a blink of an eye. This allowed me to work on big projects, using just a laptop and remote compilation (synth/impl) as well as to launch many runs when needed.
The cost of a 8 CPU 16GB RAM machine is less than half a $ per hour and would only need to be paid when effectively running. Therefore, possibly being much cheaper than a physical machine and always up to date ! (or at least as up to date as the Amazon machines are) They use nice Xeon Platinums with a whopping 25MB L3 cache.
Your company may not want to buy expensive compilation (synth/implementation) servers but may give you a monthly/yearly budget to use cloud computing giving you access to amazing machines when you need to “launch some runs”.
And yes, you need an internet connection.
Let us hear about your experiences
I hope thie guide may help you explore new workflows and will save you time with your FPGA projects. (Or maybe you just want an excuse to go for coffee while the run is running).
Feel free to comment and share you experiences with the Cloud in the world of Embedded Hardware, Reconfigurable Computing and HPC !