How we designed our Cardano Stake Pool and Cardano Dashboard
Context and Mission
ADA Ocean Australia was built to be the first fully Australian Cardano Stake Pool. At the time we registered our stake pool, all other Australian pools had all their servers, or a majority of their servers in countries other than Australia. Their reasoning was server costs, as Australian based servers are typically more expensive compared to other countries such as Germany. Other reasons might also include the misunderstanding of needing servers closer to other block producers to reduce latency, however Cardano doesn’t work like that.
This mentality caused a lot of clustering in the United States and Germany, of which they both currently contain over half of all Cardano’s Relay servers. If over half of the relays are in those countries, it is a good assumption that the Block Producers are also in those countries. This is a potential issue as Cardano is designed to be a worldwide framework, and clustering to certain countries could have side effects such as political interferences or impacts from network outages.
Since Cardano is based on Proof-of-Stake, Stake Pools can only produce blocks for the slots they are assigned. If a Pool misses their block, the Cardano protocol must wait for the next assigned slot, which results in delayed processing of transactions. This concept is represented by Chain Density which is a way to measure the network health (5% Chain Density means blocks are being made every 20 seconds and is therefore healthy).
While ADA Ocean Australia isn’t currently a “mission based” pool, our main selling point was that we were 100% based and operated in Australia. This gives Cardano more decentralisation both physically and politically. Since AOAUS is operated in Australia, we wanted to support the I.T industry in Australia.
Our Setup
ADA Ocean Australia operates on a hybrid of Cloud and Bare-metal servers. This gives us a mixture of reliable Cloud infrastructure, and performance and freedom of Bare-metal.
The Pool [AOAUS]
Our pool has been designed for maximum up-time with a number of safeguards in place. These cover Load, Geo-Location, Fail Overs, and Multi-cloud.
Geo-Location: The ADA Ocean Australia pool has Relay servers spread across Australia, with servers in Sydney, Melbourne, Perth and Brisbane. These not only cover us in the event of a major outage in a region, but also gives us direct access to the entry points of Australia’s sub-marine cable network. This setup has already proven itself as we recently experienced a blackout in QLD, Australia and our pool remained 100% operational.
Load: Our servers have enough overhead to support at least 25 concurrent outbound peers and have been able to sustain up to 40 inbound peers per server and can theoretically go higher. As a failsafe, we have an unlisted bare-metal server to protect us against a direct attack. This location has direct DDOS protection of 3.5Gbps.
Fail overs: Our block producer has a number of fail safes to make sure it stays operational. Firstly, it is managed by a world leader in Cloud hosting with a 99.99% uptime guarantee. From an operating system level, our server has a “live patch” ability which enables us to apply software updates without needing to restart the server. In the event the underlining server experiences faults, the Virtual Machine can be “live migrated” to another server with no downtime. We have personally experienced the Virtual Machine being live migrated and it works very smoothly. In the event the server is completely dead or can’t be contacted (such as a datacentre or whole region outage) we have a block producer in another geolocation ready to go.
Multi-Cloud: One of the issues we wanted to address from the beginning was the possibility of hosting providers taking issue with blockchain applications. While Cardano is not a resource intensive blockchain like Bitcoin, it’s still grouped into blockchain and cryptocurrencies, and some hosting providers can be very ignorant of the differences. Being aware of this, we’ve researched many possible hosting providers in Australia and spread our servers across them. This is essential due to the fact it can take a while to migrate / setup a new server with a new provider. It is a lot safer to have several servers already setup and if one provider restricts our account, we still have several other hosting providers we can fall back on.
Bare-metal: Our bare-metal server is a self managed Ryzen 9 with 128GB of RAM and a Samsung 980 Pro SSD. While we have other, lower powered bare-metal servers, our main rig handles most of our projects. In particular, it runs our Brisbane Relay Mainnet node, the Backup Block Producing Node, our Cardano Testnet pool, and testnet.adatools.io.
While this server is our most powerful server and easily outshines our cloud servers, especially when it comes to single core workloads and it’s high SSD IOPS, it’s a server that we can’t guarentee to remain running as reliably as our Cloud based servers. This was recently made more obvious to us when we had a recent power outage in Queensland, Australia. While the UPS held up, our internet provider requires power to operate and therefore the server had no internet connection.
ADATools.io
ADATools has been difficult to manage due to the high resource demands of not only running cardano-graphql (an API connector for cardano-node and cardano-db-sync), but the amount of traffic it receives. As of writing this article, it receives an average 700 unique visitors and 237,000 requests per day. A majority of these requests come from Germany and the United States, and a small spread from the rest of the world.
User experience has always been our highest priority and for that reason we’ve made some major decisions. Firstly, we wanted to make sure that our servers responded as fast as possible. To be able to achieve this we firstly needed to make sure that users were connecting to a server that is as close as possible to them to reduce latency. The most obvious way to do this is to use Cloudflare’s Load Balancer along with Argo Tunnel. This allowed us to direct traffic to the closest server based on the users physical location.
This biggest “gotcha” that users find themselves in is AWS’s pricing model, which is not only complex from the start, but can also have a big impact on performance. That is why we have done extensive testing on various hosting providers and server configurations instead of defaulting to AWS.
To begin with, we’ve found that when it comes to number crunching, Cardano typically prefers higher frequency CPUs over multi-cores, especially during an epoch changeover and building. There are a number of Cardano dashboards that lock-up during the changeover and while ADATools does as well, its duration is a lot shorter than most.
For those that aren’t aware of how CPU “turbo-boosting” works, a single core on a CPU can overclock the most when there are no other cores on the CPU being used. This is a trap developers can fall into, even with “dedicated” vCPUs. Just because a core is dedicated to you, it does not mean that core will turbo-boost because all the other cores might be fully used by other tenants on the server (known as “noise neighbours”). Hosting providers that offer “dedicated” vCPUs therefore aren’t as powerful as they could be compared to having control of the whole CPU.
The next trap developers fall into is SSD IOPS, especially when using AWS. When it comes to databases, IOPS can be more crucial than the CPU. During our testing, we found that a typical AWS EC2 would give us a maximum of 3000 IOPS. If you want more, you pay dearly for it. For reference, Digital Ocean, Vultr and Linode would give us an average of 75,000 IOPS for their basic hosting. Digital Ocean and Vultr both have “Premium” / “High Frequency” virtual machines that give you more than 160,000 IOPS.
This is still small when compared to a Samsung 980 Pro SSD which is rated at 1,000,000 IOPS, which could be considered overkill but for accessing databases, the higher the IOPS the faster it can read the data. A real-life reference is a query on a AWS EC2 took about 45 seconds, and the same query on Digital Ocean took 4 seconds.
The combination of all our research led us to the conclusion that bare-metal was the way to go as it ticked all the boxes for our requirements. While there are many bare-metal hosting providers to choose from, we liked in particular Vultr’s bare-metal servers. They have recently changed their bare-metal options which are much more pricey compared to what we’re currently using.
Moving Forward:
We’re always looking for ways to improve our pool and ADATools.io, and our research is still on going. We have many projects and ideas, and it’s all about priorities. ADATools.io has many features we would love to implement, however it’s been put on the back-burner for a while as we’re currently taking part in the Cardano Plutus Pioneer Program and we have a few awesome ideas for smart contracts that we are focusing on.
If you have not seen it already, check out our website at www.adaocean.com.au and our Cardano Dashboard / Explorer at https://ADATools.io.
If you enjoyed this article and would like more, leave a “clap” and a comment below.
Follow us on Twitter: https://twitter.com/ADAOceanAus
