TonicsCloud Update 2: Deploying, Resizing and Signaling Instances

Here is an update on the TonicsCloud project, I will be sharing some challenges I encountered, my workarounds, and ehm, just general progress with the TonicsCloud project in general, first, let's talk about the deployment strategy.

Deployment Strategy

So, here is my strategy for the deployment process:

1. A script that is run immediately after the instance is deployed (Stackscript in Linode, user data in aws), this way, the instance can be TonicsCloud ready, my script doesn't contain much, I install Incus, remove openssh-server, created a one-shot systemd service that would run on boot and would only ever run once, subsequent boot won't run the service, once the service is started, it handles Incus configuration. [Encountered an issue!]
2. Support only one distro, I went with Debian 11.
3. Should be done through the VPS Provider API, I built TonicsCloud to support multiple providers, for now, I am using the Linode Provider, which I find really cool.
4. Once the deployment or the provisioning is done, the VPS Provider should send an event message (i.e webhook), in short, it should send the event in stages, for example, when it is provisioning, when it is booting, when it is in the ready state, etc, then I could update the records in my table.  [Not so simple!] 
5. Backup is enabled daily by default for all plans, the VPS Provider should handle that automatically, this way, TonicsCloud can focus on other things

Yh, the above was the deployment strategy I planned on implementing, but as you can see, I encountered some issues, the first is using a script that is run immediately after the instance is deployed, the issue I had here was that it was damn slow, Incus isn't in the default Debian repo, so, I was using an external repo, and sometimes, it would take a bit of time for the repo to respond, in some cases, it took around 2-5 minutes, not good at all.

My solution was to remove step 1 and replace that with a ghost image, also known as a golden image, the image comprises all the dependencies for the instance to be TonicsCloud ready, which works well for me, took around 5-15secs to deploy an instance with this method, much better.

The next issue was the fact that Linode does not support webhook (most major VPS provider also doesn't support it, e.g DO, VULTR, etc, don't know why since it would be easier and less taxing on their system), so, my workaround is polling, fortunately, the Tonics CMS already supports a Job and a Schedule Manager, so, all I had to do, is to create a Scheduler for that, it is set to run every 10secs.

It can even be set to run every 5secs, Scheduler in Tonics is not very taxing, every schedule is run in its own process, as long as the VPS provider can return the result faster, we are good.

Below is how I implemented the polling:

The polling implementation I've used works by periodically making requests to the instances endpoint. Since handling everything on a single page would be impractical, the data is paginated. For example, let's say the first page contains 15 results.

In this implementation, we update the status of those records in the table and then pass the nextPage parameter (if available) to the next scheduler job. This scheduler job will run at a later time to continue polling and processing the remaining pages.

This process repeats until we reach the end of the page. At that point, we start again from the first page, ensuring that we cover all the data. This approach allows us to systematically retrieve and update the status of instances in an organized manner.

The possibility of a race condition arises if two scheduler jobs happen to run at the same time. However, this doesn't pose a significant problem because both jobs will be updating the same data.

The updates from each job will be applied, ensuring that the data remains accurate and consistent.

Let's do some math, say, I run the schedule every 5 secs, then that is 12 times per minute, multiply by the number of page_size, say, 200 per page, that is 2400 instances processed per minute minus any latency or other unforeseen issues.

If I get to the point where I need to handle more than that, then I can use a parallel scheduler which is also supported in Tonics CMS, this way, I can run multiple schedules of the same type in parallel, though, I would have to change the way the instance status results are updated in the table, but, till then, for now, this is fine.

Here is an image of the Scheduler logs:

Deployment, Signaling, and Resizing Demo

Talk is cheap, here is a demo deploying instances, signaling, and resizing instances in Tonics:

That is it, for now, let me go work on other features...