How Much Does it Cost? 💸 😬

[nelsonic]

As noted by @LuchoTurtle in #97 (comment) 💬
This "hobby" app is costing us considerably more money than we originally expected. 📈
The most recent invoice on Fly.io was $48.61 for Mar 1 - Apr 1, 2024 https://fly.io/dashboard/dwyl-img-class/billing

The current month (April 2024) Amount Due is already $14.34 and we're only on the 4^th!!

If we extrapolate the total will be 7.5 x ($14.34 - $5) + $5 = $75 💸 🔥

This is already more than we spend on our Internet & Phone bill ... 🤯
If the cost could be kept to $10/month it would be fine. 👌

Todo

• @LuchoTurtle please have a think about what can be done to reduce or cap the cost. ✂️

I'm keen to keep this app available for people to test without having to run it on localhost. 💻
But if the casual visitor is costing us this kind of cash, imagine if this got to the top of HN! 😬

[ndrean]

Not sure to understand these costs: Fly.io pricing).

1. 19760s = 5h29min20s => is this the uptime?

2. Every time you start the app, you need to upload 2G of data of the models (volume is pruned with the VM), see Fly.io docs). This means you recreate a volume of 2G and load 2G into memory. Is this the meaning of the 2 highlighted lines?

[LuchoTurtle]

@ndrean the models are not pruned with the VM every time the app is started (it was a misconception that was fixed in #82). Currently, the models are being saved in the volume and they are not re-downloaded every time it restarts. You can see it in the logs, actually:

2024-04-05T05:04:33.414 app[080e325c904168] mad [info] 05:04:33.414 [info] ℹ️ No download needed: Salesforce/blip-image-captioning-base

2024-04-05T05:04:33.415 app[080e325c904168] mad [info] 05:04:33.414 [info] ℹ️ No download needed: openai/whisper-small

2024-04-05T05:04:33.415 app[080e325c904168] mad [info] 05:04:33.415 [info] ℹ️ No download needed: sentence-transformers/paraphrase-MiniLM-L6-v2

The problem is that models take up a fair amount of space (Salesforce/blip-image-captioning-base, especially). So we're basically paying for every additional space the models take over the free tier.

However, the cost is definitely bigger with the RAM usage. There's no way around this, as it's used to run inference on the images being uploaded. Although GPUs are much better at this, the costs are severely higher.

@nelsonic unfortunately there's no way around this. There's been people using the application, which is great. But, as with any LLM/ML-based application, it's hard to make it in any free-tier cloud solution without putting the money in.

The app has already been optimized to reduce costs (inbound/outbound data reduced with persistent storages, reducing file limit of images by optimizing it before feeding into the model, reducing Hz of audio file on the client-side before feeding it into the model).

Unfortunately, we have to pull the plug. With increased activity, this "hobbyist" project sucks money (even more so when they're stopped, as shown in your image).

I'd love to have this online for any person to check it out. But it's not feasible :(

I'm going to shut down the machine now.
I'll keep the database, though. It has the images and index files saved. So we can still have the uploaded data and have the app running normally by just spawning a new machine whenever we want to. The machine will look for the index file in the database (since it doesn't have any on its own filesystem), download the models and the index file and it will resume where it stopped gracefully :)