Basically this project completely bypasses the manual browser interface of Google Colab/Kaggle. It completely reverse engineers everything and it helps us distribute heavy compute and machine learning tasks across multiple free Google accounts to utilize GPU allocation. If you have 10 different Google accounts, you can distribute tasks across all 10 accounts programmatically without using their web interface and execute your tasks
It took me about 40 hours to reverse engineer everything, find out all the endpoints, and try out a lot of stuff. None of that got tracked on Hackatime unfortunately because everything was done on PowerShell, Fiddler, and HTTP Toolkit, etc. That's sad
Initially I just made the UI to quickly test stuff instead of writing command-line instructions. Turns out that the UI was already 90% done so I just thought of making it its own execute table so it's easy to test out stuff without running it programmatically
only the colab_example.py was generated with ai, the entire file. To be honest, I've barely tested it, and I don't even know whether it works, but I'm pretty sure it should give you a good idea of how the SDK works. There was literally zero AI used to write any amount of code, but there was heavy use of AI to teach me how to reverse engineer the endpoints and give me ideas of what I could do and how I could do it. Basically, AI worked as a guide to teach me the skill of reverse engineering.
See the thing about Google Colab is that it's completely a closed-source platform and none of its internal APIs, web socket multiplexing, authentication protocols, and container life cycle are exposed anywhere. There's zero documentation and there's no work done by the community to reverse engineer it to this degree. Obviously there would be a lot of projects but when I tried to search for it I could not find anything because the reason I did this was that my laptop basically has an integrated graphics card. There is literally no way for me to run powerful models on my laptop and Google Colab lets me do this but I wanted to distribute the tasks across accounts so I could leave it overnight
There was a lot of guesswork and a lot of things to figure out, such as:
- the Jupiter Kernel protocol over websockets
- intercepting and emulating how Colab establishes websocket channels to send code execute requests and call live locks and check tab completions and handle kernel interruptions etc.
- there was also an internal control plane API so I had to reconstruct all the undocumented REST endpoints, such as Colab.PA.GoogleAPI.com, used by Colab to request machine assignments and retrieve hardware information and query compute unit consumption etc
- And then I had to figure out how I interact with their WebSocket-based terminal. I had to reverse engineer all their WebSocket shell endpoints so I could get a direct terminal session into their Linux container. There is a lot more that's difficult to discuss and difficult to explain without going into too many technical details
Depending on your workflow, you can basically use this project in three different ways:
you literally don't need to use a command line. It works like a native GUI and there you can add your accounts with one single click. You can test everything, you can do quick account switching, and you don't really communicate with the CLI.
that's again executable and that's a compiled CLI binary where you can interact with Collab purely on the CLI.
this is the most important and the feature-rich one. It's for programmatic workflows and you can simply import the Collab SDK module.
refer to the example file for a tutorial on everything such as (file transfers, drive mounts, terminal manipulation, parallel threads, etc.), colab_example.py
If you are using the compiled CLI executable (or running via python colab_sdk.py), here are the core commands to automate your workflows:
You can register multiple accounts under different profiles to distribute your workloads:
# Add a new account profile (opens a browser window for login)
colab-sdk auth add my_account
# List all authenticated profiles
colab-sdk auth list
# Verify if account tokens are valid
colab-sdk auth statusKeep track of your active machines and resources:
# List active runtimes across all authenticated accounts
colab-sdk runtimes list --all
# Inspect health, specs, and compute metrics of a specific runtime
colab-sdk runtimes info <ENDPOINT_ID>
# Safely kill / unassign a runtime when you are done
colab-sdk runtimes kill <ENDPOINT_ID>Execute scripts or notebooks directly:
# Run a python snippet on a T4 GPU runtime
colab-sdk exec "import torch; print(torch.cuda.is_available())" --account my_account --gpu T4
# Execute a local notebook on an L4 GPU runtime and save the result
colab-sdk run job.ipynb --account my_account --gpu L4 --output result.ipynbThe Collab UI basically has almost every single feature from Collabs website, including extra quality of life features such as easy account switching and task distribution, etc.
You can spin up web servers and it automatically installs Cloudflare, creates a tunnel, and gives you the exact IP for the tunnel. There is a file browser where you can quickly upload, download, and delete files. You can connect it via Google Drive and there is a terminal. There are a lot of features. You can view all of those in the screenshots below
