7 9 44

Mitko Vasilev

mitkox

AI & ML interests

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

Recent Activity

posted an update 1 day ago

I run Claude Code with Qwen3 Coder Flash locally on my MacBook Air. It works offline, zero cloud, zero internet, zero EU AI Act anxiety. No limit with all tokens on the house. It’s not great, not terrible- adequate performance for an on device AI agent chewing through code on a 1.24 kg laptop. I wrote an interpreter to broker peace between Claude Code and my local AI runtime. Make sure you own your AI. AI in the cloud is not aligned with you; it’s aligned with the company that owns it.

posted an update 3 days ago

XBai o4 claims to beat Claude Opus 4 and o3-mini, and they provide verifiable proof. My skepticism circuits overloaded, but my local AI FOMO module screamed louder. I've thrown this 33B monoblock LLM onto a single GPU and used Roo Code for some… let’s call it “vibe testing”. It’s terrifyingly competent. As an architect, it’s the best open-weight model I’ve touched this side of 2025.

posted an update 4 days ago

We’ve reached a point where on device AI coding that is free, offline, and capable isn’t just a theoretical possibility; it’s sitting on my lap, barely warming my thighs. My local MacBook Air setup includes a Qwen3 Coder Flash with a 1M context, Cline in a VSCode IDE. No internet, no cloud, no ID verification- this is the forbidden tech. Current stats: All agentic tools work great local, sandboxed, and MCP OK model output precision 17 tokens/sec. Not great, not terrible 65K tokens context, the model can do 1M, but let’s be real, my MacBook Air would probably achieve fusion before hitting that smoothly Standard backend and cache off for the test All inference and function calling happen locally, offline, untethered. The cloud didn’t even get a memo.

View all activity

Organizations

posted an update 1 day ago

Post

1834

I run Claude Code with Qwen3 Coder Flash locally on my MacBook Air. It works offline, zero cloud, zero internet, zero EU AI Act anxiety. No limit with all tokens on the house.

It’s not great, not terrible- adequate performance for an on device AI agent chewing through code on a 1.24 kg laptop. I wrote an interpreter to broker peace between Claude Code and my local AI runtime.

Make sure you own your AI. AI in the cloud is not aligned with you; it’s aligned with the company that owns it.

3 replies

posted an update 3 days ago

Post

3190

XBai o4 claims to beat Claude Opus 4 and o3-mini, and they provide verifiable proof. My skepticism circuits overloaded, but my local AI FOMO module screamed louder.
I've thrown this 33B monoblock LLM onto a single GPU and used Roo Code for some… let’s call it “vibe testing”. It’s terrifyingly competent. As an architect, it’s the best open-weight model I’ve touched this side of 2025.

1 reply

posted an update 4 days ago

Post

2524

We’ve reached a point where on device AI coding that is free, offline, and capable isn’t just a theoretical possibility; it’s sitting on my lap, barely warming my thighs.
My local MacBook Air setup includes a Qwen3 Coder Flash with a 1M context, Cline in a VSCode IDE. No internet, no cloud, no ID verification- this is the forbidden tech.
Current stats:
All agentic tools work great local, sandboxed, and MCP
OK model output precision
17 tokens/sec. Not great, not terrible
65K tokens context, the model can do 1M, but let’s be real, my MacBook Air would probably achieve fusion before hitting that smoothly
Standard backend and cache off for the test
All inference and function calling happen locally, offline, untethered. The cloud didn’t even get a memo.

3 replies

posted an update 7 days ago

Post

2579

I got 370 tokens/sec of Qwen3-30B-A3B 2507 on my desktop Z8 GPU workstation. My target is 400 t/s, and the last 10 % always tastes like victory!

3 replies

posted an update 12 days ago

Post

2076

I run Qwen3-Coder 480B locally on my Z8, with a 1-million token context window. It’s the equivalent of parallel-parking a Nimitz-class carrier in a kiddie pool. Thanks to whatever dark pact the llama.cpp, CUDA, and kernel folks signed, hybrid inferencing + VRAM↔RAM offload let me stream the model’s synapses across Xeon, RAM, and four lonely A6000s without summoning either the OOM killer or a small house fire.

posted an update 19 days ago

Post

253

I needed a distraction-free AI dev system. Omarchy-AI is an opinionated, minimalist, purpose-built OS layer for AI engineers. Arch Linux, stripped, hardened, and injected with pure, uncut AI developer ergonomics.

TL;DR Omarchy-AI. Vertical AF. One Job, Done Stupidly Well. Only cares about AI engineering.

It’s built on top of Arch Linux & Omarchy and further optimized for:
- Offline, on-the-go AI development. Yes, even on your gaming laptop, or freshly minted DIGITS
- Seamless shift to GPU server backends, because your PC shouldn’t train a 1T Kimi K2 model
- Pre-baked RAG pipelines, agentic workflows, and model fine-tuning
- Actual productivity to spend hours hacking local AI agents, not debugging uv conflicts.

How It Works (The Geeky Bits)
It’s One Curl Command to Rule Them All. Turning a vanilla Arch into a batteries-included local AI dev beast with Hyprland, CUDA, llama.cpp, gcc, and every CLI tool you pretend to know about.

Hyprland: Because Your GPU Deserves Glam Shots. Picked it for the nihilists, tweakers, and keyboard cowboys. It’s an independent Wayland compositor that works great and has zero questions like “how do I get this pretty?”

Make sure you own your AI. AI in the cloud is not aligned with you; it’s aligned with the company that owns it.

1 reply

posted an update 6 months ago

Post

3479

llama.cpp is 26.8% faster than ollama.
I have upgraded both, and using the same settings, I am running the same DeepSeek R1 Distill 1.5B on the same hardware. It's an Apples to Apples comparison.

Total duration:
llama.cpp 6.85 sec <- 26.8% faster
ollama 8.69 sec

Breakdown by phase:
Model loading
llama.cpp 241 ms <- 2x faster
ollama 553 ms

Prompt processing
llama.cpp 416.04 tokens/s with an eval time 45.67 ms <- 10x faster
ollama 42.17 tokens/s with an eval time of 498 ms

Token generation
llama.cpp 137.79 tokens/s with an eval time 6.62 sec <- 13% faster
ollama 122.07 tokens/s with an eval time 7.64 sec

llama.cpp is LLM inference in C/C++; ollama adds abstraction layers and marketing.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

7 replies

posted an update 7 months ago

Post

707

Stargate to the west of me
DeepSeek to the east
Here I am
Stuck in the middle with the EU

It will likely be a matter of sparkle to get export control on frontier research and models on both sides, leaving us in a vacuum.

Decentralized training infrastructure and on device inferencing are the future.

posted an update 7 months ago

Post

558

On device AI reasoning ODA-R using speculative decoding with draft model DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-32B. DSPy compiler for reasoning prompts in math, engineering, code...

posted an update 7 months ago

Post

1456

Training a model to reason in the continuous latent space based on Meta's Coconut.
If it all works will apply it on the MiniCPM-o SVD-LR.
Endgame is a multimodal, adaptive, and efficient foundational on device AI model.

2 replies

replied to their post 7 months ago

DDR5 on HP Z8 G5

replied to their post 7 months ago

exactly Q2 med with ~190GB RAM

posted an update 7 months ago

Post

2535

Can it run DeepSeek V3 671B is the new 'can it run Doom'.

How minimalistic can I go with on device AI with behemoth models - here I'm running DeepSeek V3 MoE on a single A6000 GPU.

Not great, not terrible, for this minimalistic setup. I love the Mixture of Experts architectures. Typically I'm running my core LLM distributed over the 4 GPUs.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

5 replies

posted an update about 1 year ago

Post

2557

I started Friday with decentralized AI using Gemma-2, and it all works without blockchain. This is what I did:

1. Pinned Gemma-2 9B in the Interplanetary Filesystem IPFS with the LoRA fine-tuning adapters.
2. Set up a llama-ipfs server to fetch and cache the model and adapters on the fly and inference locally.

Now, I can use my on device AI platform across:

• All my macOS automation workflows
• All my browsers
• My Copilot++ in VSCode
• My Open Apple Intelligence (OAI, not to be confused with the other closed OAI owned by a nonprofit foundation and BigTech)

The llama-ipfs server’s RPC support lets me decentralize inferencing across all my devices, supercharging computing and energy efficiency.

Make sure you own your AI. AI in the cloud is not aligned with you, it’s aligned with the company that owns it.

2 replies

posted an update about 1 year ago

Post

2233

I'm decentralizing my AI end2end, from the AI model distribution to on device AI inferencing. llama-ipfs - llama.cpp integrated with Interplanetary File System for distributing peer2peer and loading AI models without the need for cloud storage or AI model Hub.

llama.cpp now supports decentralized inferencing with RPC, allowing the distribution of workload across all home devices. This functionality can be enhanced with a P2P ad-hoc VPN, enabling the extension of distributed inferencing to any device on any network.

Imagine an open-source AI that's as decentralized as a potluck dinner - everyone brings something to the table, and there's ZERO need for blockchain. It's like a digital fortress, with security and privacy baked right in, not to mention a dollop of integrity and trust. This could be the secret sauce for an enterprise AI platform, complete with an integrated IT policy. It might just be the cherry on top for the next generation of Apple Intelligence and Copilot+ PCs.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

posted an update about 1 year ago

Post

707

I'm decentralizing my AI. I'll be using Radicle for decentralized Git and IPFS for distributing AI models.

I believe there is a significant opportunity to democratize open AI development moving forward. I appreciate that Radicle is open-source, prioritizes local operations, functions offline, seeds data peer-to-peer from my node, is programmable, and incorporates built-in security features.

IPFS is great decentralized data storage, and I have already begun seeding SLMs and LoRa adapters. Tomorrow will add my collection of LLMs, VLMs, etc models and datasets I'm actively using. I have 10Gbps fiber optics at home so my node has enough bandwidth.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

2 replies

posted an update about 1 year ago

Post

3462

I've made an on device AI comparison between open source, Apple Intelligence, and Microsoft Copilot+ PC. This OS and applications level integration will bring GenAI to everyone, be it consumers or businesses, over the next year.

Communities and BigTech hold divergent visions regarding the problems they aim to solve, ways to lock in users and enterprises, as well as their commercialization and GTM strategies.

I'm aware that this table has the potential to expand into an epic 30-page saga during an in-depth analysis, but hey, it's a beginning. Do you think I should throw in a few more comparisons? I'm all ears for your thoughts and critiques!

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it

1 reply

replied to their post about 1 year ago

The screenshot is a preview of Copilot+ . Using Phi Silica which comes with the Azure Edge AI package.

posted an update about 1 year ago

Post

2439

Me: I want on device AI: fast, without latency, with real privacy, convenient for use and development.

Microsoft: The best I can do is Copilot+. You need a special Qualcomm chip and Windows 11 24H2. Today I can give you only Recall, taking screenshots and running a visual model to write context about what you are doing in the unencrypted Semantic Index database for embeddings. I'm giving you SLMs Phi Silica, accessible only via API and SDK. In the autumn I can give you the developer tools for C#/C++ and you can use them.

Apple: The best I can do is Apple Intelligence. You need a special Apple chip and macOS 15. Today I can give you only marketing. In the autumn I can give you on-device 3B quantized to 3.5bit mysterious SLMs and diffusion models with LoRA adapters. We will have an encrypted Semantic Index database for embeddings and agentic flows with function calling. We will call all of them with different names. In the autumn I will give you the developer tools in Swift and you can use them.

Open Source: The best I can do is llama.cpp. You can run it on any chip and OS. Today you can run AI inferencing on device and add other open source components for your solution. I can give you local AI models SLMs/LLMs - from wqen2-0.5B to Llama3-70B. You can have an encrypted local embeddings database with PostgreSQL/pgvector or SQLite-Vec. I can give you a wide choice of integrations and open-source components for your solution- from UIs to agentic workflows with function calling. Today I can give you the developer tools in Python/C/C++/Rust/Go/Node.js/JS/C#/Scala/Java and you can use them.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

4 replies

posted an update about 1 year ago

Post

2459

I've spent some time checking the promises vs reality of on-device AI between Apple Intelligence and Microsoft Copilot+. Reading the marketing documentation is good, but not enough. Hands-on tests are the best, unfortunately, both are not there yet.

Both are looking to lock developers behind local API to the SLM inferencing engine and SDK mix of open source and proprietary code. Both can not work air-gapped and offline for meaningful workflows, only some basic ones and both require the hybrid AI local/remote plane calling back either APIs on Azure or the Apple Private Cloud Compute.

Some of the Copilot+ functionally is available in Windows App SDK 1.6 exp2. It's focused on the old-school enterprise developers and not sure if they will be the early adaptors of GenAI-backed apps... I still have the Recall on my dev-PC as they have removed it.

Apple Intelligence is hard to get beyond the vague description and the State of the Union video. Even the current beta of macOS 15 and xcode don't have any "A.I." in them. At the moment it is all promises and a lack of technical documentation and code.

Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

2 replies

Mitko Vasilev

AI & ML interests

Recent Activity

Organizations

mitkox's activity