Turn your regular JavaScript code into a neural net; no training, exact, yet also fully differentiable!
A program's made of if-else, loops, and state. Handwired transforms them into matrix multiplication + reglu over floats. For example:
const a = 5
const result = (a > 4) ? 10 : 12The above can be expressed as basic arithmetic:
const a = 5
// Math.max(0, x) is ReLU. For an integer a, above4 is 1 if a > 4, and 0 otherwise
const above4 = Math.max(0, a - 4) - Math.max(0, a - 5)
// Both branches compute, and above4 picks one
const result = above4 * 10 + (1 - above4) * 12You can calculate that by hand and see that both arrive at the same result.
We then collect the + and * and relu into a giant matmul, and either run it on JS or WebGPU, for fun.
Because we basically only use these 2 primitives:
- the full program stays differentiable
- the weights don't need training since they're written manually
- existing gpu techniques, such as fusion, sparsity, reordering, have a great time optimizing such minimal primitives
This unlocks lots of funny use-cases, like:
- unfreezing the weights and train them (what does that even do? Let's find out!)
- attaching a specific frozen program to a generic neural net and train end-to-end (now a nanoGPT can do both Shakespeare and your particular UI?)
- achieving GPU-level of parallelism with JS code that was written in a traditional single-threaded way (see the JSON search demo that runs in milliseconds over 100k documents)
Try the demos live: https://chenglou.me/handwired/
git clone && bun install, then bun start to run them locally. Check docs/, src/ and template/ to get started! (lab/ is a bunch of old experiments you can check or ignore)
