I love the idea, somehow, that these systems had to be built with domain knowledge integrated into the choices of layers and the order of algorithms, using rather brilliant discoveries for optimizing lines and edges, or whatever was relevant to the problem. That's an order of specificity that's lacking from our modern reliance on "one shot" solutions. Letting a one shot solver grind its brains against creating a custom solution to a specific set of transformations is kind if a closing of the circle when it comes to performance and resources for a certain task.
I'm looking forward to the day we can have an optical device that runs only on light input, that looks like a piece of quartz, that you can place over a handwritten number (0-9) and on the top it displays the number in a clean font. Where the neural network to handle this can be translated into physical transformations once it's discovered, no inference processing power required.
Or maybe the bitter lesson is a flawed over simplification. Computer vision is only possible because we used intuition about significant features in vision to make it tractable for GPUs with custom designed kernels. Tossing a generic model with more compute is pointless, it's the invariants and assumptions that make the models effective.
Same for sequence modeling with attention, highly generic RNNs which can in theory model any sequence are just incapable, regardless of compute thrown at it. And a ton of engineered features like tokens are required to make LLMs work (just using the pure data is not scalable).
Bitter lesson was only true back when the engineered features were hand made vs statistically derived, and then it stuck around making people think a bigger model is all we need to get to AGI, when the reality is the exact opposite.
Technically an N-layer nnet can model "any function" but tossing a 10T plain nnet will not even approach a 10M param model with proper features, structure, and design. I call this the savory lesson (intuition and design matter)
Thanks for putting this together. Very nicely demystifies the weight-space programming phenomenon which I otherwise found quite magical and mysterious.
It's not that the models magically "intuit" meaning from a blob of weights, but they methodically iterate on setting the weights with reasoning similar to a human engineer's reasoning.
Interesting question. My guess is it would learn faster at first, then drift away from the hand-built solution, unless you somehow hold it close for the first part of training. Would love to see someone try it.
I'm looking forward to the day we can have an optical device that runs only on light input, that looks like a piece of quartz, that you can place over a handwritten number (0-9) and on the top it displays the number in a clean font. Where the neural network to handle this can be translated into physical transformations once it's discovered, no inference processing power required.
https://opg.optica.org/prj/fulltext.cfm?uri=prj-7-8-823 (2019).
The mistake was thinking that domain knowledge does not scale alongside the generalized scaling AI.
Same for sequence modeling with attention, highly generic RNNs which can in theory model any sequence are just incapable, regardless of compute thrown at it. And a ton of engineered features like tokens are required to make LLMs work (just using the pure data is not scalable).
Bitter lesson was only true back when the engineered features were hand made vs statistically derived, and then it stuck around making people think a bigger model is all we need to get to AGI, when the reality is the exact opposite.
Technically an N-layer nnet can model "any function" but tossing a 10T plain nnet will not even approach a 10M param model with proper features, structure, and design. I call this the savory lesson (intuition and design matter)
It's not that the models magically "intuit" meaning from a blob of weights, but they methodically iterate on setting the weights with reasoning similar to a human engineer's reasoning.
It's interesting to see how tolerant of tedium the models are.