Day 3 - Lessons from ML
03 May 2023Morning Session 1: Lessons from AI
- LLMs(Large Language Models) give the impression of knowing.
- Language before motion?
- Brain developped and evolved for finer/refine movement/motion control
- However, with those LLMs they came with no motion at all.
- Do we really change wieghts of multiple synapses (ex during this discussion) or only a few?
Morning Session 2
Emre Neftci
- They don’t actually need spikes to do the task? we need to think about practical tasks?
- What are the time steps needed to represent the information content in the signal? and we don’t need that fine time steps or resolution to solve a task?
- Having a coarse time scales still works very well even on tasks that has information content in the order of ms?????
Christian Pehle I couldnt follow :D
Eleni Lessons form ChatGPT
- Attention mechanism:
- Query, key and x: input
- $s = Q.K^T$ looking at teh coexistence of tokens in a sentence then apply a softmax on top of that
- They stack layers of this to go beyong two pairs.
- Query, key and x: input
-
They ask to predict mask words in the sentence
- Lessons:
- Simplicity/ complexity
- Scalability: ChatGPT3: 175B parameters