Posts

This post begins a series of extensions to our previous post, The Neural Matthew Effect: Low Effective Degrees of Freedom in Training. We want to understand, at least partially, how a full-rank matrix can turn into an effectively low-rank one during training. Here, we use low rank in a broad sense: only a small fraction of the neural connections play a decisive role in the function represented by the network. Several quantities can describe this phenomenon, including effective rank and stable rank; see the previous post for their definitions. ...

Introduction Suppose we want to simulate the motion of atoms in a molecule or a material. In principle, the forces on the nuclei come from quantum mechanics. In practice, this is hard because the full electron-nuclear Schrodinger equation is far too expensive to solve at every molecular dynamics step. The usual starting point is the Born-Oppenheimer approximation. Since nuclei are much heavier and move much more slowly than electrons, we treat the nuclear positions as fixed while solving the electronic problem. Let ...

I use the Neural Matthew Effect to describe a recurring pattern in deep network training: a small number of directions, modules, or connections carry most of the learning signal. In parameter space, this appears as low-rank updates; in gradient space, as concentration along dominant directions; at the functional level, as the strengthening of existing circuits; and at the structural level, as increasingly uneven interactions among neurons. Throughout this article, low rank does not mean strictly low algebraic rank. It means low effective rank. For neural network weights, gradients, and the Hessian, strict rank is often unstable: an arbitrarily small perturbation can turn a matrix into full rank, but that does not mean all directions are equally important. The more useful question is not “how many singular values are nonzero?”, but “how many directions contain most of the spectral mass?” ...

Some trivial rambling below; thus, no English. 在高中时由于接触过真正的数学竞赛国家队的成员，知道自己和他们比起来智能水平就像淘宝客服，故在填报志愿时以自己过于愚钝为由放弃了选择数学系。同样是碳基生命，智能竟能如此参差，那如果是能够超脱于生物树桎梏而快速进化的人造智能体，那又将是怎样伟岸的存在？正是抱着这样的想法，我进入了智能科学与工程专业。 随着学习的加深，我发现似乎AGI的实现只需要工程就够了。但脱离了高中的压抑环境后，我的自我价值感逐渐显现，对工程上不求甚解的枚举的容忍度变得更低，于是实现AGI所需要的工程与我对工程的厌恶形成了冲突。再加之我逐渐发现自己的想法其实别人也能想到，无非或早或晚，所以我个人在AGI上的参与其实对于历史车轮而言无足轻重。于是个人偏好的微小叙事与历史进程的宏大叙事都不能够支持我继续在工程上花费时间，所以现在的我离开了工程的赛道（严格来说并不是离开，毕竟我实际上从来没有发过工程论文）。 总结我的大学4年，我会说很多时间是被浪费在工程的细节上的，我看的论文的涉猎面很广，但大部分都是工程。而智能工程专业常规课内的教学也只提供工程实践。（你会说machine learning课程算科学，确实可以这么说，但那涉及的内容太浅显了，几乎是控制论的翻版，而不是真正关于智能的科学。） 一个经典的问题是我会因此后悔吗？但私以为“后悔”本身是个谬论——它假设人能带着现有信息回到过去做出不同选择，这前提就不成立。即便强行假设，如果一个人获得新信息后仍找不到更优解，那或许是由于能力不足，或是因为内在状态未曾更新——用《中国2185》里的话说，“不变就是死了”。所以，只要人还在成长、还在思考，所谓的“后悔”在逻辑上几乎成为恒真命题，反而失去了讨论的意义。 现在我又回到了原点，是选择因自己智力残疾而放弃搞理论AI还是身残志坚。答案已经清晰——我选择后者。在微观层面，这关乎个人的意义坚守；在宏大层面，我盼望能在有生之年，目睹或参与构建AI的大一统理论，将其作为人类献给AGI的一份最后的礼物。在那之后，文明史的主角，就不再是人类了。

What happens when you ssh to a remote machine? This blog will not analyze from a cryptographic way but from an engineering way. We will touch on different components of Linux and how they interact. This blog is a brief introduction. We sacrifice precision for the sake of clarity without compromising the core message. The version of the ssh in the following text is OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f 31 Mar 2020 A different build may have different behaviors. ...

Prerequisite Basic knowledge of the structure of Transformer and RNN Understand how to train AI models in NLP Notations: Symbol Meaning $x_t$ token at time $t$ $x_{:t}$ tokens before time $t$ $x_{a:b}$ tokens after time $a$ (included) and before time $b$ (excluded) $p$ the distribution of ground truth $q$ model’s prediction $v$ vocabulary size (# of different tokens) $d$ embedding dimension (dimension of hidden states) Main This blog will explore some potential factors contributing to the distinction between text generated by LLMs and human’s text. ...

Puzzle Play music “Pascual: España Cañí Pasodoble” to set the vibe. In a bullring, a brave matador is engaged in a splendid fight with a furious bull. The matador is agile, meaning that he can change his velocity immediately (even at accelerations up to 100g), but he has a maximum velocity. In contrast, the bull has no upper limit on its velocity, but it has a maximum acceleration due to its massive build. The bull wants to gore the matador while the matador wants to dodge the bull’s charge. Can the bull knock down the matador in a finite time? Or can the matador always dodge the bull’s charge for as long as he wants? Note: we model both the matador and the bull as a point without volume. ...

The transition from the Lagrangian to the Hamiltonian formulation can be viewed as an operation that reduces a higher-order dynamical system to a first-order one. Specifically, given the Euler–Lagrange equation \[ \frac{d}{dt}\left(\frac{\partial L}{\partial \dot{q}_i}\right) - \frac{\partial L}{\partial q_i} = 0, \] let \[ p_i = \frac{\partial L}{\partial \dot{q}_i}. \] Then we have \[ \dot{p}_i = \frac{\partial L}{\partial q_i}. \] We would also like to introduce an equation that describes the evolution of \( q_i \): ...

How do we prove $x^2+2x+1 \geq 0$ by sum of squares(SOS)? Quite trivial $\text{LHS} = (x+1)^2 \geq 0$ Then, how to prove this by SOS? $$1024 x^{12} + 1024 x^{11} - 2304 x^{10} - 2304 x^9 + 1856 x^8 + 1856 x^7 - 672 x^6 - 688 x^5 + 96 x^4 + 120 x^3 + x^2 - 8 x + 21 > 0$$Note that $$ \begin{align*} & 1024 x^{12} + 1024 x^{11} - 2304 x^{10} - 2304 x^9 + 1856 x^8 + 1856 x^7 - 672 x^6 - 688 x^5 + 96 x^4 + 120 x^3 + x^2 - 8 x + 21 \\ =& \frac{953178310898788044405634417898675810098347314821670074294737731307329417300202257499761536478657283822781507633364642416166327278492431062378651278370657928662612204203551165032096670314311671897547870594071651961090916102146117367523334512377797063623918959641854483477219062452053117354701873293279721322207383101430406097244237282245639156393092313795413669142611049 x^{12}}{855628317881711343628447665505529353642205279926550721248508119951156125518248969209845948736002719280414541370639384443936720924614544681187805334420694749567400421129263862969704529965010066154381893028055449128137526516108909143872797671529580790648030301664407011259998219690161313333579998909002703175128634549361882868091571588610793879524527809926169868860955417600} \\ & +\frac{20730892603614552086030236791584376390584458016906737607737484410976414001063114786504571319170688941259188382102808494189006085284963556422616888222898922617541385760899893376535242468971667371887252593392744651799829664240827458028060434738895363938051839773370848251341085723828176528969737296885610550104284856922624395816262451042617170959861753944569051610115364975096073666531052259740236406867828388007841283785939762576367433335933838418591322151049217429629270341561448035039752629166572755121441300348424582513530988930032979543430868119319256707428034767673 \left(x^6-\frac{537192259959723196414608024276178584105895770740349990419864409450146859446360918141274261404347215931631797055464128276914916157874933032752787127367563050763626059546908325367179084235774333657723622129680953103417722133971649513536014377636992875729296614390814113462253672326830450300193835 x^5}{573436804488876102142203069716235581838573141198680442118406207394949540289519077190803351051243463518468145511138665086473396444014968827035817391723483821802914447475306468210874522306578884970990388833275010785029852472999820007452095481245595738959406070890816673619069129251781055609514481}\right)^2}{6206525292964998561634089879171420703703077222813245725237731062283243057781605384272715894893115688039561739976314528141440276348710982794310284305563578483338134927900379708528960238116906764065435465250988664286812424413087069804415610909501395560804213981736329016091614450301007081812989768771400716805356955326219422447174363315272321570725848811409107568938852180691630516174689224926595786145141068421753037366221895943861317367364438368687862455344412538507280470763541217796022531535278362226193304037113002012436942592035108049513404198504622679784592044800} \\ & +\frac{89049395936672352040168549073731104935508445899958191436161862318618887575473347824917288559803543779125075800645083670230091642794205786249413530141184376511491703526443536337740075607815663586644005996838313922814771433710991249266480332814634194309626307993872905338940207675367318393363919505638197260130676409010048261104018415934500084622706963281661906523350259253983921638707587 \left(x^6+\frac{1250035259177316292055552469386970079830476326367794092844767479864825753282605141675706859648589867835412955837749840441264047429340506451409551822678359781540767273795999093416096481298696046487871220 x^5}{8622143587826850910117502112533330587731675813130997925666043593533539304265371562476845710341277127329332016845355859683327927946544194588985145498192793223955868643181370645313715205428657738385563819}-\frac{250068555521657649046677562622926995400361492206054337746046763604698328360725525321979519080245347110841949526008580342735436820548070105753437244690660363635344349057576996446918551957050730363661467736795517432 x^4}{247000887631547219650362262267498217114359645814025727199754456680281704352551452248260792050311392387975177735397802907626080272289713051779703241439169224797693110058833779475283650861940201841910326516809614621}\right)^2}{341822602918637684462759803331862949268521644937598626784273477398092190810659753651937927998386651744723163423943913812097353506749855595813602901610410889636152000488013806377115269602857144079515823849827076508204604117044432065154126771800620953408005355692583886319844446768050888556173228514949931617518643948118914444108098243115937619771018949965589413350297216046547214310400} \\ & +\frac{114110582938612998704158659404514255051669057393030752659470219065049417311463468557376015892172386233408723378189272123039495190457682922779027347858586303115779007600668186892567924675431960334050645546500616838293445049 \left(x^6+\frac{170372215743602129824191290442816200114290980759432737980698537578762149431655574586767607141481342217936642217045286295 x^5}{79198978232815290130528655478941018199689976888747889206980181686545761334890064169317612189500341634270976992383855892}-\frac{10203953103929899682581976196753751277654908019629863829295675407122586862724024696782342783750780067856543386194463907683978113 x^4}{13451588692120113302853290112643611602282673701089314990715540684831684684396619555438945669199900573439855658045046218504735514}-\frac{1156921360237789757302485482040694552188225079328629147544405729241765179914524617193983587603792399575398269867536010269771984801215 x^3}{579480989267842360973616884762574144214735300369226600485034777161864144519121973828754340483462516803215541892922546046965501207606}\right)^2}{1366111482246106172696369013109580206661129472861465130693169324770543975649855414453178742347115059279451705556403540118352861235205455721219880514785580582513596385591992213465538746668601219072613820297212205392956700}\\ & +\frac{855170648996299453282826318597469637692688774342377279122588090583182914361725138849041618786410014540391013069612470521 \left(x^6+\frac{208824848701751946066177483431560679149593637501714562853898328 x^5}{1622430576418306376968793749919537014833913740547776937187465121}-\frac{1102891048931550800774133577726231389446852281283462307407302195510467736 x^4}{551125008995526254989473607432028592184518611427649841031577162693270089}-\frac{61451711586630000225725826920536158859746855484781118310242171140952718523128 x^3}{593547856562956888467288338364108993067921931542293187544982814791584554100775}+\frac{232473948787618155607377486386406989276269768205862051739439363838088892436 x^2}{237921979767491444391645439166417035735774063213781213491056607955450278625}\right)^2}{1734819496159721395301097762699887999566121344999250931454978497836495899386698353810674840031645076695859122690108416}\\ & +\frac{152625114303613097562970499342414182175024924456131965 \left(x^6+\frac{28563189249509387706470170536420805 x^5}{9315046557962779041251696063459844}-\frac{6454490216193107995064656609442859126128240 x^4}{6064787888379130847288192308311499387086309}-\frac{16732721532793552664373217343255126836816 x^3}{3553024507622509278691554818253293871345}+\frac{2619886181080527170645281731667006 x^2}{10684687385267909341688179834740825}+\frac{879726740067337452506895818594 x}{518598620844921096038838025275}\right)^2}{2087473500409210051198263446146771163219546655799312} \\ & +\frac{79747153878767503695019075279803 \left(x^6+\frac{740175552623215917780704 x^5}{8159383732850597703519153}-\frac{334868393274810914740397696 x^4}{120845381198458989757409079}-\frac{1582860475816375677208 x^3}{7966887686614702087275}+\frac{19812948366188776778252 x^2}{9085183041159657600375}+\frac{1284036836607616 x}{15467432289695097}-\frac{2247064464063328}{5155810763231699}\right)^2}{721328386522315927536627776512} \\ \gt &0 \end{align*} $$ ...

Notation $[n]$ denotes $\{1,2,\cdots,n\}$. $[m:n]$ denotes $\{m,m+1,\cdots,n\}$ if $m\leq n$. It’s an empty set otherwise. $a_{m:n}$ denotes $a_m,a_{m+1},\cdots,a_n$. $\{\{1,1,2^{(3)}\}\}$ denotes the multiset with two 1s and three 2s in it. The Problem Given several positive integers $a_{1:n}$, what’s the maximum value that can be achieved by four basic operations, additions, subtractions, multiplications and divisions? Every number is used exactly once. (Provider: 雷伊布是真的狗 aka 雷狗) For example, given 1,1,2,3, a possible result is $4/3=(1+1)\times 2/3$ but the largest result is $12=(1+1)\times 2\times3$. ...