Code Review: Clojure Lexer

Summary

このビデオスクリプトでは、プログラミング言語Clojureに関する議論が行われています。ClojureはJVM上で動作し、Lisp系言語の特徴を備えています。話者はClojureの関数定義とパターンマッチング、そしてレキシカル解析器の動作について詳しく説明し、コードを通して言語の柔軟性と強さを強調しています。また、Clojureスクリプトを実行するためのツールとしてbabashkaというインタープリタの存在も紹介されています。このツールはスクリプト作成に便利で、並列処理も可能と話題に挙がっています。

Takeaways

• 🤔 脚本中讨论了编程语言Elixir和Clojure，以及它们在JVM上运行的背景。
• 📝 强调了代码的可读性，提到了代码清洁的重要性，以及在代码中使用清晰的命名约定。
• 📈 讨论了函数的多态性，即同一个函数名可以有多个不同的实现，这与参数的数量和类型有关。
• 🔍 提到了模式匹配的概念，这是一种编程技术，允许根据输入数据的结构来执行不同的代码路径。
• 📚 讨论了在编程中使用递归，即函数调用自身以简化复杂任务的处理。
• 🌈 幽默地提到了代码编辑器Emacs和Lisp程序员对括号颜色的偏好，暗示了工具对提高编程体验的重要性。
• 📖 提到了`token`文件和`token create`函数，这可能与解析输入字符串并生成相应的标记（token）有关。
• 🔢 讨论了如何处理和解析输入字符串，包括字符位置和字符串切片的概念。
• 📦 提到了状态管理，以及如何在函数式编程中显式传递状态，而不是隐式地在类或对象中维护状态。
• 🧵 讨论了并发和多线程的概念，以及如何在Clojure中实现这些功能，以及它们在脚本和批处理任务中的潜在用途。
• 🚧 以幽默的方式讨论了Bash脚本的复杂性，以及在达到一定的复杂度后，Bash脚本可能变得难以管理。

Q & A

ElixirとClojureの違いは何ですか？

ElixirはErlang VM上で動作する関数型プログラミング言語であり、ClojureはJVM上で動作するLisp系の関数型プログラミング言語です。

Clojureの命名にはどのような誤解があると話されていますか？

話者は、Clojureの名前がJVM上で始まったからと自分で作り上げた誤解を持っていましたが、実際はそうではありません。

Clojureにおける関数のオーバーロードはどのように機能しますか？

Clojureでは、defnマクロを使用して関数を定義し、引数の数が異なる複数の関数をオーバーロードすることができます。

Clojureで文字列を処理する際に使用される'lex'関数はどのような働きをしますか？

lex関数は入力された文字列を解析し、トークンを作成する際に使用されます。入力が0の場合にはエラーメッセージを返し、それ以外では文字列を継続的に解析していきます。

Clojureにおけるパターンマッチングとは何ですか？

パターンマッチングは、関数を呼び出す際に引数に応じて異なる動作を定義することができる機能です。

Clojureでの文字列の'unpacked'状態とは何を意味しますか？

unpacked状態とは、文字列が分割され、各要素が個別に参照できる状態を指します。

Clojureで'token'とは何を意味していますか？

tokenは、文字列から生成された解析済みのデータの単位を指し、特定の操作や識別子を表すために使用されます。

Clojureの'token create'関数はどのように使用されますか？

token create関数は、特定の操作や識別子を表す新しいトークンを作成するために使用されます。

Clojureにおける'lazy sequence'とは何ですか？

lazy sequenceは遅延評価されたシーケンスで、必要な場合にのみ評価され、メモリ効率が良くなります。

Clojureでのエディタのカラーリング機能はなぜ重要だと考えられていますか？

カラーリングはコードの可読性を高めるため非常に重要で、特に括弧が多く使われるClojureでは、色の異なる階層を視覚的に区別しやすくなります。

Clojureの開発においてEmacsの目的は何だと話されていますか？

EmacsはLisp言語の開発者たちが括弧のカラーリングを実現するために開発されたエディタであり、その機能はその後他の言語にも拡張されました。

Clojureで状態を管理する際のアプローチはどのようなものでしょうか？

Clojureでは状態を明示的に関数に渡す必要があり、状態を隠蔽する構造はあまり使用されません。

Clojureにおける'predicate'とは何を意味していますか？

predicateは、特定の条件を満たすかどうかを判定する関数であり、文字列の解析プロセスで使用されます。

Clojureでの'cons'操作とはどのような機能ですか？

consは新しいリストを作成する操作で、新しい要素をリストの先頭に追加することができます。

Clojureでの'babashka'とはどのようなツールですか？

babashkaはClojureのインタープリターであり、スクリプト実行のための便利なツールです。

Outlines

00:00

😀 プログラミング言語クロージャーの紹介とその機能の解説

この段落では、プログラミング言語Elixirとクロージャーについて話しています。話者はクロージャーのスペルを思い出し、JVMで始まったことからそのスペルを覚えるという個人的な話もあります。次に、関数Lexの定義とその動作について説明し、複数の引数を持つ関数のオーバーロードについても触れています。最後に、コードの読み方と理解について話し、コードのクリーンさを重要視する意見も述べています。

05:03

😉 トークンの生成とクロージャーでのリスト操作の解説

段落2では、トークンの生成とリスト操作について詳しく説明しています。トークンの種類や位置、そしてそれらを生成する関数token-two-upとtoken-createの働きについて話されています。また、関数Lexがどのように再帰的に呼び出され、文字列を操作し、トークンを作成するプロセスについても解説しています。さらに、エディタのプラグインやemacsの機能についても言及しており、それらがLispプログラマーにとってどのような意味を持つかについても議論しています。

10:04

🤔 babashkaの紹介とクロージャーでの並列処理の利点

最後の段落では、babashkaというインタープリタとそれに付随するスクリプト用ユーティリティについて紹介しています。babashkaを使用することで、bashスクリプトをクロージャーで記述し、より柔軟性のある並列処理が可能になるという利点について説明しています。また、bashでのスクリプト作成の難しさと、babashkaがそれを簡素化する理由についても議論しており、話者は自身がクロージャーでストリーミングを書いた経験も共有しています。

Keywords

💡Elixir

Elixirは、Erlang VM上で動く関数型プログラミング言語です。このビデオでは、Elixirとクロージャーという別の話題を比較して議論していますが、最終的にはクロージャーに焦点を当てています。Elixirは高信頼性と並行性を特徴としており、分散システムやリアルタイムアプリケーションの開発に適しています。

💡クロージャー

クロージャーは、関数型プログラミングにおける重要な概念で、関数とその関数が参照する変数を束包にまとめたものです。ビデオでは、クロージャーのスペルや由来について冗談めかして議論し、その後は実際にクロージャー言語に関するコードを分析しています。クロージャーは、関数型プログラミングでの高階関数や部分適用などのテクニックに不可欠です。

💡JVM

JVMとはJava Virtual Machineの略で、Javaバイトコードを実行する仮想マシンです。ビデオでは、クロージャーがJVM上で動作する言語であると触れていますが、これは誤った情報です。実際には、クロージャーはGoogleが開発した言語で、独自の仮想マシン上で動作します。この誤解は、話者によるジョークとして提示されています。

💡Lex

Lexは、言語処理において用語解析や構文解析を行うツールやライブラリを指し、このビデオでは特定のコードにおける関数の呼び出しを指しています。Lexは入力文字列をトークンに分割し、次に解析される形に変える役割を果たします。ビデオでは、Lex関数がどのように動作し、コードを解析するプロセスの一部として呼び出されるかが議論されています。

💡関数オーバーロード

関数オーバーロードは、同じ名前の関数を異なる引数型や引数数で複数定義し、コンパイル時に呼び出し元の引数に基づいて適切な関数が選択されるプログラミングの概念です。ビデオでは、defnという特殊な構文を使って関数オーバーロードを実装し、異なる引数を持つ複数のLex関数を定義する例が示されています。

💡パターンマッチング

パターンマッチングは、関数型プログラミング言語でよく使われる技術で、値とパターンを比較して一致するものを見つけるプロセスです。ビデオでは、パターンマッチングがどのようにLex関数の動作に適用され、異なる引数に基づいて異なるコードパスを実行するかが説明されています。

💡トークン

トークンは、コンパイラ理論において、ソースコードを解析する際に分割される基本的な要素を指します。ビデオでは、トークンの作成とその属性（演算子、識別子、文字など）が議論されており、コード解析プロセスにおけるトークンの役割が強調されています。

💡ラムダ計算

ラムダ計算は、関数型プログラミングの基礎をなす数学的モデルであり、無名関数を定義し、関数を第一級オブジェクトとして扱うことができます。ビデオでは、ラムダ計算とクロージャー言語の関係が触れられており、ラムダ計算がクロージャー言語の設計に多大な影響を与えたと説明されています。

💡babashka

babashkaは、Clojureのスクリプト言語として知られており、Unixシェルスクリプトの代替として使用されます。ビデオでは、babashkaが提供する機能と利便性について触れられており、Clojureのコードをスクリプトとして実行し、タスクを自動化することができると説明されています。

💡emacs

emacsは、高度なテキストエディタであり、プログラミングを含む様々なタスクに使用されています。ビデオでは、emacsがLispプログラマーにとって理想的なエディタとして設計されたと触れられており、その機能とカスタマイズ可能性が強調されています。

Highlights

The discussion begins with a decision to explore Elixir and closure.

Misunderstandings about the spelling of 'closure' lead to a humorous moment.

The origin of the 'J' in 'closure' is attributed to its start on the JVM, which is a myth.

The conversation touches on the concept of pattern matching in programming.

A function named 'Lex' is introduced with various input lengths for different operations.

The importance of clean code is emphasized, with a self-critique on the current code's cleanliness.

The concept of recursive functions and their role in processing strings is discussed.

The use of 'cons' in creating new lists is explained.

The transcript includes a debate on the importance of correct spelling and the influence of popular usage.

The participants discuss the process of token creation and the function 'token create'.

The concept of a lazy sequence and its use in programming is introduced.

The transcript includes a humorous take on the history of Emacs and its creation for the purpose of coloring parentheses.

The discussion highlights the absence of a class or state in the presented code, emphasizing functional programming.

The participants delve into the intricacies of passing state to functions in a stateless programming paradigm.

The transcript includes a moment of realization about the nature of 'predicate' and its use in the code.

The concept of casting a lazy sequence to a string is discussed.

The conversation ends with a discussion about Babashka, an interpreter for Clojure that offers scripting utilities.

The practical applications of threading in Clojure are highlighted, contrasting it with the complexities of parallel processing in Bash.

Transcripts

we could do Elixir but I also want to

see closure you want to see closure yeah

we can do closed air uh um let's go like

this uh clo wait how do you spell it'll

just be clj uh someone was being very

cute that's very cutesy for someone who

doesn't know closure I would just not

even know what to do here so we'll start

with closure

uh here uh marker

closure start

uh all right flip you gotta go back a

little bit and get the beginning of

closure in here okay it's with a J

though because like it started on the

jvm yeah it's closure is that really why

it has a j

that is my head that is the myth that I

made up for myself about how to remember

why to spell it that way

[Laughter]

news

closer oh it's because it started on the

jvm what well if we say it enough times

though then that will be what people

think it doesn't matter like Jiff first

GIF you know what I mean like it doesn't

really matter what the Creator says it's

just because the Creator says it wrong

doesn't mean

yeah it doesn't mean I have to be wrong

about things all right so let's do okay

I cannot read this immediately

dude are you just pumped up is this just

list is closer just lisp a lisp yeah

okay so there's a function called Lex

which has an input

if it has length zero

no it's going to say call Lex with a

length 0 on your input or something I

think I actually don't even know how to

read this part

are these off oh no no no So Def n you

can Define multiple different ways to

call it

so I think this is like when we did in

uh

right chat anybody exactly I see I see

what you're saying so def end means that

there's more than one it's the operator

oh nope

there's only one definitely you're

overloading yeah okay nice so it's like

when we did it in um

Airline you know how you could Define a

function with different number length

arguments and then depending on if they

kind of like matched up then it would do

this one right so you call Lex from the

output with just an input so it says I'm

going to call myself starting at spot

zero position zero okay so would that be

this so it doesn't go like this yeah so

then it's gonna do the next one so that

means we have a function this is

function length one this is function

length two because it actually has both

input already and scope and this one and

this one is function length six

uh four I think right where do I where

did you get six well because it still

has input and position but it also has

apparently pred no there's different

paths I'm pretty I'm pretty sure so it's

like the first one is if you call Lex

with one no it's gonna call the first

thing oh maybe two it's gonna do the

second if you call it with four it's

gonna do the last

that's actually like pattern matching

okay I yeah I'm too stupid for this to

tell you the truth

this is this I mean I I'm like it's PK

bit mask

S

I don't know what the chpk actually

means but I think it's like character or

something and then there's oh character

at some position maybe and then there's

the rest of the string

uh

so it's like unpacking the string so you

can't really reference like you you

unpack it at the beginning right it's

Peak

it's Peak Plus rest of spring

based on the input

maybe something like that oh oh that's

because you do zero position and then

the input and this depacks it as yep

yep

okay cool damn okay this is this is can

I I honestly feel emotionally uh

difficult right now okay so let's see if

it's empty we got an end to file nice if

it is so the first thing you do is you

get like your Opera you do that like the

line above there is the first thing that

you do I'm not a it turns it into a

string I think okay by the way shy Ryan

is correct this is not clean code

not clean code I need clean I know it

looks pretty clean to me okay because

there's a zero right here where's the

definite how oh my goodness don't eat

yourself you're gonna what write the

character Zero I don't know in other

places what does zero mean

pathetic all right

um all right so let's see so we have uh

so okay so it gets that so if

conditional if uh we're empty then we

have the end of file right

yep there's no more there's no more

string left to do so we make a list out

of the things that we had or whatever

return a list yep okay so this must

recursively descend back into input or

into deflects by doing yeah skips white

space that's cool next input I wonder

what that means

it uh it must just move it forward one

uh character one like character and then

put I guess yeah so we're sitting right

here okay okay okay uh two op up so

there must be something called the two

Ops somewhere around here

yeah it's in there's another file called

token

two up nice okay so there you go so if

it's within this list I'll call it a

list I don't actually know what it is if

it matches these this this list or

something yep then we will apparently

check out some cons I'm not really sure

what that means that creates a new list

con that's like uh the kind of things

that you would do in uh it's called cons

when you create a new list okay that it

must be short for condescending

yeah it's for people who put the

parentheses on the wrong side of the

identifier

here to tell me how stupid I am uh

uh let's see so

yep I can't actually understand any of

this at this point I've I'm completely

lost first part they make they're making

the list the first thing they put in

there is they say hey let's create a new

token so that's oh create is this file

the whole name space the slashes or

namespace Oh weird okay

then we're going to do a kind which is

an operator

and then we're going to do operator

position

that must be the type of token Plus

its current start

I'm not sure why that all gets past the

cons

yeah I'm a little bit confused but I

understand this a lazy sequence must be

a just a pointer to a string that starts

at the current position up more but then

we do elixir

it's calling Lex again so it's calling

itself the function again oh this one

needs to move forward two spots because

this is a the this is the two style

operation right so this is like less

than or equal to so it's saying hey we

gotta add 2 to our current position and

then do the rest of the string

yeah but

I really don't know what this means

other than I don't know what's going on

in that line because I'm not sure what

op and pass is for oh

oh I think it's because look at the last

the last one is when it gets called

oh maybe not because I was thinking

maybe it's matching the last case of

pred pause input and kind so is it

passing those to there or not no it's

not it's only passing because it

because Lex is only called right here

right it's passing all of them to token

create yes I didn't even see that yeah

it's passing into token you don't have

rainbow parents on so it's impossible to

read this yeah all of the lispers are

angry at you because you don't have

rainbow parents rainbow parents so were

you unable to be a lisper back in the

60s when it was in created

yeah it was impossible they were just

waiting for editor plugins yeah so I

just didn't see that it wasn't so token

create we could go look at that in the

token file TJ do you realize that means

emacs was created

such that

lispers could invent a way to have

rainbow prints that is literally the

goal of emacs was they built a whole

operating system just so they could

color their parentheses I wow it makes

so much more sense now I I have to say

this this is perfectly reasonable now

and before I didn't get it at all but

now now I get it all right so you want

me to go back to token right yeah yeah

so where's like Korea there should just

be a function in here somewhere called

create there we go

so liberal so second is the literal okay

so op must be the literal and this is

the position within the string yep right

yep yep okay weird that they have both

the literal and the and it's offset into

the string but whatever it doesn't

really matter or if it's a single okay

it's in the book it has the two but yeah

okay Single Character whatever pretty

straightforward I think we know what's

going on here

notice so in this one one thing that's

interesting that's a little different

than the other ones we've seen so far

right is that uh

there's no there's no like class or

state or anything at all there's not

even like uh like something that holds

like Alexa right you just have one

function and the function does the whole

thing which is kind of cool

it's like you have to pass explicitly

the state to the function every time

yeah okay yeah yeah it reminds me of

what I used to do see a bunch

see you always I mean you still see it a

lot in a lot of c-style libraries that

you actually hold on and create the

state and pass it to all the underlying

libraries like this is what my websocket

is doing and then it goes off like WS

lay I think does the exact same thing it

gives me that feelings all over again I

kind of get this

hmm

operator operand operand

well sing me an operand okay so is

letters so we do this this must be Lex

so here we go this is the thing this is

this one right here

yeah oh yep so this must be so predicate

so that must be the predicate so

predicate position input kind

and so it's getting the car nice is that

that must be what's happening okay so

like inside of those let statements

those get executed in order and it's

like uh sequentially you're like adding

each of these to the current scope so

you like split the predicate with the

input split with using the predicate on

the input right now you have this

identifier which is the characters

you're gonna move forward this number of

positions and then you get a kind

there's an equal sign in between each of

those right yeah yeah there's an idea

okay so this is casting this thing to a

string because it must be some sort of

lazy sequence or something like that so

it's casting it to a string it's

creating how many positions it actually

moved which is the account of characters

inside of this and then the kind which

is a token identifier if kind isn't

passed in

yep

okay

and then it creates a new identifier and

that says okay go do the rest go do the

rest now right because it's always

calling Lex at the end okay because you

pass it with nil so letter is nil so it

does not have a kind it just does

identifier right here right

wow okay okay I'm with it I'm with it

that was difficult I'm not gonna lie to

you that was kind of difficult I'm glad

I had you here because I don't think I

could we could have worked our way

through it through this without you

well I did write some closure on stream

recently and I didn't run it on the jvm

so boom okay what does that mean

uh lispy clouds in the chat right now

showed me and there's this cool uh like

I you could call it kind of like a run

time

uh it's called babashka or I always I

can't remember because I always think

Babushka but like it's babashka I'm

pretty sure it's definitely uh

it like

it's basically like an interpreter for

foreclosure that you can use and it

comes with a bunch of uh utilities for

like scripting so then instead of

running bash you could write closure and

you get all of these sort of like

different things it was really cool okay

that sounds

it was cool

you could do threading and stuff which

was nice like sometimes you want to kick

off eight jobs wait for them to finish

but you're like it's kind of a nightmare

in bash so it's not a closure shout out

it's not a nightmare and best you not

know about parallel

I do but what if you want to run

different kinds of things it's

complicated

and then they need to they need to go do

other things okay reasonable

it is it's always not that bad in bash

until it is that's true bash has like a

bash is kind of like a digital what do

they call that oscilloscope where it's

like it's not bad or it's bad there's

just like not really an in-between stage

it's a step function it's a step

function you've written not enough bash

or you've written too much bash and

there is nothing in between yep