cool um so this time I'm going to talk
about word representation and text
classifiers these are kind of the
foundations that you need to know uh in
order to move on to the more complex
things that we'll be talking in future
classes uh but actually the in
particular the word representation part
is pretty important it's a major uh
thing that we need to do for all NLP
models so uh let's go into it
so last class I talked about the bag of
words model um and just to review this
was a model where basically we take each
word we represent it as a one hot Vector
uh like
this and we add all of these vectors
together we multiply the resulting
frequency vector by some weights and we
get a score out of this and we can use
this score for binary classification or
if we want to do multiclass
classification we get you know multiple
scores for each
class and the features F were just based
on our word identities and the weights
were
learned and um if we look at what's
missing in bag of words
models um we talked about handling of
conjugated or compound
words we talked about handling of word
similarity and we talked about handling
of combination features and handling of
sentence structure and so all of these
are are tricky problems uh we saw that
you know creating a rule-based system to
solve these problems is non-trivial and
at the very least would take a lot of
time and so now I want to talk about
some solutions to the problems in this
class so the first the solution to the
first problem or a solution to the first
problem is uh subword or character based
models and that's what I'll talk about
first handling of word similarity this
can be handled uh using Word edings
and the word embeddings uh will be
another thing we'll talk about this time
handling of combination features uh we
can handle through neural networks which
we'll also talk about this time and then
handling of sentence structure uh the
kind of standard way of handling this
now is through sequence-based models and
that will be uh starting in a few
classes so uh let's jump into
it so subword models uh as I mentioned
this is a really really important part
all of the models that we're building
nowadays including you know
state-of-the-art language models and and
things like this and the basic idea
behind this is that we want to split uh
in particular split less common words up
into multiple subboard tokens so to give
an example of this uh if we have
something like the companies are
expanding uh it might split companies
into compan
e and expand in like this and there are
a few benefits of this uh the first
benefit is that this allows you to
parameters between word varieties or
compound words and the other one is to
reduce parameter size and save compute
and meming and both of these are kind of
like equally important things that we
need to be uh we need to be considering
so does anyone know how many words there
are in
English any
ideas
yeah two
million pretty good um any other
ideas
yeah
60,000 some models use 60,000 I I think
60,000 is probably these subword models
uh when you're talking about this so
they can use sub models to take the 2
million which I think is a reasonable
guess to 6 60,000 any other
ideas 700,000 okay pretty good um so
this was a per question it doesn't
really have a good answer um but two 200
million's probably pretty good six uh
700,000 is pretty good the reason why
this is a trick question is because are
company and companies different
words uh maybe maybe not right because
if we know the word company we can you
know guess what the word companies means
um what about automobile is that a
different word well maybe if we know
Auto and mobile we can kind of guess
what automobile means but not really so
maybe that's a different word there's
all kinds of Shades of Gray there and
also we have really frequent words that
everybody can probably acknowledge our
words like
the and
a and um maybe
car and then we have words down here
which are like Miss spellings or
something like that misspellings of
actual correct words or
slay uh or other things like that and
then it's questionable whether those are
actual words or not so um there's a
famous uh law called Zip's
law um which probably a lot of people
have heard of it's also the source of
your zip
file um which is using Zip's law to
compress uh compress output by making
the uh more frequent words have shorter
bite strings and less frequent words
have uh you know less frequent bite
strings but basically like we're going
to have an infinite number of words or
at least strings that are separated by
white space so we need to handle this
somehow and that's what subword units
do so um 60,000 was a good guess for the
number of subword units you might use in
a model and so uh by using subw units we
can limit to about that much
so there's a couple of common uh ways to
create these subword units and basically
all of them rely on the fact that you
want more common strings to become
subword
units um or actually sorry I realize
maybe before doing that I could explain
an alternative to creating subword units
so the alternative to creating subword
units is to treat every character or
maybe every bite in a string as a single
thing that you encode in forent so in
other words instead of trying to model
the companies are expanding we Model T h
e space c o m uh etc etc can anyone
think of any downsides of
this
yeah yeah the set of these will be very
will be very small but that's not
necessarily a problem
right yeah um and any other
ideas
yeah yeah the resulting sequences will
be very long um and when you say
difficult to use it could be difficult
to use for a couple of reasons there's
mainly two reasons actually any any IDE
about
this any
yeah yeah that's a little bit of a
separate problem than the character
based model so let me get back to that
but uh let let's finish the discussion
of the character based models so if it's
really if it's really long maybe a
simple thing like uh let's say you have
a big neural network and it's processing
a really long sequence any ideas what
happens basically you run out of memory
or it takes a really long time right so
you have computational problems another
reason why is um think of what a bag of
words model would look like if it was a
bag of characters
model it wouldn't be very informative
about whether like a sentence is
positive sentiment or negative sentiment
right because instead of having uh go o
you would have uh instead of having good
you would have go o and that doesn't
really directly tell you whether it's
positive sentiment or not so those are
basically the two problems um compute
and lack of expressiveness in the
underlying representations so you need
to handle both of those yes so if we uh
move from
character better expressiveness and we
assume that if we just get the bigger
and bigger paragraphs we'll get even
better
yeah so a very good question I'll repeat
it um and actually this also goes back
to the other question you asked about
words that look the same but are
pronounced differently or have different
meanings and so like let's say we just
remembered this whole sentence right the
companies are
expanding um and that was like a single
embedding and we somehow embedded it the
problem would be we're never going to
see that sentence again um or if we go
to longer sentences we're never going to
see the longer sentences again so it
becomes too sparse so there's kind of a
sweet spot between
like long enough to be expressive and
short enough to occur many times so that
you can learn appropriately and that's
kind of what subword models are aiming
for and if you get longer subwords then
you'll get things that are more
expressive but more sparse in shorter
subwords you'll get things that are like
uh less expressive but less spice so you
need to balance between them and then
once we get into sequence modeling they
start being able to model like which
words are next to each other uh which
tokens are next to each other and stuff
like that so even if they are less
expressive the combination between them
can be expressive so um yeah that's kind
of a preview of what we're going to be
doing
next okay so um let's assume that we
want to have some subwords that are
longer than characters but shorter than
tokens how do we make these in a
consistent way there's two major ways of
doing this uh the first one is bite pair
encoding and this is uh very very simple
in fact it's so
simple
that we can implement
it in this notebook here which you can
click through to on the
slides and it's uh
about 10 lines of code um and so
basically what B pair encoding
does
is that you start out with um all of the
vocabulary that you want to process
where each vocabulary item is split into
uh the characters and an end of word
symbol and you have a corresponding
frequency of
this you then uh get statistics about
the most common pairs of tokens that
occur next to each other and so here the
most common pairs of tokens that occur
next to each other are e s because it
occurs nine times because it occurs in
newest and wildest also s and t w
because those occur there too and then
you have we and other things like that
so out of all the most frequent ones you
just merge them together and that gives
you uh new s new
EST and wide
EST and then you do the same thing this
time now you get EST so now you get this
uh suffix EST and that looks pretty
reasonable for English right you know
EST is a common suffix that we use it
seems like it should be a single token
and um so you just do this over and over
again if you want a vocabulary of 60,000
for example you would do um 60,000 minus
number of characters merge operations
and eventually you would get a B of
60,000 um and yeah very very simple
method to do this um any questions about
that
yeah
yeah so uh just to repeat the the
comment uh this seems like a greedy
version of Huffman encoding which is a
you know similar to what you're using in
your zip file a way to shorten things by
getting longer uh more frequent things
being inced as a single token um I think
B pair encoding did originally start
like that that's part of the reason why
the encoding uh thing is here I think it
originally started there I haven't read
really deeply into this but I can talk
more about how the next one corresponds
to information Theory and Tuesday I'm
going to talk even more about how
language models correspond to
information theories so we can uh we can
discuss maybe in more detail
to um so the the alternative option is
to use unigram models and unigram models
are the simplest type of language model
I'm going to talk more in detail about
them next time but basically uh the way
it works is you create a model that
generates all word uh words in the
sequence independently sorry I thought I
had a
um I thought I had an equation but
basically the
equation looks
like
this so you say the probability of the
sequence is the product of the
probabilities of each of the words in
the
sequence
and uh then you try to pick a vocabulary
that maximizes the probability of the
Corpus given a fixed vocabulary size so
you try to say okay you get a vocabulary
size of
60,000 how do you um how do you pick the
best 60,000 vocabulary to maximize the
probability of the the Corpus and that
will result in something very similar uh
it will also try to give longer uh
vocabulary uh sorry more common
vocabulary long sequences because that
allows you to to maximize this
objective um the optimization for this
is performed using something called the
EM algorithm where basically you uh
predict the uh the probability of each
token showing up and uh then select the
most common tokens and then trim off the
ones that are less common and then just
do this over and over again until you
drop down to the 60,000 token lat so the
details for this are not important for
most people in this class uh because
you're going to just be using a toolkit
that implements this for you um but if
you're interested in this I'm happy to
talk to you about it
yeah is there
problem Oh in unigram models there's a
huge problem with assuming Independence
in language models because then you
could rearrange the order of words in
sentences um that that's something we're
going to talk about in language model
next
time but the the good thing about this
is the EM algorithm requires dynamic
programming in this case and you can't
easily do dynamic programming if you
don't make that
assumptions um and then finally after
you've picked your vocabulary and you've
assigned a probability to each word in
the vocabulary you then find a
segmentation of the input that maximizes
the unigram
probabilities um so this is basically
the idea of what's going on here um I'm
not going to go into a lot of detail
about this because most people are just
going to be users of this algorithm so
it's not super super
important um the one important thing
about this is that there's a library
called sentence piece that's used very
widely in order to build these um in
order to build these subword units and
uh basically what you do is you run the
sentence piece
train uh model or sorry uh program and
that gives you uh you select your vocab
size uh this also this character
coverage is basically how well do you
need to cover all of the characters in
your vocabulary or in your input text um
what model type do you use and then you
run this uh sentence piece en code file
uh to uh encode the output and split the
output and there's also python bindings
available for this and by the one thing
that you should know is by default it
uses the unigram model but it also
supports EP in my experience it doesn't
make a huge difference about which one
you use the bigger thing is how um how
big is your vocabulary size and if your
vocabulary size is smaller then things
will be more efficient but less
expressive if your vocabulary size is
bigger things will be um will
be more expressive but less efficient
and A good rule of thumb is like
something like 60,000 to 80,000 is
pretty reasonable if you're only doing
English if you're spreading out to
things that do other languages um which
I'll talk about in a second then you
need a much bigger B regular
say so there's two considerations here
two important considerations when using
these models uh the first is
multilinguality as I said so when you're
using um subword
models they're hard to use
multilingually because as I said before
[Music]
they give longer strings to more
frequent strings basically so then
imagine what happens if 50% of your
Corpus is English another 30% of your
Corpus is
other languages written in Latin script
10% is
Chinese uh 5% is cerlic script languages
four 4% is 3% is Japanese and then you
have like
0.01% written in like burmes or
something like that suddenly burmes just
gets chunked up really really tiny
really long sequences and it doesn't
work as
well um so one way that people fix this
um and actually there's a really nice uh
blog post about this called exploring
B's vocabulary which I referenced here
if you're interested in learning more
about that um but one way that people
were around this is if your
actual uh data
distribution looks like this like
English uh
Ty we actually sorry I took out the
Indian languages in my example
apologies
so
um what you do is you essentially create
a different distribution that like
downweights English a little bit and up
weights up weights all of the other
languages um so that you get more of
other languages when creating so this is
a common work around that you can do for
this um the
second problem with these is
arbitrariness so as you saw in my
example with bpe e s s and t and of
board symbol all have the same probabil
or have the same frequency right so if
we get to that point do we segment es or
do we seg uh EST or do we segment e
s and so this is also a problem and it
actually can affect your results
especially if you like don't have a
really strong vocabulary for the
language you're working in or you're
working in a new
domain and so there's a few workarounds
for this uh one workaround for this is
uh called subword
regularization and the way it works is
instead
of just having a single segmentation and
getting the kind of
maximally probable segmentation or the
one the greedy one that you get out of
BP instead you sample different
segmentations in training time and use
the different segmentations and that
makes your model more robust to this
kind of variation and that's also
actually the reason why sentence piece
was released was through this um subword
regularization paper so that's also
implemented in sentence piece if that's
something you're interested in
trying cool um are there any questions
or discussions about this
yeah
already
yeah so this is a good question um just
to repeat the question it was like let's
say we have a big
multilingual um subword
model and we want to add a new language
in some way uh how can we reuse the
existing model but add a new
language it's a good question if you're
only using it for subord
segmentation um one one nice thing about
the unigram
model here is this is kind of a
probabilistic model so it's very easy to
do the kind of standard things that we
do with probabilistic models which is
like let's say we had an
old uh an
old vocabulary for
this um we could just
interpolate
um we could interpolate like this and
just you know uh combine the
probabilities of the two and then use
that combine probability in order to
segment the new language um things like
this have been uh done before but I
don't remember the exact preferences uh
for them but that that's what I would do
here another interesting thing is um
this might be getting a little ahead of
myself but there's
a there's a paper that talks about um
how you can take things that or trained
with another
vocabulary and basically the idea is um
you pre-train on whatever languages you
have and then uh you learn embeddings in
the new language you freeze the body of
the model and learn embeddings in the
new language so that's another uh method
that's used it's called on the cross
lingual printability
representations and I'll probably talk
about that in the last class of this uh
thing so you can remember that
then cool any other
questions
yeah is bag of words a first step to
process your data if you want to do
Generation Um do you mean like
uh a word based model or a subword based
model
or like is
this so the subword segmentation is the
first step of creating just about any
model nowadays like every model every
model uses this and they usually use
this either to segment characters or
byes um characters are like Unicode code
points so they actually correspond to an
actual visual character and then bites
are many unicode characters are like
three by like a Chinese character is
three byes if I remember correctly so um
the bbased segmentation is nice because
you don't even need to worry about unic
code you can just do the like you can
just segment the pile like literally as
is and so a lot of people do it that way
too uh llama as far as I know is
bites I believe GPT is also bites um but
pre previous to like three or four years
ago people used SCS I
cool um okay so this is really really
important it's not like super complex
and
practically uh you will just maybe maybe
train or maybe just use a tokenizer um
but uh that that's an important thing to
me cool uh next I'd like to move on to
continuous word eddings
so the basic idea is that previously we
represented words with a sparse Vector
uh with a single one
also known as one poot Vector so it
looked a little bit like
this and instead what continuous word
embeddings do is they look up a dense
vector and so you get a dense
representation where the entire Vector
has continuous values in
it and I talked about a bag of words
model but we could also create a
continuous bag of words model and the
way this works is you look up the
values of each Vector the embeddings of
each Vector this gives you an embedding
Vector for the entire sequence and then
you multiply this by a weight
Matrix uh where the so this is column so
the rows of the weight Matrix uh
correspond to to the size of this
continuous embedding and The Columns of
the weight Matrix would correspond to
the uh overall um
to the overall uh number of labels that
you would have here and then that would
give you sces and so this uh basically
what this is saying is each Vector now
instead of having a single thing that
represents which vocabulary item you're
looking at uh you would kind of hope
that you would get vectors where words
that are similar uh by some mention of
by some concept of similar like syntatic
uh syntax semantics whether they're in
the same language or not are close in
the vector space and each Vector element
is a feature uh so for example each
Vector element corresponds to is this an
animate object or is this a positive
word or other Vector other things like
that so just to give an example here
this is totally made up I just made it
in keynote so it's not natural Vector
space but to Ill illustrate the concept
I showed here what if we had a
two-dimensional vector
space where the two-dimensional Vector
space the xais here is corresponding to
whether it's animate or not and the the
Y AIS here is corresponding to whether
it's like positive sentiment or not and
so this is kind of like our ideal uh
goal
here um so why would we want to do this
yeah sorry
guys what do the like in the one it's
one
yep
like so what would the four entries do
here the four entries here are learned
so they are um they're learned just
together with the model um and I'm going
to talk about exactly how we learn them
soon but the the final goal is that
after learning has happened they look
they have these two properties like
similar words are close together in the
vectorace
and
um that's like number one that's the
most important and then number two is
ideally these uh features would have
some meaning uh maybe human
interpretable meaning maybe not human
interpretable meaning but
yeah so um one thing that I should
mention is I I showed a contrast between
the bag of words uh the one hot
representations here and the dense
representations here and I used this
look look up operation for both of them
and this this lookup
operation actually um can be viewed as
grabbing a single Vector from a big
Matrix of word
embeddings and
so the way it can work is like we have
this big vector and then we look up word
number two in a zero index Matrix and it
would just grab this out of that Matrix
and that's practically what most like
deep learning libraries or or whatever
Library you use are going to be
doing but another uh way you can view it
is you can view it as multiplying by a
one hot vector and so you have this
Vector uh exactly the same Matrix uh but
you just multiply by a vector uh 0 1 z z
and that gives you exactly the same
things um so the Practical imple
implementations of this uh uh tend to be
the first one because the first one's a
lot faster to implement you don't need
to multiply like this big thing by a
huge Vector but there
are advantages of knowing the second one
uh just to give an example what if you
for whatever reason you came up with
like an a crazy model that predicts a
probability distribution over words
instead of just words maybe it's a
language model that has an idea of what
the next word is going to look like
and maybe your um maybe your model
thinks the next word has a 50%
probability of being capped 30%
probability of being
dog and uh 2% probability uh sorry uh
20% probability being
bir you can take this vector and
multiply it by The Matrix and get like a
word embedding that's kind of a mix of
all of those word which might be
interesting and let you do creative
things so um knowing that these two
things are the same are the same is kind
of useful for that kind of
thing um any any questions about this
I'm G to talk about how we train next so
maybe maybe I can goow into
that okay cool so how do we get the
vectors uh like the question uh so up
until now we trained a bag of words
model and the way we trained a bag of
words model was using the structured
perceptron algorithm where if the model
got the answer wrong we would either
increment or decrement the embeddings
based on whether uh whether the label
was positive or negative right so I
showed an example of this very simple
algorithm you don't even uh need to
write any like numpy or anything like
that to implement that
algorithm uh so here here it is so we
have like 4X why in uh data we extract
the features we run the classifier uh we
have the predicted why and then we
increment or decrement
features but how do we train more
complex models so I think most people
here have taken a uh machine learning
class of some kind so this will be
reviewed for a lot of people uh but
basically we do this uh by doing
gradient
descent and in order to do so we write
down a loss function calculate the
derivatives of the L function with
respect to the
parameters and move uh the parameters in
the direction that reduces the loss
mtion and so specifically for this bag
of words or continuous bag of words
model um we want this loss of function
to be a loss function that gets lower as
the model gets better and I'm going to
give two examples from binary
classification both of these are used in
NLP models uh reasonably frequently
uh there's a bunch of other loss
functions but these are kind of the two
major
ones so the first one um which is
actually less frequent is the hinge loss
and then the second one is taking a
sigmoid and then doing negative log
likelyhood so the hinge loss basically
what we do is we uh take the max of the
label times the score that is output by
the model and zero and what this looks
like is we have a hinged loss uh where
if Y is equal to one the loss if Y is
greater than zero is zero so as long as
we get basically as long as we get the
answer right there's no loss um as the
answer gets more wrong the loss gets
worse like this and then similarly if
the label is negative if we get a
negative score uh then we get zero loss
and the loss increases if we have a
positive
score so the sigmoid plus negative log
likelihood the way this works is you
multiply y * the score here and um then
we have the sigmoid function which is
just kind of a nice function that looks
like this with zero and one centered
around
zero and then we take the negative log
of this sigmoid function or the negative
log
likelihood and that gives us a uh L that
looks a little bit like this so
basically you can see that these look
very similar right the difference being
that the hinge loss is uh sharp and we
get exactly a zero loss if we get the
answer right and the sigmoid is smooth
uh and we never get a zero
loss um so does anyone have an idea of
the benefits and disadvantages of
these I kind of flashed one on the
screen already
but
so I flash that on the screen so I'll
give this one and then I can have a quiz
about the sign but the the hinge glass
is more closely linked to accuracy and
the reason why it's more closely linked
to accuracy is because basically we will
get a zero loss if the model gets the
answer right so when the model gets all
of the answers right we will just stop
updating our model whatsoever because we
never we don't have any loss whatsoever
and the gradient of the loss is zero um
what about the sigmoid uh a negative log
likelihood uh there there's kind of two
major advantages of this anyone want to
review their machine learning
test sorry what was
that for for R uh yeah maybe there's a
more direct I think I know what you're
saying but maybe there's a more direct
way to say that um
yeah yeah so the gradient is nonzero
everywhere and uh the gradient also kind
of increases as your score gets worse so
those are that's one advantage it makes
it easier to optimize models um another
one linked to the ROC score but maybe we
could say it more
directly any
ideas okay um basically the sigmoid can
be interpreted as a probability so um if
the the sigmoid is between Zer and one
uh and because it's between zero and one
we can say the sigmoid is a
probability um and that can be useful
for various things like if we want a
downstream model or if we want a
confidence prediction out of the model
so those are two uh advantages of using
a s plus negative log likelihood there's
no probabilistic interpretation to
something transing theas
basically cool um so the next thing that
that we do is we calculate derivatives
and we calculate the derivative of the
parameter given the loss function um to
give an example of the bag of words
model and the hinge loss um the hinge
loss as I said is the max of the score
and times y in the bag of words model
the score was the frequency of that
vocabulary item in the input multiplied
by the weight here and so if we this is
a simple a function that I can just do
the derivative by hand and if I do the
deriva by hand what comes out is if y *
this value is greater than zero so in
other words if this Max uh picks this
instead of this then the derivative is y
* stre and otherwise uh it
is in the opposite
direction
then uh optimizing gradients uh we do
standard uh in standard stochastic
gradient descent uh which is the most
standard optimization algorithm for
these models uh we basically have a
gradient over uh you take the gradient
over the parameter of the loss function
and we call it GT so here um sorry I
switched my terminology between W and
Theta so this could be W uh the previous
value of w
um and this is the gradient of the loss
and then uh we take the previous value
and then we subtract out the learning
rate times the
gradient and uh there are many many
other optimization options uh I'll cover
the more frequent one called Adam at the
end of this uh this lecture but um this
is the basic way of optimizing the
model so
then my question now is what is this
algorithm with respect
to this is an algorithm that is
taking that has a loss function it's
calculating derivatives and it's
optimizing gradients using stochastic
gradient descent so does anyone have a
guess about what the loss function is
here and maybe what is the learning rate
of stas
gradient I kind of gave you a hint about
the L one
actually and just to recap what this is
doing here it's um if predicted Y is
equal to Y then it is moving the uh the
future weights in the direction of Y
times the frequency
Vector
yeah
yeah exactly so the loss function is
hinge loss and the learning rate is one
um and just to show how that you know
corresponds we have this if statement
here and we have the increment of the
features and this is what the um what
the L sorry the derivative looked like
so we have
if this is moving in the right direction
for the label uh then we increment
otherwise we do nothing so
basically you can see that even this
really simple algorithm that I you know
implemented with a few lines of python
is essentially equivalent to this uh
stochastic gradient descent that we
doing
models so the good news about this is
you know this this is really simple but
it only really works forit like a bag of
words model or a simple feature based
model uh but it opens up a lot of uh new
possibilities for how we can optimize
models and in particular I mentioned uh
that there was a problem with
combination features last class like
don't hate and don't love are not just
you know hate plus don't and love plus
don't it's actually the combination of
the two is really
important and so um yeah just to give an
example we have don't love is maybe bad
uh nothing I don't love is very
good and so in order
to solve this problem we turn to neural
networks and the way we do this is we
have a lookup of dense embeddings sorry
I actually I just realized my coloring
is off I was using red to indicate dense
embeddings so this should be maybe red
instead of blue but um we take these
stents embeddings and then we create
some complicated function to extract
combination features um and then use
those to calculate
scores
um and so we calculate these combination
features and what we want to do is we
want to extract vectors from the input
where each Vector has features
um sorry this is in the wrong order so
I'll I'll get back to this um so this
this was talking about the The
Continuous bag of words features so the
problem with the continuous bag of words
features was we were extracting
features
um like
this but then we were directly using the
the feature the dense features that we
extracted to make predictions without
actually allowing for any interactions
between the features um and
so uh neural networks the way we fix
this is we first extract these features
uh we take these these features of each
word embedding and then we run them
through uh kind of linear transforms in
nonlinear uh like linear multiplications
and then nonlinear transforms to extract
additional
features and uh finally run this through
several layers and then use the
resulting features to make our
predictions and when we do this this
allows us to do more uh interesting
things so like for example we could
learn feature combination a node in the
second layer might be feature one and
feature five are active so that could be
like feature one corresponds to negative
sentiment words like hate
despise um and other things like that so
for hate and despise feature one would
have a high value like 8.0 and then
7.2 and then we also have negation words
like don't or not or something like that
and those would
have
don't would have a high value for like 2
five and so these would be the word
embeddings where each word embedding
corresponded to you know features of the
words and
then um after that we would extract
feature combinations in this second
layer that say oh we see at least one
word where the first feature is active
and we see at least one word where the
fifth feature is active so now that
allows us to capture the fact that we
saw like don't hate or don't despise or
not hate or not despise or something
like
that so this is the way uh kind of this
is a deep uh continuous bag of words
model um this actually was proposed in
205 15 I don't think I have the
reference on the slide but I think it's
in the notes um on the website and
actually at that point in time they
demon there were several interesting
results that showed that even this like
really simple model did really well uh
at text classification and other simple
tasks like that because it was able to
you know share features of the words and
then extract combinations to the
features
so um in order order to learn these we
need to start turning to neural networks
and the reason why we need to start
turning to neural networks is
because while I can calculate the loss
function of the while I can calculate
the loss function of the hinged loss for
a bag of words model by hand I
definitely don't I probably could but
don't want to do it for a model that
starts become as complicated as this
with multiple Matrix multiplications
Andes and stuff like that so the way we
do this just a very brief uh coverage of
this uh for because um I think probably
a lot of people have dealt with neural
networks before um the original
motivation was that we had neurons in
the brain uh where
the each of the neuron synapses took in
an electrical signal and once they got
enough electrical signal they would fire
um but now the current conception of
neural networks or deep learning models
is basically computation
graphs and the way a computation graph
Works um and I'm especially going to
talk about the way it works in natural
language processing which might be a
contrast to the way it works in computer
vision is um we have an expression uh
that looks like this and maybe maybe
it's the expression X corresponding to
uh a
scal um and each node corresponds to
something like a tensor a matrix a
vector a scalar so scaler is uh kind
kind of Zero Dimensional it's a single
value one dimensional two dimensional or
arbitrary
dimensional um and then we also have
nodes that correspond to the result of
function applications so if we have X be
a vector uh we take the vector transpose
and so each Edge represents a function
argument and also a data
dependency and a node with an incoming
Edge is a function of that Edge's tail
node and importantly each node knows how
to compute its value and the value of
its derivative with respect to each
argument times the derivative of an
arbitrary
input and functions could be basically
arbitrary functions it can be unary Nary
unary binary Nary often unary or binary
and computation graphs are directed in
cyclic and um one important thing to
note is that you can um have multiple
ways of expressing the same function so
this is actually really important as you
start implementing things and the reason
why is the left graph and the right
graph both express the same thing the
left graph expresses X
transpose time A Time X where is whereas
this one has x a and then it puts it
into a node that is X transpose a x
and so these Express exactly the same
thing but the graph on the left is
larger and the reason why this is
important is for practical
implementation of neural networks um you
the larger graphs are going to take more
memory and going to be slower usually
and so often um in a neural network we
look at like pipe part which we're going
to look at in a
second
um you will have something you will be
able to
do
this or you'll be able to do
like
this so these are two different options
this one is using more operations and
this one is using using less operations
and this is going to be faster because
basically the implementation within
Pythor will have been optimized for you
it will only require one graph node
instead of multiple graph nodes and
that's even more important when you
start talking about like attention or
something like that which we're going to
be covering very soon um attention is a
very multi-head attention or something
like that is a very complicated
operation so you want to make sure that
you're using the operators that are
available to you to make this more
efficient
um and then finally we could like add
all of these together at the end we
could add a
constant um and then we get this
expression here which gives us kind of a
polinomial polom
expression um also another thing to note
is within a neural network computation
graph variable names are just labelings
of nodes and so if you're using a a
computation graph like this you might
only be declaring one variable here but
actually there's a whole bunch of stuff
going on behind the scenes and all of
that will take memory and computation
time and stuff like that so it's
important to be aware of that if you
want to make your implementations more
efficient than other other
things so we have several algorithms
that go into implementing neural nuts um
the first one is graph construction uh
the second one is forward
propagation uh and graph construction is
basically constructing the graph
declaring ing all the variables stuff
like this the second one is forward
propagation and um the way you do this
is in topological order uh you compute
the value of a node given its inputs and
so basically you start out with all of
the nodes that you give is input and
then you find any node in the graph
where all of its uh all of its tail
nodes or all of its children have been
calculated so in this case that would be
these two nodes and then in arbitrary
order or even in parallel you calculate
the value of all of the satisfied nodes
until you get to the
end and then uh the remaining algorithms
are back propagation and parameter
update I already talked about parameter
update uh using stochastic gradient
descent but for back propagation we then
process examples in Reverse topological
order uh calculate derivatives of
parameters with respect to final
value and so we start out with the very
final value usually this is your loss
function and then you just step
backwards in top ological order to
calculate the derivatives of all these
so um this is pretty simple I think a
lot of people may have seen this already
but keeping this in mind as you're
implementing NLP models especially
models that are really memory intensive
or things like that is pretty important
because if you accidentally like for
example calculate the same thing twice
or accidentally create a graph that is
manipulating very large tensors and
creating very large intermediate States
that can kill your memory and and cause
big problems so it's an important thing
to
be um cool any any questions about
this okay if not I will go on to the
next one so neural network Frameworks
there's several neural network
Frameworks but in NLP nowadays I really
only see two and mostly only see one um
so that one that almost everybody us
uses is pie torch um and I would
recommend using it unless you uh you
know if you're a fan of like rust or you
know esoteric uh not esoteric but like
unusual programming languages and you
like Beauty and things like this another
option might be Jacks uh so I'll explain
a little bit about the difference
between them uh and you can pick
accordingly
um first uh both of these Frameworks uh
are developed by big companies and they
have a lot of engineering support behind
them that's kind of an important thing
to think about when you're deciding
which framework to use because you know
it'll be well
supported um pytorch is definitely most
widely used in NLP especially NLP
research um and it's used in some NLP
project J is used in some NLP
projects um pytorch favors Dynamic
execution so what dynamic execution
means is um you basically create a
computation graph and and then execute
it uh every time you process an input uh
in contrast there's also you define the
computation graph first and then execute
it over and over again so in other words
the graph construction step only happens
once kind of at the beginning of
computation and then you compile it
afterwards and it's actually pytorch
supports kind of defining and compiling
and Jax supports more Dynamic things but
the way they were designed is uh is kind
of favoring Dynamic execution or
favoring definition in population
and the difference between these two is
this one gives you more flexibility this
one gives you better optimization in wor
speed if you want to if you want to do
that um another thing about Jax is um
it's kind of very close to numpy in a
way like it uses a very num something
that's kind of close to numpy it's very
heavily based on tensors and so because
of this you can kind of easily do some
interesting things like okay I want to
take this tensor and I want to split it
over two gpus um and this is good if
you're training like a very large model
and you want to put kind
of this part of the
model uh you want to put this part of
the model on GP 1 this on gpu2 this on
GPU 3 this on GPU it's slightly simpler
conceptually to do in Jacks but it's
also possible to do in
p and pytorch by far has the most
vibrant ecosystem so like as I said
pytorch is a good default choice but you
can consider using Jack if you uh if you
like new
things cool um yeah actually I already
talked about that so in the interest of
time I may not go into these very deeply
but it's important to note that we have
examples of all of
the models that I talked about in the
class today these are created for
Simplicity not for Speed or efficiency
of
implementation um so these are kind of
torch P torch based uh examples uh where
you can create the bag of words
Model A continuous bag of words
model um and
a deep continuous bag of wordss
model
and all of these I believe are
implemented in
model.py and the most important thing is
where you define the forward pass and
maybe I can just give a a simple example
this but here this is where you do the
word
embedding this is where you sum up all
of the embeddings and add a
bias um and then this is uh where you
return the the
score and then oh
sorry the continuous bag of words model
sums up some
embeddings uh or gets the embeddings
sums up some
embeddings
uh gets the score here and then runs it
through a linear or changes the view
runs it through a linear layer and then
the Deep continuous bag of words model
also adds a few layers of uh like linear
transformations in Dage so you should be
able to see that these correspond pretty
closely to the things that I had on the
slides so um hopefully that's a good
start if you're not very familiar with
implementing
model oh and yes the recitation uh will
be about playing around with sentence
piece and playing around with these so
if you have any look at them have any
questions you're welcome to show up
where I walk
through cool um any any questions about
these okay so a few more final important
Concepts um another concept that you
should definitely be aware of is the
atom Optimizer uh so there's lots of uh
optimizers that you could be using but
almost all research in NLP uses some uh
variety of the atom
Optimizer and the U the way this works
is it
optimizes
the um it optimizes model considering
the rolling average of the gradient and
uh
momentum and the way it works is here we
have a gradient here we have
momentum and what you can see is
happening here is we add a little bit of
the gradient in uh how much you add in
is with respect to the size of this beta
1 parameter and you add it into uh the
momentum term so this momentum term like
gradually increases and decreases so in
contrast to standard gradient percent
which could be
updating
uh each parameter kind of like very
differently on each time step this will
make the momentum kind of transition
more smoothly by taking the rolling
average of the
gradient and then the the second thing
is um by taking the momentum this is the
rolling average of the I guess gradient
uh variance sorry I this should be
variance and the reason why you need
need to keep track of the variance is
some uh some parameters will have very
large variance in their gradients and
might fluctuate very uh strongly and
others might have a smaller uh chain
variant in their gradients and not
fluctuate very much but we want to make
sure that we update the ones we still
update the ones that have a very small
uh change of their variance and the
reason why is kind of let's say you have
a
multi-layer
network
um or actually sorry a better
um a better example is like let's say we
have a big word embedding Matrix and
over here we have like really frequent
words and then over here we have uh
gradi
no we have like less frequent words we
want to make sure that all of these get
updated appropriately all of these get
like enough updates and so over here
this one will have lots of updates and
so uh kind of
the amount that we
update or the the amount that we update
the uh this will be relatively large
whereas over here this will not have
very many updates we'll have lots of
zero updates also
and so the amount that we update this
will be relatively small and so this
kind of squared to gradient here will uh
be smaller for the values over here and
what that allows us to do is it allows
us to maybe I can just go to the bottom
we end up uh dividing by the square root
of this and because we divide by the
square root of this if this is really
large like 50 and 70 and then this over
here is like one 0.5
uh or something we will be upgrading the
ones that have like less Square
gradients so it will it allows you to
upweight the less common gradients more
frequently and then there's also some
terms for correcting bias early in
training because these momentum in uh in
variance or momentum in squared gradient
terms are not going to be like well
calibrated yet so it prevents them from
going very three wire beginning of
training so this is uh the details of
this again are not like super super
important um another thing that I didn't
write on the slides is uh now in
Transformers it's also super common to
have an overall learning rate schle so
even um Even Adam has this uh Ada
learning rate parameter here and we what
we often do is we adjust this so we
start at low
we raise it up and then we have a Decay
uh at the end and exactly how much you
do this kind of depends on um you know
how big your model is how much data
you're tring on eventually and the
reason why we do this is transformers
are unfortunately super sensitive to
having a high learning rate right at the
very beginning so if you update them
with a high learning rate right at the
very beginning they go haywire and you
get a really weird model um and but you
want to raise it eventually so your
model is learning appropriately and then
in all stochastic gradient descent no
matter whether you're using atom or
anything else it's a good idea to
gradually decrease the learning rate at
the end to prevent the model from
continuing to fluctuate and getting it
to a stable point that gives you good
accuracy over a large part of data so
this is often included like if you look
at any standard Transformer training
recipe it will have that this so that's
kind of the the go-to
optimizer
um are there any questions or
discussion there's also tricky things
like cyclic learning rates where you
decrease the learning rate increase it
and stuff like that but I won't go into
that and don't actually use it that
much second thing is visualization of
embeddings so normally when we have word
embeddings usually they're kind of large
um and they can be like 512 or 1024
dimensions
and so one thing that we can do is we
can down weight them or sorry down uh
like reduce the dimensions or perform
dimensionality reduction and put them in
like two or three dimensions which are
easy for humans to
visualize this is an example using
principal component analysis which is a
linear Dimension reduction technique and
this is uh an example from 10 years ago
now uh one of the first major word
embedding papers where they demonstrated
that if you do this sort of linear
Dimension reduction uh you get actually
some interesting things where you can
draw a vector that's almost the same
direction between like countries and
their uh countries and their capitals
for example so this is a good thing to
do but actually PCA uh doesn't give
you in some cases PCA doesn't give you
super great uh visualizations sorry yeah
well for like if it's
like
um for things like this I think you
probably would still see vectors in the
same direction but I don't think it like
there's a reason why I'm introducing
nonlinear projections next because the
more standard way to do this is uh
nonlinear projections in in particular a
method called tisne and the way um they
do this is they try to group
things that are close together in high
dimensional space so that they're also
close together in low dimensional space
but they remove the Restriction that
this is uh that this is linear so this
is an example of just grouping together
some digits uh from the memus data
set or sorry reducing the dimension of
digits from the mest data
set according to PCA and you can see it
gives these kind of blobs that overlap
with each other and stuff like this but
if you do it with tney this is
completely unsupervised actually it's
not training any model for labeling the
labels are just used to draw the colors
and you can see that it gets pretty
coherent um clusters that correspond to
like what the actual digits
are um however uh one problem with
titney I I still think it's better than
PCA for a large number of uh
applications
but settings of tisy matter and tisy has
a few settings kind of the most
important ones are the overall
perplexity
hyperparameter and uh the number of
steps that you perform and there's a
nice example uh of a paper or kind of
like online post uh that demonstrates
how if you change these parameters you
can get very different things so if this
is the original data you run tisy and it
gives you very different things based on
the hyper parameters that you change um
and here's another example uh you have
two linear uh things like this and so
PCA no matter how you ran PCA you would
still get a linear output from this so
normally uh you know it might change the
order it might squash it a little bit or
something like this but um if you run
tisy it gives you crazy things it even
gives you like DNA and other stuff like
that so so um you do need to be a little
bit careful that uh this is not
necessarily going to tell you nice
linear correlations like this so like
let's say this correlation existed if
you use tisy it might not necessarily
come out to
TIY
cool yep uh that that's my final thing
actually I talked said sequence models
in the next class but it's in the class
after this I'm going to be talking about
language
modeling uh cool any any questions
or