great um yeah so today we're going to be
talking a little bit about generation
algorithms um this will be sort of a
tour through some of the most common
methods and we're going to talk a little
bit about the theory behind them as well
um if you're looking at the slides on
the website these might be ever so
slightly different um but yeah I'll try
to stop at each section boundary for
questions also feel free to sort of
interrupt at any point for
clarifications so we're starting off
today with some great news um let's say
that you have some friend who maybe owns
a giant tech company and they've gifted
you this absolutely massive new model M
um it's a great model it's pre-trained
with the latest architecture it's
pre-trained on um trillions of tokens of
text it's got seven billion parameters
it looks like a really promising new
model you know it's the top of all these
leaderboards um but if you actually take
your new model M and you sort of open up
this box and kind of Shake It Out maybe
from last class you know a little bit
architecturally what this model might
look like but if you actually kind of
take a closer look at it from a
different angle what you see is that m
is actually just a conditional
probability distribution um you put some
input X into your model and you get some
probability out for any given sequence
that you're sort of interested in
evaluating right um and in particular M
gives you a probability distribution
over all tokens in its vocabulary to
predict like what token you would output
next right and so this is what this
equation says um given some input X and
everything that you've predicted so far
you get the probability of the next
token in YJ and if you multiply this out
over all the probabilities in your
sequence you can calculate the
probability of any output y given your
input X so what this like super fancy
model that you spend a lot of money to
train is really just a conditional
probability distribution um but this
turns out to be okay because you can use
a conditional probability distribution
to do sort of any task that we're really
interested in in NLP um pretty much any
task right so by changing what you
consider your input X and your output y
to be you can can get outputs from this
model for things like translation for
summarization for reasoning Tas um just
by sort of changing what you consider
your inputs and outputs in this
setting but there's sort of both good
and bad things about your model being a
probability distribution instead of just
an oracle that gives you sort of a
single answer for every input um one
kind of nice thing about this
distribution um is that you can get at
an idea of something like confidence
right if you give your model the input 2
plus 2 equals and almost all the
probability mass is on the token of four
you can say like the model predicts with
pretty high confidence that 2 plus 2
equals four um versus if you give it
something that's maybe a little more
open-ended like you ask it to predict
Graham's favorite color and you see this
distribution that's sort of a lot
flatter you know the most likely output
is green but maybe we don't have a lot
of confidence that that's the correct
answer um this is really closely tied
into the idea of calibration which you
guys talked about um I guess a couple of
classes ago now the flip side of this
though is that you know Noti that for
this case like 2 plus 2al 4 not all of
the probability mass is on four um and
so models that are conditional
probability distributions can
hallucinate right um pretty much no
matter what you do there's going to be
some nonzero probability to some output
that's incorrect or
undesirable um in some cases maybe even
offensive something that you don't want
the model to Output um and this is sort
of an artifact of the way these models
are trained if there's some great work
kind of more on the theory side here
that shows that this is actually true
even if everything in your input
training data is sort of correct and
factual and doesn't have any errors
you'll still wind up with a situation
where some nonzero probability mass is
on some outputs that are undesirable or
hallucinatory for sort of most inputs
that you care about evaluating so if we
have these issues how do we actually get
a good output out of the model um and to
do that we're first going to talk about
some sampling methods um but I want to
pause here in case there are of any
questions on this idea of a model is a
conditional
distribution great so we can jump right
in so we have this model right we know
at each step at each token we might want
to decode the distribution of likelihood
over all vocabulary tokens right this
conditional distribution we've been
talking about um for the next time step
and what we want out of this is a good
output um for some definition of good
that we can sort of develop as we go
here so maybe the natural first thing to
try is we have a probability
distribution can we just sample from it
right and this is something called
ancestral sampling so at each time step
we're going to draw a token from this
distribution sort of according to its
relative probability right so if
something has twice as much probability
Mass according to the model we'll draw
it twice as often um and we can sample
from this distribution at each time step
and this is sort of this is sort of a
nice setup um we get exact samples from
the model distribution so using the
setup if you can you imagine like
drawing an almost infinite number of
samples like a ridiculously large number
and you look at their probabilities
you'd sort of get something from this
distribution with exactly the
probability that the real model
distribution is given you um so this is
great this gives us an exact sample from
the model this seems to be exactly what
we want um but you can guess probably by
the fact that we're only like 10 minutes
into class here this is not really the
end of the story um and there's actually
a couple of problems with sampling
directly from our model distribu
the one that we're really going to focus
on first here is this idea of a long
tail so a model like llama and maybe our
new model M um has 32,000 vocabulary
tokens and you can imagine maybe out of
those tokens there might be one or even
2,000 of those tokens that are sort of a
reasonable next thing to predict for a
really open-ended task right but there's
going to be all kinds of things in that
distribution um that are maybe like
punctuation there maybe tokens that
won't actually lead to the correct
answer like there's a lot of things in
this distribution that would be all
really low likelihood and this is fine
these things just get low probability
Mass but the problem is if you give sort
of a small amount of probability Mass to
30,000 different things that mass will
add up pretty quickly um and to see this
we have sort of this illustration here
um I don't know if you can see the
difference between the green and the
yellow but I've also drawn a little bar
between them this is a really longtailed
distribution and the green part of the
distribution which is a lot of tokens
with high likelihood has 50% of the
total probability the Yellow Part which
is all a lot of things that are all
individually not super likely is the
other 50% of the probability and so what
that means is if you're doing something
like ancestral sampling 50% of the time
you'll be sampling something really
unlikely from this long tail um that
seems sort of not like what we want
right um so is there anything we can do
about this and the obvious for solution
here is can we just cut off that tail
like if we know these tokens are not
super likely can we just ignore them and
there's a couple of different ways to do
that um the first of these is something
called topk sampling where we say okay
you know maybe we think there are 10
reasonable like outputs is right maybe
we'll just sample from the 10 most
probable tokens um here maybe we say if
we want to pick top six sampling we'll
sample from just the six most probable
tokens and so in this example you can
see we originally had 10 tokens and
we're going to sample from just the blue
ones just the six most likely tokens
um in this example this distribution is
pretty flat there's a lot of things that
are like kind of likely right so that
those six tokens are only 68% of the
total probability Mass um if we go like
one time step further here we might have
a distribution that's a lot peier most
of the mass is on just a single token
and so sampling from just the top six
tokens actually captures 99% of the
probability mes maybe we say that seems
a little excessive right we don't really
need um maybe all of these tokens that
are all kind of low probability maybe we
just want to sort of sample from the top
half of our distribution or something or
the top
90% um so instead of choosing a top
number of tokens to sample from you
could choose a top amount of probability
and this is something called top P or
nucleus sampling so P here is the amount
of probability from your distribution
you want to consider so if you decide
your p is about like 94% of the
probability Mass you in this first examp
example here would choose almost all of
the tokens you keep adding tokens in
until you reach an amount of total
probability that's about
094 but then when you get to the Second
Step where you have a couple of really
highly probable tokens you'd only need a
couple of tokens to add up to 094 or
even higher than 0.94 and so you would
just sample from a smaller set of tokens
so in top K sampling the total amount of
probability your sampling from can move
around in top P sampling the total
number of tokens you're sampling from
might change
um but maybe we sort of don't want to
impose a strong constraint like we want
like 94% here maybe just what we really
care about is saying that we're not
going to sample anything that's really
really unlikely right another way of
doing this is called Epsilon sampling
where we just sample tokens that have at
least some minimum amount of probability
to them right so maybe we just want
tokens that have probability of at least
0.05 here um in this first um example
everything has at least some reasonable
amount of probability so we're actually
going to sample from our full
distribution and then in the second
example when we have a lot of things
that are really unlikely we'll only
sample from sort of the more likely part
of the distribution um so all three of
these methods are sort of different ways
of trying to cut off the long tail using
sort of different
characteristics the tail of the
distribution though isn't the only thing
we could choose to modify um we could
also choose to modify this sort of
peakiness of the distribution
so if you look here at the middle of
these diagrams say this is your original
distribution over next tokens and maybe
you want to modify some properties of
this distribution like you say I want an
output that's really diverse and
interesting and open-ended like maybe
this is something like story generation
where you want to have sort of a lot of
maybe surprising things in your output
you could say I want to sort of
distribute my probability Mass more over
the token space and you can do this um
by sort of flattening this distribution
like you see on the the right here um
where now there's sort of more
probability Mass spread over this um
like wider set of tokens you could also
say the opposite right you could say
maybe I'm doing something like math
where there shouldn't really be a lot of
correct answers there should be really
only one or maybe only like a few
potential reasonable next answers and so
you can make your distribution peier or
sharper so that more of the probability
mass is on the things at the very top um
the way you do this is you modify y your
loges your outputs of the last layer of
the model before you apply softn so when
you're predicting you get your outputs
of the last layer of the model and then
you apply softmax which turns those
outputs into a distribution right they
all sum up the um like Mass over all
vocabulary tokens sums to one and so
that is sort of a distribution you could
sample from if you divide those Logics
by some number before you apply that
softmax you can make that distribution
flatter by using a number greater than
one or peier by using a number less than
one and this is this type of parameter
is called temperature um you can apply
this with any of the other methods for
sort of cutting off the long tail but
what people will often do is just apply
a temperature and then sample from that
distribution and that's what we call
temperature
sampling so these I think most of you
might already have been at least a
little bit familiar with some of these
methods I want to touch briefly on a
couple of other ideas for modifying this
distribution maybe some more complex and
more recent ideas and the one that I
want to talk about in more detail is
something called contrastive decoding so
the idea here is that we could
incorporate some extra information at
decoding time um using some other
distribution some other data or in this
case some other model so if you've ever
played around with a really like
relatively small language model maybe
something like gbt2 small um You
probably noticed you try to give it some
inputs and maybe it degenerates into
just repeating the same sequence over
and over maybe it gives you outputs that
are just completely incorrect like you
ask it a factual question and it gets it
wrong um and you don't see those
problems if you look at sort of a larger
model that's trained on more data so the
question here is can you use what that
smaller model is getting wrong to make
your larger model even better um and the
way we do this is by sort of the
intuition that if the smaller model
doesn't have a lot of probability on
some answer but the the larger model
does it's likely because that larger
model has learned something with the
smaller model didn't know and so here we
modify the probability distribution
coming out of the larger model to choose
outputs that that model thinks are very
likely and the amateur or the the weaker
model thinks are not
likely so in this example here from
their paper um if you have sort of a
input like Barack Obama was born in
Hawaii he was born in L um the smaller
model would often do something like
start repeating and actually if you
sample sort of naively from the um
larger model you can wind up in these
situations as well right so if you just
choose the most likely thing at each
step you wind up in this Loop where it's
like he was born in Hawaii he was born
in Hawaii he was born in Hawaii um and
this is behavior we generally don't want
um if you do something like nucleus or
top PE sampling you can wind up with
things that are actually completely
incorrect like he was born in Washington
DC um but if you use contrastive
decoding you take the outputs coming out
of your expert model here and you
subtract out the probabilities coming
out of the weaker model and you can wind
up with things that the higher model the
stronger model ascribed probability to
but the weaker model did not likely
because these are sort of facts that the
larger model knows that the smaller
model does not so here we actually get
the year Barack Obama was born which is
maybe a fact that the larger model knows
and the smaller model didn't know um and
so this is just one of sort of a broad
class of methods where you use external
information to improve your decoding by
modifying this distribution at each
set um those are sort of a brief tour of
a couple of different sampling methods
before we move into search
yeah
yeah is it going to improve upon just
the yeah it generally does um and the
intuition for why this might be I think
is that there are sort of these
degenerate cases like just repeating
over and over that both the expert and
the weak model would give relatively
high probability to um maybe the expert
model is like slightly less likely to do
these things but it's still like sort of
an easy case for the model to learn and
so both of those models will have high
probability for those things but the
things that are genuinely like good
outputs that only the expert would get
right those will have low probability
under the weak model and so you're sort
of subtracting out all the degenerate
behaviors and keeping to really good out
this if you're generating a longer
sequence with with
contacing how do you know which steps
you want to bring out yeah this is a
great question so for this particular
case oh yeah sorry so this was if you're
doing contrastive decoding over a really
long sequence like when do you choose to
bring in the expert right and for
contrastive decoding we're actually
going to do this at every individual
time step so we're going to use the
expert model to decode and we're going
to bring in the amateur to sort of
subtract out probabilities at each next
token prediction um you don't have to do
that I think that's that's what they do
in the paper um you could also decide to
only do this sort of if you have high
uncertainty or something if you don't
have a really sharp probability
distribution
yeah yeah how weak should the weak
predictor be um in the in the paper what
they're look at is actually not a huge
difference between the two models so you
can see here this is gpd2 XL and small
so there's a difference in parameter
counts and like a bit of a difference in
data I think here but these are actually
not like gpd2 XL is certainly not like a
super strong model now um I think they
try a couple of different settings and
the general intuition I think if I'm
remembering it correctly is that you
want a model that's not like so close in
performance to your expert that you're
basically just subtracting out useful
things but you also don't want a model
that's like so degenerate that it is not
hasn't learned anything useful about
your task at all so I think it might
depend on what task you're looking
at
yes this is for inference um so actually
everything we look at today will not
require aning of the
model Okay cool so now we're going to
step into sort of a slightly different
um set of strategies here which is maybe
we don't just want something from the
model distribution or something from a
modified distribution maybe we actually
just want the quote unquote best thing
the single most likely output given our
input right and here this would be the Y
hat the single sequence that satisfies
that has the highest score py given X
for the X that we gave the model um this
is this section is called mode seeking
search because this is the mode of the
distribution over outputs if you sampled
a huge huge number of times and you
looked at the single most likely
sequence you got it would be this y hat
and so how do we find this
thing well one idea is we know the
distribution at each individual setep
can we just pick the most likely thing
from that distribution and so in Greedy
decoding we take the argmax the single
highest probability token at each step
and we continue generating until the
single highest most the single highest
probability token is the stop token
right the end of sequence token
um for an individual token right if we
only want a single token output this is
exactly what we want this is the single
most likely output um and that's great
but if we're looking at something that
is maybe several tokens long are we
actually going to get the highest
probability thing and if you kind of
squint at this you can see that maybe we
have a problem here where the highest
probability sequence that you get from
multiplying across multiple steps
doesn't necessarily start with the token
that was highest probability at time
step one right maybe if you're doing
something like unconditional generation
the highest probability token at time
step one is always the but there could
be a really probable sentence that just
doesn't happen to start with the the
word the' and you would never find it
using GRE
decoding so this isn't going to give us
the highest probability output over a
sequence that's more than one token one
can we do anything better to try to find
this um
output and here we get into sort of one
of the most popular decoding methods the
one that you maybe heard of before which
is beam search the idea here is that we
don't want to miss a high probability
token that's hidden behind a lower
probability prefix so we want to kind of
search through a couple of different
options so that we don't discard
something too early that might have high
probability um later on in generation
and this is a type of bread first search
so we're going to look at a wide variety
of options at a given time step we're
going to pick some set of them to
continue and then we're going to look at
a wide variety of options for the next
time step instead of generating all the
way through a sequence and then
generating all the way through another
sequence um and how this works is we're
going to pick sort of a number of
candidates we'd like to explore a beam
with so in this example we're going to
pick three and we're going to say all
right here are maybe three options for
time step one for if we pick each of
those three options what would be the
three most likely things for time step
two right rather than choosing just the
single most likely thing in Greedy
decoding we're going to pick three
options and so now we have three options
for time step one three options for time
step two we now have nine options um
here right three options and then three
more for each of these and we don't want
to continue doing this because this is
going to sort of combinator explode so
we need to choose some subset of these
to continue with and the way we do that
is we look at the probability over this
two token sequence and we choose the two
that have the highest probability
overall so in this instance we've chosen
sort of one thing from this first group
and two things from the second group and
now we're back down to three hypotheses
each now two tokens long and we'll
continue generating to time step three
we'll get nine options we'll pre it back
down to three and we'll continue until
the end of generation where we now have
three sequences and we'll just pick the
one that's highest probability out of
those three to return um this is not
guaranteed to get you the highest
probability thing right you still have
this risk that you could be sort of
pruning out something that's high
probability but in general this sort of
works um much better than greedy
decoding and this is if you have a
language model and you're sort of not
what um decoding method it's using outs
are pretty good it's either beam search
or temperature samping right this is
very effective this is used um pretty
broadly there are however some issues
with beam search and one of the biggest
ones is that when you're doing this
maximum likelihood sampling you really
or the sampling to search for something
that's very high likelihood um you
really sacrifice a lot of diversity in
your outputs and in particular you could
wind up at the end of beam search with
three different outputs to choose from
that are all pretty pretty much the same
like they're slightly different token
sequences but they look very similar and
so maybe you want to S get sort of a
more diverse set um there's a couple of
different methods in this category I'm
going to very briefly shout out two of
them um but the idea here is to sort of
reintroduce some of the benefits of
sampling while still doing this kind of
search for high probability things um
diverse beam search is one of these
methods and here the idea is that we
want to modify that scoring step when we
choose which three out of our nine beams
we want to continue
to avoid choosing things that are really
really close to each other right so
maybe our highest probability thing is
some sequence a and then if we look at
the other sequences there's one that's
pretty high probability but very similar
to that sequence and there's one that's
like slightly lower probability but very
different and so maybe we would choose a
sequence that is a little lower
probability to maximize diversity in our
set to try to get like sort of a wider
range of options to choose from later in
generation so this modifies the scoring
to not just take into account likelihood
but also similarity to other
KS another option down this path is
stochastic beam search where we're going
to keep the scoring the same but rather
than choosing just the top three most
likely tokens to expand out each beam
we're actually going to sample from some
distribution and you could sample from
the model distribution directly using
ancestral sampling or you could use any
of our sampling methods we talked about
in the last section to do this and the
the idea here is sort of similar to
diverse beam search we want to get sort
of a wider exploration of our models
like output space you know we want to
sort of explore more things instead of
just seeking winding up with a bunch of
outputs that look very similar at the
end of beam search um if folks are
interested in these I think these are
both linked on the website um the the
papers that both of these ideas came
from
Yes um for stochastic
resarch the sampl probability takes into
account the current part that we already
travel okay
yeah exactly so it's this um like
selection step here but we're instead of
just doing greedy selection we're going
to do
assembling yes my question was on the T
yeah like you for something super simple
like if both of them have a high are you
like yeah so you would if it has a
really high probability under both
models it would have a lower probability
after doing this sort of contrasted
de right so if the if the smaller
model's really good at your task this
might not work very
well yeah I think in the paper they're
generally evaluating on these sort of
like open ended generation task I bet
this works a lot worse for
now
yes
you yeah this is a great question um and
so the question is how do we measure
similar beams um you can sort of Define
any kind of similarity function you like
here um anything that you'd use to
evaluate like how similar something is
to a gold reference right um I think in
the original diverse beam search they do
this by looking at like exact token
match across the two right like if these
beams are the same in all but one of the
tokens or they have like you know 50% of
the tokens are shared across the beams
and maybe these are really similar and
they should try to choose two things
that are different um but you could swap
that out for any
metc yes so
the there's kind of like a that's Happ
at
every for the stochastic be search
there's like a shering what do you mean
by a shepher so it says modify the next
sech selection because they're like um
it is searching at a different space and
it's not searching within the same 3D
SE is it searching in a different space
yeah so it's um in the same probability
distribution but it'll see a different
part of the distribution so when you're
doing the grey search you'll only ever
look at the top three tokens in the next
token distribution because you're just
selecting like the maximums um but in
sampling you could you could get the
same tokens right if they're really high
likelihood but you could also sample
something that's further down in the
distribution yeah as a followup to that
like into uh our stamping we take into
account the probability of the prefix
like the current hypothesis right
because otherwise it is the same as just
uh
in yeah so in the sampling we're taking
into account the previous the prefix
yeah so so it we will take into account
the prefix but this sampling mechanism
here could be ancestral sampling um the
only the difference here is that we're
also doing a sort of search step on top
of that to choose the maximum likelihood
things across multiple
me another important thing um is you
sample without replacement and so
normally you sample with replacement and
you might get exactly the same thing but
when you're doing stasic beam search you
sample without replacement so you get
like three ones according to the
probability but they're guaranteed to be
different right so beam search like one
of the characteristics of beam search is
you always get three different things
because you're picking the three top
when you do sampling uh like stochastic
Bean shirts you get three different
things they're not guaranteed to be the
top they could be distributed according
to the prob distribution but they're
guaranteed so um you can take a look at
the paper for more details of exactly
how it looks but that that's
so then is the main difference that
compared to plus temping that we have n
options that we're cheing tet instead of
going with the going with only one and
you can't yeah you can't simple the same
thing
right yeah so just uh repeat recording
is that n options we're keeping track of
and they're all going to be unique token
sequences at least um you can actually
get the same output sequence from two
different toen sequences if you tokenize
slightly differently um but these will
always be unique tokens
Le so that was sort of a a why like a a
set of methods that we've developed to
try to find the most probable sequence
out of the model um but in the next
section here we're going to sort of
think about whether that's actually what
we want to do at all um so what is like
is do we really want the highest
probability thing um we know that
outputs with really low probability tend
to be really like worse than outfits
with high probability right maybe I'm
trying to predict like what the next
sentence should be after the cat saw the
dog right the cat sat down is way higher
probability than the cat grew wings and
at least with the cats I've met that
sounds pretty that sounds pretty much
right right like this is a much better
output than the cat gr wings but if you
look at just the outputs with relatively
high probability it's sort of less clear
that this defines an exact ranking
between those outputs right
is the cat sat down necessarily better
than the cat ran away these both seem
like pretty reasonable outputs to me
even though one of them is slightly
higher probability and so we do we
really like necessarily need to recover
the cat that down um and this gets a
little a little more complicated still
if we look at sort of a range of outputs
so say there's sort of six outputs that
our model could give us um and here
we're looking at sort of full sequences
not individual tokens just for clarity
so maybe our outputs in order of
probability are the cat sat down it ran
away it sprinted off it got out of there
it's very small and it grew Wings right
so we're definitely sure that the cat
sat down is a better output than the cat
grew wings and if we're doing a mod
seeking search we would find that as our
most likely thing if we're if we you
know do a good job searching and we'd
return that as our output but if you
look at the rest of this distribution
you see that there's actually a whole
set of outputs after that all say
something that kind of means the cat
left the area right it's just that this
probability is split over these three
different generations and if you
actually add up the probability mass of
all three of these sequences this is
double the probability mass of the cat
sat down but because none of these
individual sequences is higher
probability if you're doing mode seeking
search you wouldn't you wouldn't be able
to see this effect right so do we really
want to return the cat sat down or do we
want to return something that means the
cat left the
area the question then is like if it's
not probability that makes an output
good what is it so we have this one
output that's really high probability
but it's very different from everything
else in our set and then we have a
couple of outputs that are all pretty
high probability and similar to a bunch
of other relatively high probability
things so maybe it's sort of less risky
to return one of these right are thing
that's higher probability but different
than everything else could be different
because it's way better or it could be
different because it's way worse um
another way to think about this is you
know maybe if you and your friends were
cheating on a test which you shouldn't
do but if you were going to do it and
all of your friends sent you their
answers um maybe one of your friends has
a slightly higher score in the class
than everyone else but they said the
answer was answer a and everyone else
said the answer was B right you still
might go with the answer that everyone
else said because like what there's it
sort of feels less risky like maybe
everyone else got the answer get that
answer and so your one friend could be
right when everyone else is wrong or
they could have made a mistake that no
one El else is making so this is sort of
the same concept right we want an output
that's relatively high probability but
also relatively low
risk and so here maybe if we were using
this criteria we'd return the cat ran
away as our sort of as our sort of
single
output so how do you find something
that's high probability and low risk
there's sort of two questions here right
we have to figure out how to estimate
probability and if we're looking at a
set of outputs like the six we saw
before maybe we can just do this by
counting right we could sample
everything from the model and get exact
probability or we could take a sample
from the model and just look at
probabilities in that set and from there
from that sample um sort of one
reasonable thing to do is just count
frequency right if something's in our
sample twice as often we just say it's
twice as frequent or it's twice as
probable um this is something called
Monte Carlos sampling if you do this um
enough times like if you sample an
infinite set this is would give you
exactly the model distri distribution um
but for the sort of reasonable size sets
we're working with maybe like a 100
samples this gives us a sort of
reasonable approximation for what we for
what we need to do here at least so
we're just going to take a sample to get
probability and we're just going to
count things in that sample to see how
likely things are that doesn't seem too
bad how do we estimate
risk the idea here is that we have a
bunch of other things in this set of
outputs and we can treat those as sort
of like pseudo references right we can
evaluate agreement between the thing
we're looking at and each of those other
references and this is sort of the same
idea of calculating similarity in
diverse beam search we're going to use
some kind of metric to compare how
similar these things are um this metric
could be anything you use Downstream it
could be like an engram overlap metric
like Rouge or blue or it could also be
something um neural or semantic like um
something like BT score or Bart
score and so this concept um is a type
of decoding called minimum based risk
decoding
and what this equation captures is
exactly the intuition that we were um
sort of talking about just a slide ago
where we're going to choose something
that is low risk which means it's
similar to a lot of other things in this
set of outputs we've sampled and we're
going to choose something that's
relatively high probability which means
that sort of when we sum up over this if
something occurs in our set a bunch of
times it's going to have pretty strong
weight in picking which um of these
outputs are similar right if sort of
there's one thing in the set that
appears a bunch of times it's going to
have a strong influence on which thing
we pick and so that kind of captures
high probability in this
setting so to see how this works we can
look at an example um in
summarization so we choose some Metric
maybe we choose um Rouge which is an
engram overlap metric for summarization
and we say we're going to sample 100
things and we're going to use this
equation to choose the one that has the
sort of lower EST risk according to MBR
um so if we do that and we look at this
sort of table of results here um you can
see that this
outperforms the other sampling methods
that we've looked at before so greedy
decoding here is just sampling the
single most likely thing in each step
beam search here is the BS with five or
10 beams and DBS is the diverse beam
search we were talking about um if we
use minimum based risk and we use grou
is the sort of determiner of similarity
we do way better across all of our
metrics but we especially do really good
at Rouge because that's sort of the
metric that we've been using to evaluate
and then if we swap this out for other
metrics you still see an performance
improvement over these um search methods
here um what's the sort of catch here
the catch here is that MBR requires you
to sample a hundred things and so this
is a lot more expensive it's a lot
slower at infin
time um yes
yes a great question why does the beam
search with more beams perform worse um
this is a well a relatively welln
phenomena called the cursive beam search
which is we actually lost your M so you
mic and we can speak okay yeah so this
is called the cursive beam search um and
the idea here is that beam search is
like an approxim search right so if you
add more beams you should be doing
better and better at finding the maximum
likelihood thing and generally you are
you get something that is higher
probability but as you add more beams
you also often get something that does
worse on your Downstream
metrics back up
there is that back online
yeah is that back is that any louder no
it
question oh there we go is that better
great um yeah so why why does this
happen right why do you get something
that's higher likelihood but um lower
performance Downstream um and this is
like another sort of degeneracy of beam
search that this idea that the thing
that is the absolute highest likelihood
in your distribution might not actually
be what you want
Downstream um this is sort of one of the
other things that people use to motivate
why you might want to do something like
MBR instead um and there's a great paper
about this problem called the inadequacy
of the mode because beam search is
looking for the mode of the
distribution well one other thing I'd
like to mention is it also goes together
with how you train your models because
most of our models are trained using
maximum likelihood maximum likelihood
isn't explicitly maximizing our ability
to get the best answer it's explicitly
maximizing our ability to estimate the
the distribution of answers so if I
say um if you said like what is what is
your favorite hobby or something like
that uh what is your favorite hobby in a
dialogue system often it'll answer I
don't know or something like that
because it like you know that that's
more likely than answering any specific
hobby like it's more likely than
answering basketball bowling you know
whatever else because you have many many
different options and so like especially
if it's something that's a little bit
more comp complicated it will avoid
answering that and in particular it ends
up answering very short things for
example um or sometimes it ends up
repeating itself over and over again or
or things like that so it also goes
together with like the training of the
model yeah and this is um one of the
this is still a problem in modern
systems so if you actually look at the
single like if you could enumerate
everything and see the single most
likely sequence it's often the empty
sequence just not opening anything at
all um and so if that's your true mode
of the distribution then doing better at
mode seeking is not always like
helpful
yes could this be influenced by the
confidence problem like um how
so seems
bees
[Music]
might right I think I I think I see
what you're saying which is that like
the the confidence gives you the
confidence of like a single exact
sequence right not the like actual sort
of semantic space of and so yeah if you
looked at just like the if you look at
just the probability scores you get the
probability of an exact string when what
you really actually care about with
confidence is the probability of sort of
like things that mean the same thing
yeah this is um part of why like
calibration is really hard for long
sequences
great so we're g to touch sort of
briefly on a couple of other things that
aren't sort of always explicitly
described in this framework but that you
can think of as variance of minimum
based risk um and if you're interested
in this analysis um I think as Graham
mentioned earlier um Alex Z is a first
year MLT and I wrote a paper about this
um which you can check out if you're
interested so the um two that I really
want to touch on here are other sort of
inference time things you can consider
which might look a little bit different
on the first BL um the first of these is
output ensembling so say you have
multiple different models and you get
outputs from all of them and now you
need to choose a best output among that
set um one of the sort of common ways to
do this is to compare like an embedding
similarity across models like does model
one think these two things are really
similar does model two think these two
things are really similar and try to
choose something that the um has really
high similarity with a lot of other
outputs um of course now that we've just
recently been talking about MBR you can
see that you can probably see that this
is um the same general formulation just
rather than summing over a set of
outputs from a single model now you're
looking at outputs over a whole set of
models um so some types of ensembling
fall into this category of minimum based
risk methods another thing in this
category is a um sort of recent decoding
method called self-consistency and the
idea here is that you want to do
something like mathematical reasoning
and you really care about getting the
final answer right but you don't
necessarily care about getting all of
the the reasoning steps in between right
so you prompt the model for an answer um
using something like Chain of Thought
right you ask it to sort of talk through
the steps it's going to do and then give
you a final answer um you sample many
puts using this and then you completely
throw away the chains of thought um and
you just take the answer from each
output um you have that set of answers
maybe you have like 20 30 100 answers
you just return the one that was most
frequently
generated um what this is doing is a
type of MBR where the metric that you
actually care about is exact match of
this answer right ignoring the rest of
the
generation um and so here we have sort
of the same intuition that we want an
output
that is high probability right we're
getting it generated a lot but also low
risk not a lot of the other outputs in
our in our set disagree with this
answer so those are a couple of
different variants of methods where
we're sort of sampling a wide set of
sequences and trying to choose the best
one um MBR is one set is one type of
sort of sequence set reranking method um
you could do other things to rerank sets
as well but this is sort of one
representative class of these yes uh or
of the of these methods before we get
into constrain generation those are sort
of the three broad categories of
inference methods we'll discuss which is
sort of sampling from some distribution
searching over some space of
distributions and doing some kind of um
analysis over a set of samples to choose
which ones they
return um does anyone have any questions
at this
point
yeah that a model
cannot yeah like why is averaging model
weights not MBR um I think it's not MBR
because the two um the key thing that I
think really makes a method MBR is this
concept of comparing between multiple um
sort of pseudo
references um and there you don't have
the same like you aage model way can you
wind up with sort of a single output on
the end that maybe is like using like
information from these two model
distributions that you've sort of smush
together um but it's not the same
concept of like comparing against pseudo
references or ranking in a
set right so now this is sort of a this
was a wide variety of methods to try to
find an output that's just sort of good
right we want an output that that is
nice out of our model um but now we'd
like to maybe enclose a few additional
constraints so say I'm asking our model
for some Hobbies I could use to stay in
to stay in shape and no matter what I
don't want the model to recommend
climbing like I I just I don't want this
as an option I've tried it I'm not a fan
um how do I get the model to stop
suggesting climbing to me and if you've
sort of played around with some of the
more recent llms you'd say maybe this is
easy right you just tell the model the
instruction that you don't want to talk
about climbing and having talked to Bard
recently I can tell you unfortunately
that it's not that easy so I tell the
model I don't want to talk about
climbing it does okay for a little bit
and then it's like all right but maybe
you want to try rap climbing um and so
we could continue trying to instruction
to our model but maybe there's sort of a
way to impose this constraint on the
decoding side instead and so I'd say all
right I'm going to do something dramatic
right I know I can manipulate the
probability distribution I'm just going
to set the probability of climbing to be
zero I don't want to see this token like
I'm I'm completely over it um and this
is sort of nice in some sense because
this is pretty easy to do um remember
we're doing a soft Max over the outputs
to choose this probability distribution
and so if we add a huge negative number
to the logic for climbing before we do
this softmax its probability is going to
be basically zero and we're never going
to see it as an output um but this
doesn't seem like a perfect solution
right because you know what if the model
recommends bouldering to me do I have to
write like a sort of a list of every
possible climbing synonym in the world
um what if there's sort of an allowable
way to use this token like I want the
model to suggest hiking because climbing
up a mountain to see a good view is
relaxing but that's a use of the word
climbing and we just said that we can't
use the word climbing um or what if we
sort of generate other related terms
before we get to the restricted term
like the model starts suggesting maybe
you can work out by going to an indoor
rock blank and then what are we going to
say there's not we can't say rock
climbing so maybe the model suggests
rock climbing is rock collecting is a
hobby to stay in shape and that doesn't
sound good either um you could continue
like sort of engineering more and more
complicated rules here but maybe we
could do something that's a little
simpler so what if I just sample a bunch
of outputs from the model and then I
check if they're about climbing and I
get rid of them if they are right um
this is sort of the advantage that it's
pretty easy to check after the fact if
the sequence has satisfied this
constraint you know we could train some
smaller model to guess if the topic of a
sentence is about climbing could check
for keywords we could have a friend
who's willing to see this content like
filter through it and throw everything
out that's not about climing that is
about climbing but if this model um
ascribes really high likelihood to this
like if this model was trained on you
know data from CS PhD students this
could be an extremely high likelihood
suggestion and so we might need to
regenerate hundreds or thousands of
sequences to find something that's not
about climing um and that feels a little
bit inefficient right so is there
something that we can do that's a little
bit better than that well really we'd
like to guess at some point during our
generation if the sequence is going to
be about climbing and maybe like
recalibrate or you know we could even
restart or sort of shape Our Generations
so that we don't wind up with a sequence
that's about climbing in the first place
um one of the methods that we'll discuss
to do this is a method called fudge um
and unfortunately in their paper they
don't have the same anti-climbing bias I
do so this example is actually about
formality instead um the idea here is
that we want a sequence output of the
model that is sort of satisfies this
constraint of being formal and the way
we're going to do this is at each step
of prediction we get the outputs of what
the model predicts is the next token
right this sort of distribution here in
blue and we also have some second
distribution which says given sort of
what we have so far How likely is this
to be a formal sentence at the end right
does a sentence that starts do you want
have a high likelihood of being formal
versus a sentence that starts do you
prefer and so this sort of guess at what
will be formal at the end of the um
generation will put High likelihood on
things that result in really formal
sentences like do you prefer or do you
thus whereas the original model might
have higher likelihood on things that
are maybe more commonly said like do you
want um so we combine these two
distributions you can just multiply them
together and then we sample from this
modified distribution which now has some
sort of high weight on things that the
model thinks are likely but also takes
into account the likelihood of
satisfying a constraint um this is
another sort of method of modifying or
sampling distribution um with some
external information here and so there's
results and sequences that wind up being
sort of more likely to be formal without
having to sample a whole bunch of
sentences and reject the ones that we
think don't satisfy this constraint so
how do we get sort of a guess of what
will be formal at the end of Generation
Um this is where the name fudge comes
from the fud stands for future
discriminator and so what they do is
they train a model on prefixes to guess
whether that sequence will be formal um
you can do this if you have a bunch of
data that's sort of sorted into formal
and not formal right every um sort of
prefix of a sentence in the formal
category is a training example right you
know a sentence that starts do you
prefer you can shop off each token to
get sort of a um set of sequ of prefixes
to sequences that have the label formal
and you can do the same thing to your
informal set and train a discriminator
to choose between them to say like
what's the probability the sentence but
will belong to the formal set when we
finish and so this idea of sort of
trying to guess at a given decoding step
if we're going to wind up with our
constraints satisfied at the end um is a
sort of key way to do constraint
decoding um and one that we'll return to
in just a couple slides here
I want to talk touch on something
slightly different which is that maybe
one of the constraints we care about is
something a little more nebulous like we
want to match human preference um the
way that we usually accomplish this
constraint is a little bit different
right um this we' usually do through
like reinforcement learning through
human feedback um and so we take sort of
our original model distribution and we
take a sort of really like tight like
distrib tion of evidence that says like
um this model says that this sequence is
really high reward this sequence is
really low reward and we try to sort of
combine them somehow through training so
we get a new model that is um quote
unquote aligned and that it has like a
higher likelihood of giving us things
that have really high reward according
to our reward distribution um you can
view this though as a type of basian
inference and so what this means is the
distribution that we really want to get
at the end is a distribution that
combines our original models
distribution and some idea of like How
likely we are to satisfy the reward
right um this we do through
reinforcement learning but if we sort of
know what these two distributions look
like we've we've just been talking about
a lot of methods that modify the
original models distribution with
external information it seems like maybe
we could just add that external
information in at decoding time to get
some of the same
effects um and it turns out you can do
exactly this so this is a paper from
last year called reward augmented
decoding and the idea here is sort of um
in the same conceptual class as fudge
but instead of um predicting whether
we're likely to satisfy the constraint
we're predicting how much reward we
think that sequence will have at the end
of generation so we take our original
model without doing any rhf and we get
the output we get the predictions for
the next token and then we use a model
that's been trained to predict the
likely reward given some prefix like a
future discriminator and we calculate
the likely reward if we pick each of
those tokens and then we use the
combination of those two distributions
to choose what to decode next um and
this sort of gives you some of the
benefits of rlf without actually having
to do reinforcement learning so it's a
way of treating like aligning to human
feedback as just another constraint that
you can impose at decoding point
so those were sort of a a subset of the
um constrains decoding strategies that
people use um before we get into the
human and the loop stack are there any
questions on
this yes for
the do you have
to right so for the discrimin do you
need to train one for every constraint
and you do yeah so you need to have some
set of data that satisfies your
constraint and some set of data that
doesn't before you can enforce a new
constraint in an alternative might be
like in the paper that's what they did
but an alternative might be just to
train a discriminator to determine
whether any constraint was violated so
if you have 100 constraints you could do
a binary prier about whether any
constraint is violated and then
also
sufficient but if you wanted to add a
new constraint you'd still have to
retrain or you have to retrain
or the the reason that this is sort of
relatively reasonable to do is that this
determination of if a constraint is
likely to be violated is sort of a a
lighter weight or an easier task to
learn you can use a relatively small
model for this versus like your big
model just that has to be able to
predict the next token for any sequence
anymore yeah another another like
interesting thing is if you think about
it normally you're predicting with your
big
softmax like this over all of your
vocabulary you can even use the same
representations here to predict with a
binary classifier uh whether the
constraint is violated let's say you
have 100
constraints this is still a vector of
size 100 compared to your vector of size
32,000 that you're using for llama right
so it's not like this adds the training
would cost some time but it adds very
little like inference time I guess
basically the rock
sound so when you do the constraint you
use like a more General
like do
notest
or I guess like in that constraint for
you can add
like, is there
like is there a way to generalize your
constraint would be like don't talk
about this whole set of hobes um you
could do that by training a
discriminator um by training one
discriminator that considers all of
those or by training like a hundred
different discriminators and then um
sort of taking like the maximum score
from any of them right like you want to
you want to be able to exclude all of
these things so you consider if any of
them are violated yeah and for um reward
augmented recoding how do we sort of
like frame that reward model or is that
just come from the previously done rhf
data that the store from there and then
you sort of like FR another
discriminator but this one
now I I fully understand yeah so how do
we get the the reward model here this is
we can use the same data that we' use
for rhf but we need a slightly different
model so for rhf we'll train a reward
model over full sequences right and here
we need to do the same trick where we
sort of look at just prefixes and try to
guess the reward Downstream but if we
have already have preference data then
we have some um like we have a data
source to do this with I think if I'm
remembering correctly they also had a
couple more sort of tricks for data
augmentation to get this to work this is
sort of like a non-trivial thing to
figure out um because like reward is
generally a secret bual
attribute and also if you don't know
very much about rhf we're going to cover
that the future class so don't worry if
this is a yeah sorry to Jump Ahead a
little no no
wores
yeah application this like why would we
doing this to ensure it could be like
our llm would want to highlight certain
qualities like we want our evence to be
more
empathetic is there
something yeah like what are the real
world applications like could we use
this to make L more empathetic or
something yeah any any real attribute
that you can sort of collect like
positive and negative data for you could
do this kind of constraints for I think
the the ones you see most commonly are
the human preference and then like
negative constraints like you don't want
your model to generate offensive content
and if you can build like a good
discriminator for is a sentence going in
a really offensive Direction you can
kind of stop it from gener
yeah would it be a good idea if you
generate a bunch of cons and ask the
model itself whether it violates the
yeah you could do that for sure could
you ask like could you generate a bunch
of samples and ask the model if it
violates the constraint um this is also
a type of sort of sample and then rerank
strategy um but yeah this would be sort
of a more um clever like less
heavyweight version of this checking if
it's about climate means right you'd
like ask the model if it violated the
constraint and if it's a good enough
model it could probably do that pretty
well I suppose in that case you don't
have to thing anything yeah yeah and
this is sort of a general like the
generating text that like satisfies a
constraint is harder than checking if a
text satisfies a constraint so even if
the model isn't good about like not
generating text about climbing when you
tell it to it might be able to tell if
text is
about yeah yeah so how do
you
have different
different you have
to yeah like how do you collect the data
to train this discriminator um generally
you're going to see like you'll look to
see if there are data sets that already
captured this attribute or you could
sort of write her istics to try to
recover it if it's an attribute that not
a lot of other people care about like
you could write your puristic to check
if text is about climbing for instance
um and then try to recover what noisy
samples of data that is or is not about
climbing maybe you could scrape a
climbing forum and then scrape like a
hiking forum and use the difference
between them um but for a lot of tests
there's actually pretty good data sets
already out there for this so there's
like in there's a lot of style transfer
tasks that are like go from informal to
formal or go from this to that or like
make this text in an iic contamin and
you can find like data from those
sources
we never like talked about F yet but I'm
really curious with like the word a
beting whether it would perform better
than like fineing on RF like certainly
more
efficient but I I was I think this is a
comparison they make in their paper but
I don't remember their pun on yeah um in
general there's this sort of a like you
can pay a onetime kind of heavy cost to
fine-tune or you can pay costs at
inference time every time to make sort
of a to make your model better in any of
these ways and depending on how much
inference you're playing do like one or
the other of these could be
better
great so now we're going to talk about
sort of methods for introducing human
interaction into the decoding process
and everything we've looked at so far
has been very sort of black booss kind
of hands off right like you give the
model M some input maybe we do some kind
of manipulation on the decoding side you
get one output back right um but in a
lot of situations where maybe you have
some high-risk application and you need
somebody to be consistently monitoring
and maybe intervening or you're doing
something where you want to do some kind
of human AI collaboration um and you
want to be able to go back and forth or
you want to have a conversation with the
model what you're actually doing is sort
of a series of decodings with human
intervention in between
um and I'm going to talk about a couple
of these strategies briefly I think if
you've used sort of a modern llm you're
probably familiar with at least a few of
them already um we'll sort of put names
to each of them and the set of examples
that we're running with here are from a
paper called wordcraft which is about um
story generation with llm assistants but
these can also be applied sort of more
generally to any kind of task where
you'd want to go back and forth with a
model um the sort of easiest or maybe
simplest place to start here is just
with interleaving text right you can
choose when the model starts and stops
decoding and you can choose when a human
is writing text in between and you can
condition your model in sort of a
mixture of human and model generated
text to choose what to continue next um
you can also do something like have the
model generate a set of text edit that
text in some way maybe the human is
imposing some really subtle constraint
like I want it to sound like my writing
style we don't have a discriminator for
this but the human can sort of modify
the text and then continue generating
from that point and that will influence
the style of the text that continues
being generative um a this case here is
sort of a you're writing a story
together and so you're going back and
forth and editing the text like that but
you can also think of any kind of
conversation with a model as the same
kind of interleaving of text right the
model gives some um text you provide
some text you go back and forth on like
who's providing the text that conditions
the
model you also might want to do things
like more fine brain replace
so here the person has highlighted some
text and said like make this more
descriptive or shorten this to two words
or maybe you want some additional
constraint like can this be happier can
this be sad like change the ending or
something um you can accomplish this in
a variety of ways um here this is done
through input manipulation so you prompt
your model differently with different
constraints you can also do this with an
actual modeling change like if you want
some kind of infilling model um
particularly for things like code this
can be helpful so you want context from
left and right sides um or you can do
this with the decoding changes that we
talked about in the previous section
right you could add a discriminator for
descriptiveness of text or you could do
some kind of sampling ranking method to
recover a more descriptive
output another thing that's very common
in this space is sampling and reranking
methods where the human is the one
choosing what to return right so in
wordcraft you see a set of choices and
you can choose text to insert but more
commonly in something like um chat gbt
or Bard you see this little option to
regenerate text right you as the human
can reject the text and say like no I
don't like this give me a different
output and this is also sort of a way of
controlling decoding um just by doing it
on on a human rather in an algorithmic
level of course you don't necessarily
need a human in here and so um some
recent work has looked at functionally
using models to make these decisions
instead um this is a a a prompting paper
called free of thought which was sort of
very popular on Twitter last summer um
and the idea here is that you're going
to generate um several smaller sequences
um like a couple of sentences a
reasoning step or a thought in the paper
and you're going to use a model to
choose which ones to continue and you
can do different sort of constraints
here like I want to sort of rank this
set of three or maybe I want to predict
if any in this set is wrong like is this
a good reasoning step and if the model
says no you no longer continue that but
the idea here is through prompting
really achieving something that's sort
of if you squint at it looks a lot like
beam search right instead of doing a um
like token level thing and making a
decision based on likelihood you're
generating sort of several sentences out
a time and making a decision based on
this models feedback right this signal
from an external source which here is a
model but could also be a human if
you're willing willing to sort of wait
around for them to make the decision and
so this is a way of sort of giving
feedback on a broader level than single
tokens um to guide a decoding process to
a final
outut so the last couple of things we'll
talk about here are sort of practical
considerations speed choosing decoding
methods um but I can take any questions
before that
to
great so how do you make this fast and
in particular if you've ever tried to
sort of Benchmark performance of a model
what you realize pretty quickly is that
the vast majority of time is actually
spent in decoding you have to generate
one token at a time you have to sort of
pass that back through the model to get
conditioning to generate the next token
and so this is um generally fairly slow
um this is sort of a a major impediment
if you're d to do something like a
streaming application where you want or
a chat application where you don't want
the person to be waiting around for an
answer um one way to do this is a method
called Spectra of decoding and this is a
method where you're using a smaller
model um not as like we're in contrast
of decoding right we're using a smaller
model to decide what not to generate but
here we're using a smaller model to
decide be what to generate um and the
idea here is that most of these tokens
are maybe not super hard to side it's
just that occasionally the bigger model
might want to go in a different
direction so these green tokens here are
generated by a smaller model our amateur
model here and the larger model acts
largely as a verifier and what it does
is it checks if the output so far is
going in a an a Direction that's sort of
in distribution for the big model like
something that's within the realm of
what it might SLE and to there's sort of
an involved discussion in this paper of
how you determine if something is in
distribution um so here the smaller
models generates like five or six tokens
that the larger model says okay this
looks great until it hits a token that
the larger model would not have
generated in that circumstance and then
the larger model rejects that token and
generates a different token instead so
you can see here each of these red and
then blue sections is where the larger
model has rejected something and has to
actually autor regressively decode a
single token by contrast if you were
doing regular decoding at each
individual token in this sequence the um
larger model would have had to make the
fall forward pass to decoda token so
here rather than de doing maybe what
probably like 20ish decoding steps to
get this full sequence the larger model
has done about eight decoring steps and
everything else is able to sort of
verify a block of tokens at once um this
sort of idea of like using a smaller
model as an approximation is pretty
powerful um and there's some great um
followup work cons specul decoding and
sort of ways to do this faster or with
stronger
guarantees um but this General concept
is I would bet probably how models like
um part of how models like chat GPT or
Bard are sort of generating text so
quickly um there's another element here
which is like the model architecture
being sparse but I think that um if you
folks talk about mixture of experts we
might get into that
later um how do you do this kind of fast
inference um libraries like BLM will
Implement things I think Implement
speculative decoding and Implement sort
of Hardware level tricks like choosing
which attention um weights to Cash wear
to do faster inflence um there's also
great libraries for doing things like
constraint decoding so things like
outlines will let you set constraints
like I want my outputs to all be Json
and it will impose additional
constraints during decoding to ensure
that that happens and then pretty much
anything in these first couple of
sections we talked about um like
sampling mode seeking search and
sometimes MBR will also be implemented
in pretty much any Library you use for
models like huggingface Fair seek or
Jacks so to kind of take a step back
here is when you get to the end of class
um there's really two broad categories
of methods that we talked about today um
given our initial distribution from the
model for a next token given our our
input we can do two kind of different
things we can each individual decoding
step choose some kind of function to
manipulate this distribution and this
could be something like short like
cutting off the long tail like modifying
the temperature or adding external
information from another model or from a
discriminator model
right or we can over a larger part of
the decoding process choose some
function to choose between sequences and
this could be like choosing between next
tokens in beam search when we pruning
beams this could be choosing from Full
sequences when we're doing something
like MB r or sample and rerank methods
um and you can do these two things in
parallel right you can choose like a
different function to manipulate the
next token distribution and then some
sort of like broader thing to choose
what you do with the full sequences you
get out of that distribution um but
there are sort of these two broad
categories of
decoding so what should you take away
from this um I think a couple of things
you decoding methods can be really
powerful to control features of your
output if you want to impose particular
constraints if you want to factor in
reward function or factor in a data
source that you maybe didn't have at
training time um and to some extent you
can do a more expensive decoding method
to compensate for a worse model or to
compensate for a model that hasn't been
trained to do exactly the thing you want
it to do um of course you can't you know
use this to make gpt2 small as good as
gp4 but you can sort of for some points
in the middle spend more um computed
inference time to pay for not spending
as much computed training time and
particularly if you don't have access to
the kind of giant gpus you might need to
continue fine-tuning your model this can
be a really a really powerful
alternative um yeah so say like you're
building like something in production
right people usually do um sort of like
that you know inance before cling to see
if it's G to work at do
that like try to see like if you have a
model that you can do some kind of
expensive decoding method for to get
good outputs is it then worth try
training that model right um there's
some great recent work on like training
models to produce the same kind of
outputs you get out of MVR without um
actually doing a really expensive
inference Stu so at some level like yeah
you can decide like this model is good
enough with its expensive method we can
try to make it cheaper by spending more
money on
funing um but that's not it's not like
necessarily guaranteed that that's will
be the case
Okay um the methods that we looked at
have these sort of trade-offs in quality
in diversity and in inference speed so
sampling from your model directly is
pretty fast to do you get really diverse
outputs but it tends to be lower quality
um whereas more restricted sampling
these sort of mode seeking search
methods tend to be higher quality but
you get less less diverse outputs and
that's why we have these methods like
diverse and stochastic resarch to
counter this a bit um and then methods
like MBR or other sample and rerank
methods tend to be very high quality
outputs but you pay for this with much
slower inference
time um but if I can kind of convince
you of anything today I think it would
be this which is that these the decoding
method you choose for your model has a
really strong impact on performance
Downstream um you can get radically
different results out of the same model
without doing any additional training
just by choosing the different decoding
method that you might want to try and so
when you sort of let your libraries pick
a quote unquote like sensible default
you can leave a lot of performance on
the train on the table so I encourage
you folks that if if you're um deploying
models in production or if you're doing
research or you know maybe look at your
outputs and your model has some
undesirable behaviors to consider if the
decoding method you're using is imposing
some kind of Intuition or some kind of
inductive bias and if you can alter that
to get some of these behaviors without
resorting to additional training
um and that's sort of the end I can take
any other
questions okay um yeah I guess we don't
have any questions we can take questions
up here um one one thing I'd like to
point out also is that um I I love the
final thing that Amanda said here
another thing is that my impression from
dealing with things is that it's a lot
easier to predict the effect of
inference time decoding time
manipulations than it is to predict the
effect of
like um fine-tuning or something like
this like just to give a an
example beam search with the maximum
likelihood trained model tends to
generate things that are shorter um
whereas greedy decoding tends to
generate things that are longer and
repeat more often and stuff like that
and if you try a few methods like this
you'll quickly find these kind of qus of
each of the methods and so by forming a
good intuition of this you will also
know how to fix these problems when you
see them it's like oh my model's
repeating itself a lot maybe I shouldn't
be using grey search I should be
switching over to something else or
something like that so um this is a good
thing to know and play around with yeah
and I think pretty underutilized too um
a lot of folks will not think about a
decoding method to fix their problem
even if like your model might actually
be perfectly fine under a different
decoding strategy
great okay thanks a lot everyone you can
uh
finish