homesampling

View code on Github

#

Sampling from Language Models with Temperature

Here we sample from the following probability distribution where V is the vocabulary, u1:∣V∣​ are the logits of the distribution and T is the temperature:

P(xi​=Vl​∣x1:i−1​)=∑j​exp(Tuj​​)exp(Tul​​)​

T=1 is normal random sampling.

Here's an experiment that uses these sampling techniques.

19importtorch20fromtorch.distributionsimportCategorical2122fromlabml\_nn.samplingimportSampler

#

Sampler with Temperature

25classTemperatureSampler(Sampler):

#

• temperature is the temperature to sample with

29def\_\_init\_\_(self,temperature:float=1.0):

#

33self.temperature=temperature

#

Sample from logits

35def\_\_call\_\_(self,logits:torch.Tensor):

#

Create a categorical distribution with temperature adjusted logits

41dist=Categorical(logits=logits/self.temperature)

#

Sample

44returndist.sample()

labml.ai