NLP（二十六）：如何微调 GPT-2 以生成文本

Artist,Songs,Popularity,Link,Genre,Genres
10000 Maniacs,110,0.3,/10000-maniacs/,Rock,Rock; Pop; Electronica; Dance; J-Pop/J-Rock; Gospel/Religioso; Infantil; Emocore
12 Stones,75,0.3,/12-stones/,Rock,Rock; Gospel/Religioso; Hard Rock; Grunge; Rock Alternativo; Hardcore; Punk Rock; Chillout; Electronica; Heavy Metal; Metal; World Music; Axé; Emocore
311,196,0.5,/311/,Rock,Rock; Surf Music; Reggae; Ska; Pop/Rock; Rock Alternativo; Hardcore
4 Non Blondes,15,7.5,/4-non-blondes/,Rock,Rock; Pop/Rock; Rock Alternativo; Grunge; Blues; Pop; Soft Rock; Power-Pop; Piano Rock; Indie; Chillout
A Cruz Está Vazia,13,0,/a-cruz-esta-vazia/,Rock,Rock
Aborto Elétrico,36,0.1,/aborto-eletrico/,Rock,Rock; Punk Rock; Pós-Punk; Post-Rock
Abril,36,0.1,/abril/,Rock,Rock; Emocore; Hardcore; Pop/Rock; Rock Alternativo; Romântico; Hard Rock; Blues; World Music
Abuse,13,0,/abuse/,Rock,Rock; Hardcore
AC/DC,192,10.8,/ac-dc/,Rock,Rock; Heavy Metal; Classic Rock; Hard Rock; Clássico; Metal; Punk Rock; Blues; Black Music; Rockabilly; Psicodelia; Funk Carioca; Rock Alternativo; Trilha Sonora; New Age; Hip Hop; New Wave; Sertanejo; Post-Rock; Pop/Rock; MPB; Electronica; Grunge; Progressivo; Pop/Punk; Funk; Forró
ACEIA,0,0,/aceia/,Rock,Rock
Acid Tree,5,0,/acid-tree/,Rock,Rock; Heavy Metal; Metal
Adam Lambert,110,1.4,/adam-lambert/,Pop,Pop; Pop/Rock; Rock; Romântico; Dance; Electronica; Emocore; Power-Pop; Axé; Gótico; R&B; Punk Rock; Pop/Punk; Black Music; Rock Alternativo; World Music; J-Pop/J-Rock; Gospel/Religioso; Hip Hop; K-Pop/K-Rock; Piano Rock; Heavy Metal; Velha Guarda; Soul Music; Hard Rock; Country; Soft Rock; Tecnopop; House; Trilha Sonora; Blues
Adrian Suirady,7,0,/adrian-suirady/,Rock,Rock; Gótico
Aerosmith,249,16.5,/aerosmith/,Rock,Rock; Hard Rock; Heavy Metal; Romântico; Pop/Rock; Classic Rock; Rock Alternativo; Blues; Metal; Chillout; Piano Rock; Funk; Gótico; Forró; Jovem Guarda; Hip Hop
Aliados,75,0.8,/aliados/,Rock,Rock; Pop/Rock; Rock Alternativo; Surf Music; Hardcore; Pop/Punk; Blues; R&B; Punk Rock; Axé
Alice Cooper,310,1.2,/alice-cooper/,Rock,Rock; Hard Rock; Heavy Metal; Punk Rock; Classic Rock; Grunge; Trilha Sonora; Gótico
Alter Bridge,74,1.4,/alter-bridge/,Rock,Rock; Hard Rock; Rock Alternativo; Heavy Metal; Grunge; Romântico; Rap; Metal; Hardcore
Amy Lee,33,0.5,/amy-lee/,Rock,Rock; Gótico; Hard Rock; Rock Alternativo; Heavy Metal; Piano Rock; Romântico; Metal; Indie; Classic Rock; New Age; Funk; Electronica; Industrial; Post-Rock; Psicodelia; Funk Carioca; Infantil; Pós-Punk; Dance; Pop; Clássico; Axé; Trilha Sonora
Anberlin,98,0.1,/anberlin/,Rock,Rock; Rock Alternativo; Hardcore; Emocore; Gospel/Religioso
Andi Deris,44,0,/andi-deris/,Rock,Rock; Hard Rock; Heavy Metal
Andrew W.K.,31,0,/andrew-w-k/,Rock,Rock
Andy (Brasil),7,0,/andy-brasil/,Rock,Rock
Angra,124,2.2,/angra/,Rock,Rock; Heavy Metal; Hard Rock; Progressivo; Metal; Black Music; Piano Rock; Post-Rock; Romântico; Psicodelia; Hardcore; Clássico; Forró; Pagode
Arthur Brown,2,0,/arthur-brown/,Rock,Rock
Asking Alexandria,77,1,/asking-alexandria/,Rock,Rock; Hard Rock; Hardcore; Heavy Metal; Emocore; Metal; Rock Alternativo; K-Pop/K-Rock; Classic Rock; Samba; Tecnopop; Grunge; Reggae; Chillout; World Music; Pop/Rock; Black Music; Gótico; Punk Rock; New Age
Autoramas,67,0.1,/autoramas/,Rock,Rock; Pop/Rock; Rock Alternativo; Progressivo; Indie; Punk Rock; Hardcore; Surf Music; Electronica; Funk; Pagode; Ska; R&B; Samba; New Age; MPB; Axé; Funk Carioca; Emocore; Grunge
Avante,21,0,/avante/,Rock,Rock

ALink,SName,SLink,Lyric,Idiom
/10000-maniacs/,More Than This,/10000-maniacs/more-than-this.html,I could feel at the time. There was no way of knowing. Fallen leaves in the night. Who can say where they're blowing. As free as the wind. Hopefully learning. Why the sea on the tide. Has no way of turning. More than this. You know there's nothing. More than this. Tell me one thing. More than this. You know there's nothing. It was fun for a while. There was no way of knowing. Like a dream in the night. Who can say where we're going. No care in the world. Maybe I'm learning. Why the sea on the tide. Has no way of turning. More than this. You know there's nothing. More than this. Tell me one thing. More than this. You know there's nothing. More than this. You know there's nothing. More than this. Tell me one thing. More than this. There's nothing.,ENGLISH
/10000-maniacs/,Because The Night,/10000-maniacs/because-the-night.html,"Take me now, baby, here as I am. Hold me close, and try and understand. Desire is hunger is the fire I breathe. Love is a banquet on which we feed. Come on now, try and understand. The way I feel under your command. Take my hand, as the sun descends. They can't hurt you now can't hurt you now, can't hurt you now. Because the night belongs to lovers. Because the night belongs to us. Because the night belongs to lovers. Cause the night belongs to us. Have I a doubt, baby, when I'm alone. Love is a ring a telephone. Love is an angel, disguised as lust. Here in our bed 'til the morning comes. Come on now, try and understand. The way I feel under your command. Take my hand, as the sun descends. They can't hurt you now, can't hurt you now, can't hurt you now. Because the night belongs to lovers. Because the night belongs to us. Because the night belongs to lovers. Because the night belongs to us. With love we sleep,. with doubt the vicious circle turns, and burns. Without you, oh I cannot live,. forgive the yearning burning. I believe it's time to heal to feel,. so take me now, take me now, take me now. Because the night belongs to lovers. Because the night belongs to us. Because the night belongs to lovers. Because the night belongs to us",ENGLISH
/10000-maniacs/,These Are Days,/10000-maniacs/these-are-days.html,"These are. These are days you'll remember. Never before and never since, I promise. Will the whole world be warm as this. And as you feel it,. You'll know it's true. That you - you are blessed and lucky. It's true - that you. Are touched by something. That will grow and bloom in you. These are days you'll remember. When May is rushing over you. With desire to be part of the miracles. You see in every hour. You'll know it's true. That you are blessed and lucky. It's true that you are touched. By something that will grow and bloom in you. These are days. These are the days you might fill. With laughter until you break. These days you might feel. A shaft of light. Make its way across your face. And when you do. You'll know how it was meant to be. See the signs and know their meaning. You'll know how it was meant to be. Hear the signs and know they're speaking. To you, to you",ENGLISH
/10000-maniacs/,A Campfire Song,/10000-maniacs/a-campfire-song.html,"A lie to say, ""O my mountain has coal veins and beds to dig.. 500 men with axes and they all dig for me.""A lie to ssay, ""O my. river where mant fish do swim, half of the catch is mine when you haul. your nets in.""Never will he believe that his greed is a blinding. ray. No devil or redeemer will cheat him. He'll take his gold to. where he's lying cold.. A lie to say, ""O my mine gave a diamond as big as a fist."". But with every gem in his pocket, the jewels he has missed. A lie to. say, ""O my garden is growing taller by the day.""He only eats the. best and tosses the rest away. Never will he be believe that his. greed is a blinding ray. No devil or redeemer can cheat him. he'll. take his gold to where he's lying cold. Six deep in the grave.. Something is out of reach. something he wanted. something is out of reach. he's being taunted. something is out of reach. that he can' beg or steal nor can he buy. his oldest pain. and fear in life. there'll not be time. his oldest pain. and fear in life. there'll not be time. A lie to say ""O my forest has trees that block the sun and. when I cut them down I don't answer to anyone.""No, no, never will he. believe that his greed is a blinding ray no devil or redeemer can. cheat. him. He'll take his gold where he's lying cold..",ENGLISH
/10000-maniacs/,Everyday Is Like Sunday,/10000-maniacs/everyday-is-like-sunday.html,"Trudging slowly over wet sand. Back to the bench where your clothes were stolen. This is a coastal town. That they forgot to close down. Armagedon - come armagedon come armagedon come. Everyday is like sunday. Everyday is silent and grey. Hide on a promanade. Etch on a post card:. How I dearly wish I was not here. In the seaside town. That they forgot to bomb. Come, come nuclear bomb!. Everyday is like sunday. Everyday is silent and grey. Trudging back over pebbles and sand. And a strange dust lands on your hands. (and on your face). Everyday is like sunday. Win yourself a cheap tray. Share some grease tea with me. Everyday is silent and grey",ENGLISH
/10000-maniacs/,Don't Talk,/10000-maniacs/dont-talk.html,"Don't talk, I will listen. Don't talk, you keep your distance. For I'd rather hear some truth tonight. Than entertain your lies,. So take you poison silently. Let me be let me close my eyes. Don't talk, I'll believe it. Don't talk, listen to me instead,. I know that if you think of it,. Both long enough and hard. The drink you drown your troubles. In is the trouble you're in now. Talk talk talk about it,. If you talk as if you care. But when your talk is over. Tilt that bottle in the air,. Tossing back more than your share. Don't talk, I can guess it. Don't talk, well now your restless. And you need somewhere to put the blame. For how you feel inside. You'll look for a close. And easy mark and you'll see me as fair game. Talk talk talk about it,. Talk as if you care. But when your talk is over tilt. That bottle in the air. Tossing back more than your share. You talk talk talk about it,. You talk as if you care. I'm marking every word. And can tell this time for sure,. Your talk is the finest I have heard. So don't talk, I'll be sleeping,. Let me go on dreaming. How your eyes they glow so fiercely. I can tell your inspired. By the name you just chose for me. Now what was it?. O, never mind it. We will talk talk. Talk about this when your head is clear. I'll discuss this in the morning,. But until then you may talk but I won't hear",ENGLISH
/10000-maniacs/,Across The Fields,/10000-maniacs/across-the-fields.html,"Well they left then in the morning, a hundred pairs of wings. In the light moved together in the colors of the morning. I looked to the clouds in the cirrus sky and they'd gone.. Across the marshes, across the fields below.. I fell through the vines and I hoped they would catch me below.. If only to take me with them there,. Tell me the part that shinesIn your heart on the wind.. And the reeds blew in the morning.. Take me along to the places. You've gone when my eyes looked away.. Tell me the song that you sing in the trees in the dawning.. Tell me the part that shines in your heart. And the rays of love forever,. Please take me there..",ENGLISH
/10000-maniacs/,Planned Obsolescence,/10000-maniacs/planned-obsolescence.html,[ music: Dennis Drew/lyric: Natalie Merchant ]. . science. is truth for life. watch religion fall obsolete. science. will be truth for life. technology as nature. science. truth for life. in fortran tongue the. answer. with wealth and prominence. man so near perfection. possession. it's an absence of interim. secure no demurrer. defense against divine. defense against his true. image. human conflict number five. discovery. dissolved all illusion. mystery. destroyed with conclusion. and illusion never restored. any modern man can see. that religion is. obsolete. piety. obsolete. ritual. obsolete. martyrdom. obsolete. prophetic vision. obsolete. mysticism. obsolete. commitment. obsolete. sacrament. obsolete. revelation. obsolete.,ENGLISH
/10000-maniacs/,Rainy Day,/10000-maniacs/rainy-day.html,"On bended kneeI've looked through every window then.. Touched the bottom, the night a sleepless day instead. A day when love came,came easy like what's lost now found.. Beneath a blinding light that would surround.. We were without, in doubt. We were about saving for a rainy day.. I crashed through mirrors,. I crashed through floors of laughter then.. In a blind scene, no ties would moor us to this room.. A day when love came, came easy like what's lost now found.. And you would save me, and I held you like you were my child.. If I were you, defiant you, alone upon a troubled way.. I would send my heart to you. To save it for a rainy day..",ENGLISH
/10000-maniacs/,Anthem For Doomed Youth,/10000-maniacs/anthem-for-doomed-youth.html,For whom do the bells toll. When sentenced to die. The stuttering rifles. Will stifle the cry. The monstrous anger. The fear's rapid rattle. A desert inferno. Kids dying like cattle. Don't tell me. We're not prepared. I've seen today's marine. He's eighteen and he's eager. He can be quite mean. No mock'ries for them. No prayers or bells. The demented choirs. The wailing of shells. The boys holding candles. On untraveled roads. The fear spreads like fire. As shrapnel explodes. I think it's wrong. To conscript our youth. Against their will. When plenty of our citizenry. Really like to kill. What sign posts will lead. To armageddon's fires. What bugles will call them. From crowded grey shires. The women sit quiet. With death on their minds. A slow dusk descending. The drawing of blinds. Make the hunters all line up. It's their idea of fun. And let those be forgiven. Who never owned a gun. Was it him or me. Or the wailing of the dead. The laughing soldiers. Cast their lots. And you can cut the dread.,ENGLISH
/10000-maniacs/,All That Never Happens,/10000-maniacs/all-that-never-happens.html,"She walks alone on the brick lane,. the breeze is blowing.. A year had changed her forever,. just like her grey home.. He used to live so close here,. we'd look for places I can't remember.. The world was safe when she knew him,. she tried to hold him, hold on forever.. For all that never happens and all that never will be,. a candle burning for the love we seldom keep.. The earth was raw in her fingers,. she overturned it.. Considered planting some flowers,. they wouldn't last long,. no one to tend them.. It's funny how these things go,. you were the answer to all the questions.. The memories made her weary,. she shuddered slowly,. she didn't want to.. As a distant summer he began to whisper,. and threw a smile her way.. She looked into the glass,. liquid surface showing that they were melding,. together present past.. So where can I go from here?. The color fading,. he didn't answer.. She felt him slip from her vision.. She tried to hold him, hold on forever.. So close forever,. in a silent frozen sleep..",ENGLISH
/10000-maniacs/,Back O'The Moon,/10000-maniacs/back-o-the-moon.html,Jenny. Jenny you don't know the nights I hide. below a second story room. to whistle you down. the man who's let to divvy up. time is a miser. he's got a silver coin. only lets it shine for hours. while you sleep it away. there's one rare and odd style of living. part only known to the everybody Jenny. a comical where's the end parade. of the sort people here would think unusual. Jenny. tonight upon the mock brine of a Luna Sea. far off we sail on to Back O'The Moon. Jenny. Jenny you don't know the days I've tried. telling backyard tales. so to maybe amuse. o your mood is never giddy. if you smile I'm delighted. but you'd rather pout. such a lazy child. you dare fold your arms. tisk and say that I lie. there's one rare and odd style of thinking. part only known to the everybody Jenny. the small step and giant leap takers. got the head start in the race toward it. Jenny. tonight upon the mock brine of a Luna Sea. far off we sail on to the Back O'The Moon. that was a sigh. but not meant to envy you. when your age was mine. some things were sworn true. morning would come. and calendar pages had. new printed seasons on. their opposite sides. Jenny. Jenny you don't know the nights I hide. below a second story room. to whistle you down. o the man who's let to divvy up. time is a miser. he's got a silver coin. lets it shine for hours. while you sleep it away. there's one rare and odd style of living. part only known to the everybody Jenny. out of tin ships jump the bubble head boys. to push their flags into powdered soils and cry. no second placers. no smart looking geese in bonnets. dance with pigs in high button trousers. no milk pail for the farmer's daughter. no merry towns of sweet walled houses. here I've found. Back O' the Moon. not here. I've found. Back O' the Moon,ENGLISH

importpandas as pd
from transformers importGPT2LMHeadModel, GPT2Tokenizer
importnumpy as np
importrandom
importtorch
from torch.utils.data importDataset, DataLoader
from transformers importGPT2Tokenizer, GPT2LMHeadModel, AdamW, get_linear_schedule_with_warmup
from tqdm importtqdm, trange
importtorch.nn.functional as F
importcsv

### Prepare data
lyrics = pd.read_csv('lyrics-data.csv')
lyrics = lyrics[lyrics['Idiom']=='ENGLISH']

#Only keep popular artists, with genre Rock/Pop and popularity high enough
artists = pd.read_csv('artists-data.csv')
artists = artists[(artists['Genre'].isin(['Rock'])) & (artists['Popularity']>5)]
df = lyrics.merge(artists[['Artist', 'Genre', 'Link']], left_on='ALink', right_on='Link', how='inner')
df = df.drop(columns=['ALink','SLink','Idiom','Link'])

#Drop the songs with lyrics too long (after more than 1024 tokens, does not work)
df = df[df['Lyric'].apply(lambda x: len(x.split(' ')) < 350)]

#Create a very small test set to compare generated text with the reality
test_set = df.sample(n = 200)
df = df.loc[~df.index.isin(test_set.index)]

#Reset the indexes
test_set =test_set.reset_index()
df =df.reset_index()

#For the test set only, keep last 20 words in a new column, then remove them from original column
test_set['True_end_lyrics'] = test_set['Lyric'].str.split().str[-20:].apply(' '.join)
test_set['Lyric'] = test_set['Lyric'].str.split().str[:-20].apply(' '.join)

classSongLyrics(Dataset):  
    def __init__(self, control_code, truncate=False, gpt2_type="gpt2", max_length=1024):

        self.tokenizer =GPT2Tokenizer.from_pretrained(gpt2_type)
        self.lyrics =[]

        for row in df['Lyric']:
          self.lyrics.append(torch.tensor(
                self.tokenizer.encode(f"<|{control_code}|>{row[:max_length]}<|endoftext|>")
            ))               
        iftruncate:
            self.lyrics = self.lyrics[:20000]
        self.lyrics_count =len(self.lyrics)
        
    def __len__(self):
        returnself.lyrics_count

    def __getitem__(self, item):
        returnself.lyrics[item]
    
dataset = SongLyrics(df['Lyric'], truncate=True, gpt2_type="gpt2")    

#Get the tokenizer and model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

#Accumulated batch size (since GPT2 is so big)
defpack_tensor(new_tensor, packed_tensor, max_seq_len):
    if packed_tensor isNone:
        returnnew_tensor, True, None
    if new_tensor.size()[1] + packed_tensor.size()[1] >max_seq_len:
        returnpacked_tensor, False, new_tensor
    else:
        packed_tensor = torch.cat([new_tensor, packed_tensor[:, 1:]], dim=1)
        return packed_tensor, True, None

deftrain(
    dataset, model, tokenizer,
    batch_size=16, epochs=5, lr=2e-5,
    max_seq_len=400, warmup_steps=200,
    gpt2_type="gpt2", output_dir=".", output_prefix="wreckgar",
    test_mode=False,save_model_on_epoch=False,
):
    acc_steps = 100device=torch.device("cuda")
    model =model.cuda()
    model.train()

    optimizer = AdamW(model.parameters(), lr=lr)
    scheduler =get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=warmup_steps, num_training_steps=-1)

    train_dataloader = DataLoader(dataset, batch_size=1, shuffle=True)
    loss=0
    accumulating_batch_count =0
    input_tensor =None

    for epoch inrange(epochs):

        print(f"Training epoch {epoch}")
        print(loss)
        for idx, entry intqdm(enumerate(train_dataloader)):
            (input_tensor, carry_on, remainder) = pack_tensor(entry, input_tensor, 768)

            if carry_on and idx != len(train_dataloader) - 1:
                continue
            input_tensor =input_tensor.to(device)
            outputs = model(input_tensor, labels=input_tensor)
            loss =outputs[0]
            loss.backward()

            if (accumulating_batch_count % batch_size) ==0:
                optimizer.step()
                scheduler.step()
                optimizer.zero_grad()
                model.zero_grad()

            accumulating_batch_count += 1input_tensor =None
        ifsave_model_on_epoch:
            torch.save(
                model.state_dict(),
                os.path.join(output_dir, f"{output_prefix}-{epoch}.pt"),
            )
    return model

model = train(dataset, model, tokenizer)

defgenerate(
    model,
    tokenizer,
    prompt,
    entry_count=10,
    entry_length=30, #maximum number of words
    top_p=0.8,
    temperature=1.,
):
    model.eval()
    generated_num =0
    generated_list =[]

    filter_value = -float("Inf")

    with torch.no_grad():

        for entry_idx intrange(entry_count):

            entry_finished =False
            generated =torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)

            for i inrange(entry_length):
                outputs = model(generated, labels=generated)
                loss, logits = outputs[:2]
                logits = logits[:, -1, :] / (temperature if temperature > 0 else 1.0)

                sorted_logits, sorted_indices = torch.sort(logits, descending=True)
                cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

                sorted_indices_to_remove = cumulative_probs >top_p
                sorted_indices_to_remove[..., 1:] =sorted_indices_to_remove[
                    ..., :-1].clone()
                sorted_indices_to_remove[..., 0] =0

                indices_to_remove =sorted_indices[sorted_indices_to_remove]
                logits[:, indices_to_remove] =filter_value

                next_token = torch.multinomial(F.softmax(logits, dim=-1), num_samples=1)
                generated = torch.cat((generated, next_token), dim=1)

                if next_token in tokenizer.encode("<|endoftext|>"):
                    entry_finished =True

                ifentry_finished:

                    generated_num = generated_num + 1
                    output_list =list(generated.squeeze().numpy())
                    output_text =tokenizer.decode(output_list)
                    generated_list.append(output_text)
                    break
            
            if notentry_finished:
              output_list =list(generated.squeeze().numpy())
              output_text = f"{tokenizer.decode(output_list)}<|endoftext|>"generated_list.append(output_text)
                
    returngenerated_list

#Function to generate multiple sentences. Test data should be a dataframe
deftext_generation(test_data):
  generated_lyrics =[]
  for i inrange(len(test_data)):
    x = generate(model.to('cpu'), tokenizer, test_data['Lyric'][i], entry_count=1)
    generated_lyrics.append(x)
  returngenerated_lyrics

#Run the functions to generate the lyrics
generated_lyrics = text_generation(test_set)

#Loop to keep only generated text and add it as a new column in the dataframe
my_generations=[]

for i inrange(len(generated_lyrics)):
  a = test_set['Lyric'][i].split()[-30:] #Get the matching string we want (30 words)
  b = ' '.join(a)
  c = ' '.join(generated_lyrics[i]) #Get all that comes after the matching string
  my_generations.append(c.split(b)[-1])

test_set['Generated_lyrics'] =my_generations


#Finish the sentences when there is a point, remove after that
final=[]

for i inrange(len(test_set)):
  to_remove = test_set['Generated_lyrics'][i].split('.')[-1]
  final.append(test_set['Generated_lyrics'][i].replace(to_remove,''))

test_set['Generated_lyrics'] = final

#Using BLEU score to compare the real sentences with the generated ones
importstatistics
from nltk.translate.bleu_score importsentence_bleu

scores=[]

for i inrange(len(test_set)):
  reference = test_set['True_end_lyrics'][i]
  candidate = test_set['Generated_lyrics'][i]
  scores.append(sentence_bleu(reference, candidate))

statistics.mean(scores)