基于循环神经网络的自然语言处理

### 基于循环神经网络的自然语言处理 #### 1. 实验结果初步展示 在进行MNIST数据提取和训练后，我们得到了如下结果： ```plaintext Extracting MNIST_data/train-images-idx3-ubyte.gz Extracting MNIST_data/train-labels-idx1-ubyte.gz Extracting MNIST_data/t10k-images-idx3-ubyte.gz Extracting MNIST_data/t10k-labels-idx1-ubyte.gz Epoch: 1,step:50, Minibatch Loss= 0.822081, Training Accuracy= 0.69531 Epoch: 1,step:100, Minibatch Loss= 0.760435, Training Accuracy= 0.75781 Epoch: 1,step:150, Minibatch Loss= 0.322639, Training Accuracy= 0.89844 Epoch: 1,step:200, Minibatch Loss= 0.408063, Training Accuracy= 0.85156 Epoch: 1,step:250, Minibatch Loss= 0.212591, Training Accuracy= 0.93750 Epoch: 1,step:300, Minibatch Loss= 0.158679, Training Accuracy= 0.94531 Epoch: 1,step:350, Minibatch Loss= 0.205918, Training Accuracy= 0.92969 Epoch: 1,step:400, Minibatch Loss= 0.131134, Training Accuracy= 0.95312 Epoch: 2,step:50, Minibatch Loss= 0.161183, Training Accuracy= 0.94531 Epoch: 2,step:100, Minibatch Loss= 0.237268, Training Accuracy= 0.91406 Epoch: 2,step:150, Minibatch Loss= 0.130443, Training Accuracy= 0.94531 Epoch: 2,step:200, Minibatch Loss= 0.133215, Training Accuracy= 0.93750 Epoch: 2,step:250, Minibatch Loss= 0.179435, Training Accuracy= 0.95312 Epoch: 2,step:300, Minibatch Loss= 0.108101, Training Accuracy= 0.97656 Epoch: 2,step:350, Minibatch Loss= 0.099574, Training Accuracy= 0.97656 Epoch: 2,step:400, Minibatch Loss= 0.074769, Training Accuracy= 0.98438 Optimization Finished! Testing Accuracy: 0.954102 ``` 从上述输出可以看出，仅仅运行2个周期，在测试数据集上就达到了95%的准确率。 #### 2. 基于LSTM的下一个单词预测和句子完成 我们使用来自《爱丽丝梦游仙境》的一小段文本进行模型训练，利用长短期记忆网络（LSTM）从给定词汇表中预测下一个单词。具体步骤如下： 1. **输入输出设定**：选取三个单词的序列作为输入，后续的一个单词作为输出。 2. **模型选择**：采用两层LSTM模型而非单层模型。 3. **数据处理**：从语料库中随机选择输入和输出集，并以大小为1的小批量进行输入。 以下是具体的代码实现： ```python # load the required libraries import numpy as np import tensorflow as tf from tensorflow.contrib import rnn import random import collections import time # Parameters learning_rate = 0.001 training_iters = 50000 display_step = 500 n_input = 3 # number of units in RNN cell n_hidden = 512 # Function to read and process the input file def read_data(fname): with open(fname) as f: data = f.readlines() data = [x.strip() for x in data] data = [data[i].lower().split() for i in range(len(data))] data = np.array(data) data = np.reshape(data, [-1, ]) return data # Function to build dictionary and reverse dictionary of words. def build_dataset(train_data): count = collections.Counter(train_data).most_common() dictionary = dict() for word, _ in count: dictionary[word] = len(dictionary) reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys())) return dictionary, reverse_dictionary # Function to one-hot the input vectors def input_one_hot(num): x = np.zeros(vocab_size) x[num] = 1 return x.tolist() # Read the input file and build the required dictionaries train_file = 'alice in wonderland.txt' train_data = read_data(train_file) dictionary, reverse_dictionary = build_dataset(train_data) vocab_size = len(dictionary) # Place holder for Mini-batch input output x = tf.placeholder("float", [None, n_input, vocab_size]) y = tf.placeholder("float", [None, vocab_size]) # RNN output node weights and biases weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocab_size])) } biases = { 'out': tf.Variable(tf.random_normal([vocab_size])) } # Forward pass for the recurrent neural network def RNN(x, weights, biases): x = tf.unstack(x, n_input, 1) # 2 layered LSTM Definition rnn_cell = rnn.MultiRNNCell([rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden)]) # generate prediction outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32) # there are n_input outputs but # we only want the last output return ```