{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2020 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "qFdPvlXBOdUN" }, "source": [ "# Introduction to Tensors" ] }, { "cell_type": "markdown", "metadata": { "id": "MfBg1C5NB3X0" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AL2hzxorJiWy" }, "outputs": [], "source": [ "import tensorflow as tf\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": { "id": "VQ3s2J8Vgowq" }, "source": [ "Tensors are multi-dimensional arrays with a uniform type (called a `dtype`). You can see all supported `dtypes` at `tf.dtypes`.\n", "\n", "If you're familiar with [NumPy](https://numpy.org/devdocs/user/quickstart.html), tensors are (kind of) like `np.arrays`.\n", "\n", "All tensors are immutable like Python numbers and strings: you can never update the contents of a tensor, only create a new one.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "DRK5-9EpYbzG" }, "source": [ "## Basics\n", "\n", "First, create some basic tensors." ] }, { "cell_type": "markdown", "metadata": { "id": "uSHRFT6LJbxq" }, "source": [ "Here is a \"scalar\" or \"rank-0\" tensor . A scalar contains a single value, and no \"axes\"." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "d5JcgLFR6gHv" }, "outputs": [], "source": [ "# This will be an int32 tensor by default; see \"dtypes\" below.\n", "rank_0_tensor = tf.constant(4)\n", "print(rank_0_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "tdmPAn9fWYs5" }, "source": [ "A \"vector\" or \"rank-1\" tensor is like a list of values. A vector has one axis:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "oZos8o_R6oE7" }, "outputs": [], "source": [ "# Let's make this a float tensor.\n", "rank_1_tensor = tf.constant([2.0, 3.0, 4.0])\n", "print(rank_1_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "G3IJG-ug_H4u" }, "source": [ "A \"matrix\" or \"rank-2\" tensor has two axes:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cnOIA_xb6u0M" }, "outputs": [], "source": [ "# If you want to be specific, you can set the dtype (see below) at creation time\n", "rank_2_tensor = tf.constant([[1, 2],\n", " [3, 4],\n", " [5, 6]], dtype=tf.float16)\n", "print(rank_2_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "19m72qEPkfxi" }, "source": [ "\n", "\n", " \n", " \n", " \n", "\n", "\n", " \n", "\n", " \n", " \n", "\n", "
A scalar, shape: []A vector, shape: [3]A matrix, shape: [3, 2]
\n", " \"A\n", " \n", " \"The\n", " \n", " \"A\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "fjFvzcn4_ehD" }, "source": [ "Tensors may have more axes; here is a tensor with three axes:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sesW7gw6JkXy" }, "outputs": [], "source": [ "# There can be an arbitrary number of\n", "# axes (sometimes called \"dimensions\")\n", "rank_3_tensor = tf.constant([\n", " [[0, 1, 2, 3, 4],\n", " [5, 6, 7, 8, 9]],\n", " [[10, 11, 12, 13, 14],\n", " [15, 16, 17, 18, 19]],\n", " [[20, 21, 22, 23, 24],\n", " [25, 26, 27, 28, 29]],])\n", "\n", "print(rank_3_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "rM2sTGIkoE3S" }, "source": [ "There are many ways you might visualize a tensor with more than two axes." ] }, { "cell_type": "markdown", "metadata": { "id": "NFiYfNMMhDgL" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "
A 3-axis tensor, shape: [3, 2, 5]
\n", " \n", " \n", " \n", " \n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "oWAc0U8OZwNb" }, "source": [ "You can convert a tensor to a NumPy array either using `np.array` or the `tensor.numpy` method:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "J5u6_6ZYaS7B" }, "outputs": [], "source": [ "np.array(rank_2_tensor)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "c6Taz2gIaZeo" }, "outputs": [], "source": [ "rank_2_tensor.numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "hnz19F0ocEKD" }, "source": [ "Tensors often contain floats and ints, but have many other types, including:\n", "\n", "* complex numbers\n", "* strings\n", "\n", "The base `tf.Tensor` class requires tensors to be \"rectangular\"---that is, along each axis, every element is the same size. However, there are specialized types of tensors that can handle different shapes:\n", "\n", "* Ragged tensors (see [RaggedTensor](#ragged_tensors) below)\n", "* Sparse tensors (see [SparseTensor](#sparse_tensors) below)" ] }, { "cell_type": "markdown", "metadata": { "id": "SDC7OGeAIJr8" }, "source": [ "You can do basic math on tensors, including addition, element-wise multiplication, and matrix multiplication." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-DTkjwDOIIDa" }, "outputs": [], "source": [ "a = tf.constant([[1, 2],\n", " [3, 4]])\n", "b = tf.constant([[1, 1],\n", " [1, 1]]) # Could have also said `tf.ones([2,2], dtype=tf.int32)`\n", "\n", "print(tf.add(a, b), \"\\n\")\n", "print(tf.multiply(a, b), \"\\n\")\n", "print(tf.matmul(a, b), \"\\n\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2smoWeUz-N2q" }, "outputs": [], "source": [ "print(a + b, \"\\n\") # element-wise addition\n", "print(a * b, \"\\n\") # element-wise multiplication\n", "print(a @ b, \"\\n\") # matrix multiplication" ] }, { "cell_type": "markdown", "metadata": { "id": "S3_vIAl2JPVc" }, "source": [ "Tensors are used in all kinds of operations (or \"Ops\")." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Gp4WUYzGIbnv" }, "outputs": [], "source": [ "c = tf.constant([[4.0, 5.0], [10.0, 1.0]])\n", "\n", "# Find the largest value\n", "print(tf.reduce_max(c))\n", "# Find the index of the largest value\n", "print(tf.math.argmax(c))\n", "# Compute the softmax\n", "print(tf.nn.softmax(c))" ] }, { "cell_type": "markdown", "metadata": { "id": "0MNM-q7-MZLz" }, "source": [ "Note: Typically, anywhere a TensorFlow function expects a `Tensor` as input, the function will also accept anything that can be converted to a `Tensor` using `tf.convert_to_tensor`. See below for an example." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_wch0N8xNEt-" }, "outputs": [], "source": [ "tf.convert_to_tensor([1,2,3])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ngqIeWYeNJVI" }, "outputs": [], "source": [ "tf.reduce_max([1,2,3])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ThVMxqbVNOq3" }, "outputs": [], "source": [ "tf.reduce_max(np.array([1,2,3]))" ] }, { "cell_type": "markdown", "metadata": { "id": "NvSAbowVVuRr" }, "source": [ "## About shapes" ] }, { "cell_type": "markdown", "metadata": { "id": "hkaBIqkTCcGY" }, "source": [ "Tensors have shapes. Some vocabulary:\n", "\n", "* **Shape**: The length (number of elements) of each of the axes of a tensor.\n", "* **Rank**: Number of tensor axes. A scalar has rank 0, a vector has rank 1, a matrix is rank 2.\n", "* **Axis** or **Dimension**: A particular dimension of a tensor.\n", "* **Size**: The total number of items in the tensor, the product of the shape vector's elements.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "E9L3-kCQq2f6" }, "source": [ "Note: Although you may see reference to a \"tensor of two dimensions\", a rank-2 tensor does not usually describe a 2D space." ] }, { "cell_type": "markdown", "metadata": { "id": "VFOyG2tn8LhW" }, "source": [ "Tensors and `tf.TensorShape` objects have convenient properties for accessing these:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "RyD3yewUKdnK" }, "outputs": [], "source": [ "rank_4_tensor = tf.zeros([3, 2, 4, 5])" ] }, { "cell_type": "markdown", "metadata": { "id": "oTZZW9ziq4og" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", " \n", "
A rank-4 tensor, shape: [3, 2, 4, 5]
\n", "\"A\n", " \n", "\"A\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "MHm9vSqogsBk" }, "outputs": [], "source": [ "print(\"Type of every element:\", rank_4_tensor.dtype)\n", "print(\"Number of axes:\", rank_4_tensor.ndim)\n", "print(\"Shape of tensor:\", rank_4_tensor.shape)\n", "print(\"Elements along axis 0 of tensor:\", rank_4_tensor.shape[0])\n", "print(\"Elements along the last axis of tensor:\", rank_4_tensor.shape[-1])\n", "print(\"Total number of elements (3*2*4*5): \", tf.size(rank_4_tensor).numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "2ZGZp_JOOPOv" }, "source": [ "But note that the `Tensor.ndim` and `Tensor.shape` attributes don't return `Tensor` objects. If you need a `Tensor` use the `tf.rank` or `tf.shape` function. This difference is subtle, but it can be important when building graphs (later)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ptq0-y6APCpD" }, "outputs": [], "source": [ "tf.rank(rank_4_tensor)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HslrDOEBPICN" }, "outputs": [], "source": [ "tf.shape(rank_4_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "bQmE_Vx5JilS" }, "source": [ "While axes are often referred to by their indices, you should always keep track of the meaning of each. Often axes are ordered from global to local: The batch axis first, followed by spatial dimensions, and features for each location last. This way feature vectors are contiguous regions of memory.\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "\n", "
Typical axis order
\n", "\"Keep\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "FlPoVvJS75Bb" }, "source": [ "## Indexing" ] }, { "cell_type": "markdown", "metadata": { "id": "apOkCKqCZIZu" }, "source": [ "### Single-axis indexing\n", "\n", "TensorFlow follows standard Python indexing rules, similar to [indexing a list or a string in Python](https://docs.python.org/3/tutorial/introduction.html#strings), and the basic rules for NumPy indexing.\n", "\n", "* indexes start at `0`\n", "* negative indices count backwards from the end\n", "* colons, `:`, are used for slices: `start:stop:step`\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SQ-CrJxLXTIM" }, "outputs": [], "source": [ "rank_1_tensor = tf.constant([0, 1, 1, 2, 3, 5, 8, 13, 21, 34])\n", "print(rank_1_tensor.numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "mQYYL56PXSak" }, "source": [ "Indexing with a scalar removes the axis:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "n6tqHciOWMt5" }, "outputs": [], "source": [ "print(\"First:\", rank_1_tensor[0].numpy())\n", "print(\"Second:\", rank_1_tensor[1].numpy())\n", "print(\"Last:\", rank_1_tensor[-1].numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "qJLHU_a2XwpG" }, "source": [ "Indexing with a `:` slice keeps the axis:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "giVPPcfQX-cu" }, "outputs": [], "source": [ "print(\"Everything:\", rank_1_tensor[:].numpy())\n", "print(\"Before 4:\", rank_1_tensor[:4].numpy())\n", "print(\"From 4 to the end:\", rank_1_tensor[4:].numpy())\n", "print(\"From 2, before 7:\", rank_1_tensor[2:7].numpy())\n", "print(\"Every other item:\", rank_1_tensor[::2].numpy())\n", "print(\"Reversed:\", rank_1_tensor[::-1].numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "elDSxXi7X-Bh" }, "source": [ "### Multi-axis indexing" ] }, { "cell_type": "markdown", "metadata": { "id": "Cgk0uRUYZiai" }, "source": [ "Higher rank tensors are indexed by passing multiple indices.\n", "\n", "The exact same rules as in the single-axis case apply to each axis independently." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Tc5X_WlsZXmd" }, "outputs": [], "source": [ "print(rank_2_tensor.numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "w07U9vq5ipQk" }, "source": [ "Passing an integer for each index, the result is a scalar." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PvILXc1PjqTM" }, "outputs": [], "source": [ "# Pull out a single value from a 2-rank tensor\n", "print(rank_2_tensor[1, 1].numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "3RLCzAOHjfEH" }, "source": [ "You can index using any combination of integers and slices:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "YTqNqsfJkJP_" }, "outputs": [], "source": [ "# Get row and column tensors\n", "print(\"Second row:\", rank_2_tensor[1, :].numpy())\n", "print(\"Second column:\", rank_2_tensor[:, 1].numpy())\n", "print(\"Last row:\", rank_2_tensor[-1, :].numpy())\n", "print(\"First item in last column:\", rank_2_tensor[0, -1].numpy())\n", "print(\"Skip the first row:\")\n", "print(rank_2_tensor[1:, :].numpy(), \"\\n\")" ] }, { "cell_type": "markdown", "metadata": { "id": "P45TwSUVSK6G" }, "source": [ "Here is an example with a 3-axis tensor:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GuLoMoCVSLxK" }, "outputs": [], "source": [ "print(rank_3_tensor[:, :, 4])" ] }, { "cell_type": "markdown", "metadata": { "id": "9NgmHq27TJOE" }, "source": [ "\n", "\n", "\n", "\n", "\n", " \n", " \n", "\n", "
Selecting the last feature across all locations in each example in the batch
\n", "\"A\n", " \n", "\"The\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "t9V83-thHn89" }, "source": [ "Read the [tensor slicing guide](https://tensorflow.org/guide/tensor_slicing) to learn how you can apply indexing to manipulate individual elements in your tensors." ] }, { "cell_type": "markdown", "metadata": { "id": "fpr7R0t4SVb0" }, "source": [ "## Manipulating Shapes\n", "\n", "Reshaping a tensor is of great utility. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EMeTtga5Wq8j" }, "outputs": [], "source": [ "# Shape returns a `TensorShape` object that shows the size along each axis\n", "x = tf.constant([[1], [2], [3]])\n", "print(x.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "38jc2RXziT3W" }, "outputs": [], "source": [ "# You can convert this object into a Python list, too\n", "print(x.shape.as_list())" ] }, { "cell_type": "markdown", "metadata": { "id": "J_xRlHZMKYnF" }, "source": [ "You can reshape a tensor into a new shape. The `tf.reshape` operation is fast and cheap as the underlying data does not need to be duplicated." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pa9JCgMLWy87" }, "outputs": [], "source": [ "# You can reshape a tensor to a new shape.\n", "# Note that you're passing in a list\n", "reshaped = tf.reshape(x, [1, 3])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Mcq7iXOkW3LK" }, "outputs": [], "source": [ "print(x.shape)\n", "print(reshaped.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "gIB2tOkoVr6E" }, "source": [ "The data maintains its layout in memory and a new tensor is created, with the requested shape, pointing to the same data. TensorFlow uses C-style \"row-major\" memory ordering, where incrementing the rightmost index corresponds to a single step in memory." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7kMfM0RpUgI8" }, "outputs": [], "source": [ "print(rank_3_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "TcDtfQkJWzIx" }, "source": [ "If you flatten a tensor you can see what order it is laid out in memory." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "COnHEPuaWDQp" }, "outputs": [], "source": [ "# A `-1` passed in the `shape` argument says \"Whatever fits\".\n", "print(tf.reshape(rank_3_tensor, [-1]))" ] }, { "cell_type": "markdown", "metadata": { "id": "jJZRira2W--c" }, "source": [ "Typically the only reasonable use of `tf.reshape` is to combine or split adjacent axes (or add/remove `1`s).\n", "\n", "For this 3x2x5 tensor, reshaping to (3x2)x5 or 3x(2x5) are both reasonable things to do, as the slices do not mix:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zP2Iqc7zWu_J" }, "outputs": [], "source": [ "print(tf.reshape(rank_3_tensor, [3*2, 5]), \"\\n\")\n", "print(tf.reshape(rank_3_tensor, [3, -1]))" ] }, { "cell_type": "markdown", "metadata": { "id": "6ZsZRUhihlDB" }, "source": [ "\n", "\n", "\n", " \n", " \n", " \n", "\n", "
\n", "Some good reshapes.\n", "
\n", "\"A\n", " \n", " \"The\n", " \n", "\"The\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "nOcRxDC3jNIU" }, "source": [ "Reshaping will \"work\" for any new shape with the same total number of elements, but it will not do anything useful if you do not respect the order of the axes.\n", "\n", "Swapping axes in `tf.reshape` does not work; you need `tf.transpose` for that. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "I9qDL_8u7cBH" }, "outputs": [], "source": [ "# Bad examples: don't do this\n", "\n", "# You can't reorder axes with reshape.\n", "print(tf.reshape(rank_3_tensor, [2, 3, 5]), \"\\n\") \n", "\n", "# This is a mess\n", "print(tf.reshape(rank_3_tensor, [5, 6]), \"\\n\")\n", "\n", "# This doesn't work at all\n", "try:\n", " tf.reshape(rank_3_tensor, [7, -1])\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "qTM9-5eh68oo" }, "source": [ "\n", "\n", "\n", " \n", " \n", " \n", "\n", "
\n", "Some bad reshapes.\n", "
\n", "\"You\n", " \n", "\"Anything\n", " \n", "\"The\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "N9r90BvHCbTt" }, "source": [ "You may run across not-fully-specified shapes. Either the shape contains a `None` (an axis-length is unknown) or the whole shape is `None` (the rank of the tensor is unknown).\n", "\n", "Except for [tf.RaggedTensor](#ragged_tensors), such shapes will only occur in the context of TensorFlow's symbolic, graph-building APIs:\n", "\n", "* [tf.function](function.ipynb) \n", "* The [keras functional API](https://www.tensorflow.org/guide/keras/functional).\n" ] }, { "cell_type": "markdown", "metadata": { "id": "fDmFtFM7k0R2" }, "source": [ "## More on `DTypes`\n", "\n", "To inspect a `tf.Tensor`'s data type use the `Tensor.dtype` property.\n", "\n", "When creating a `tf.Tensor` from a Python object you may optionally specify the datatype.\n", "\n", "If you don't, TensorFlow chooses a datatype that can represent your data. TensorFlow converts Python integers to `tf.int32` and Python floating point numbers to `tf.float32`. Otherwise TensorFlow uses the same rules NumPy uses when converting to arrays.\n", "\n", "You can cast from type to type." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5mSTDWbelUvu" }, "outputs": [], "source": [ "the_f64_tensor = tf.constant([2.2, 3.3, 4.4], dtype=tf.float64)\n", "the_f16_tensor = tf.cast(the_f64_tensor, dtype=tf.float16)\n", "# Now, cast to an uint8 and lose the decimal precision\n", "the_u8_tensor = tf.cast(the_f16_tensor, dtype=tf.uint8)\n", "print(the_u8_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "s1yBlJsVlFSu" }, "source": [ "## Broadcasting\n", "\n", "Broadcasting is a concept borrowed from the [equivalent feature in NumPy](https://numpy.org/doc/stable/user/basics.broadcasting.html). In short, under certain conditions, smaller tensors are \"stretched\" automatically to fit larger tensors when running combined operations on them.\n", "\n", "The simplest and most common case is when you attempt to multiply or add a tensor to a scalar. In that case, the scalar is broadcast to be the same shape as the other argument. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "P8sypqmagHQN" }, "outputs": [], "source": [ "x = tf.constant([1, 2, 3])\n", "\n", "y = tf.constant(2)\n", "z = tf.constant([2, 2, 2])\n", "# All of these are the same computation\n", "print(tf.multiply(x, 2))\n", "print(x * y)\n", "print(x * z)" ] }, { "cell_type": "markdown", "metadata": { "id": "o0SBoR6voWcb" }, "source": [ "Likewise, axes with length 1 can be stretched out to match the other arguments. Both arguments can be stretched in the same computation.\n", "\n", "In this case a 3x1 matrix is element-wise multiplied by a 1x4 matrix to produce a 3x4 matrix. Note how the leading 1 is optional: The shape of y is `[4]`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6sGmkPg3XANr" }, "outputs": [], "source": [ "# These are the same computations\n", "x = tf.reshape(x,[3,1])\n", "y = tf.range(1, 5)\n", "print(x, \"\\n\")\n", "print(y, \"\\n\")\n", "print(tf.multiply(x, y))" ] }, { "cell_type": "markdown", "metadata": { "id": "t_7sh-EUYLrE" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
A broadcasted add: a [3, 1] times a [1, 4] gives a [3,4]
\n", "\"Adding\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "9V3KgSJcKDRz" }, "source": [ "Here is the same operation without broadcasting:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "elrF6v63igY8" }, "outputs": [], "source": [ "x_stretch = tf.constant([[1, 1, 1, 1],\n", " [2, 2, 2, 2],\n", " [3, 3, 3, 3]])\n", "\n", "y_stretch = tf.constant([[1, 2, 3, 4],\n", " [1, 2, 3, 4],\n", " [1, 2, 3, 4]])\n", "\n", "print(x_stretch * y_stretch) # Again, operator overloading" ] }, { "cell_type": "markdown", "metadata": { "id": "14KobqYu85gi" }, "source": [ "Most of the time, broadcasting is both time and space efficient, as the broadcast operation never materializes the expanded tensors in memory. \n", "\n", "You see what broadcasting looks like using `tf.broadcast_to`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GW2Q59_r8hZ6" }, "outputs": [], "source": [ "print(tf.broadcast_to(tf.constant([1, 2, 3]), [3, 3]))" ] }, { "cell_type": "markdown", "metadata": { "id": "Z2bAMMQY-jpP" }, "source": [ "Unlike a mathematical op, for example, `broadcast_to` does nothing special to save memory. Here, you are materializing the tensor.\n", "\n", "It can get even more complicated. [This section](https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-on-arrays-broadcasting.html) of Jake VanderPlas's book _Python Data Science Handbook_ shows more broadcasting tricks (again in NumPy)." ] }, { "cell_type": "markdown", "metadata": { "id": "o4Rpz0xAsKSI" }, "source": [ "## tf.convert_to_tensor\n", "\n", "Most ops, like `tf.matmul` and `tf.reshape` take arguments of class `tf.Tensor`. However, you'll notice in the above case, Python objects shaped like tensors are accepted.\n", "\n", "Most, but not all, ops call `convert_to_tensor` on non-tensor arguments. There is a registry of conversions, and most object classes like NumPy's `ndarray`, `TensorShape`, Python lists, and `tf.Variable` will all convert automatically.\n", "\n", "See `tf.register_tensor_conversion_function` for more details, and if you have your own type you'd like to automatically convert to a tensor." ] }, { "cell_type": "markdown", "metadata": { "id": "05bBVBVYV0y6" }, "source": [ "## Ragged Tensors\n", "\n", "A tensor with variable numbers of elements along some axis is called \"ragged\". Use `tf.ragged.RaggedTensor` for ragged data.\n", "\n", "For example, This cannot be represented as a regular tensor:" ] }, { "cell_type": "markdown", "metadata": { "id": "VPc3jGoeJqB7" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
A `tf.RaggedTensor`, shape: [4, None]
\n", "\"A\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VsbTjoFfNVBF" }, "outputs": [], "source": [ "ragged_list = [\n", " [0, 1, 2, 3],\n", " [4, 5],\n", " [6, 7, 8],\n", " [9]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "p4xKTo57tutG" }, "outputs": [], "source": [ "try:\n", " tensor = tf.constant(ragged_list)\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "0cm9KuEeMLGI" }, "source": [ "Instead create a `tf.RaggedTensor` using `tf.ragged.constant`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XhF3QV3jiqTj" }, "outputs": [], "source": [ "ragged_tensor = tf.ragged.constant(ragged_list)\n", "print(ragged_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "sFgHduHVNoIE" }, "source": [ "The shape of a `tf.RaggedTensor` will contain some axes with unknown lengths:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Eo_3wJUWNgqB" }, "outputs": [], "source": [ "print(ragged_tensor.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "V9njclVkkN7G" }, "source": [ "## String tensors\n", "\n", "`tf.string` is a `dtype`, which is to say you can represent data as strings (variable-length byte arrays) in tensors.\n", "\n", "The strings are atomic and cannot be indexed the way Python strings are. The length of the string is not one of the axes of the tensor. See `tf.strings` for functions to manipulate them." ] }, { "cell_type": "markdown", "metadata": { "id": "5P_8spEGQ0wp" }, "source": [ "Here is a scalar string tensor:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sBosmM8MkIh4" }, "outputs": [], "source": [ "# Tensors can be strings, too here is a scalar string.\n", "scalar_string_tensor = tf.constant(\"Gray wolf\")\n", "print(scalar_string_tensor)" ] }, { "cell_type": "markdown", "metadata": { "id": "CMFBSl1FQ3vE" }, "source": [ "And a vector of strings:" ] }, { "cell_type": "markdown", "metadata": { "id": "IO-c3Tq3RC1L" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
A vector of strings, shape: [3,]
\n", "\"The\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "41Dv2kL9QrtO" }, "outputs": [], "source": [ "# If you have three string tensors of different lengths, this is OK.\n", "tensor_of_strings = tf.constant([\"Gray wolf\",\n", " \"Quick brown fox\",\n", " \"Lazy dog\"])\n", "# Note that the shape is (3,). The string length is not included.\n", "print(tensor_of_strings)" ] }, { "cell_type": "markdown", "metadata": { "id": "76gQ9qrgSMzS" }, "source": [ "In the above printout the `b` prefix indicates that `tf.string` dtype is not a unicode string, but a byte-string. See the [Unicode Tutorial](https://www.tensorflow.org/tutorials/load_data/unicode) for more about working with unicode text in TensorFlow." ] }, { "cell_type": "markdown", "metadata": { "id": "ClSBPK-lZBQp" }, "source": [ "If you pass unicode characters they are utf-8 encoded." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GTgL53jxSMd9" }, "outputs": [], "source": [ "tf.constant(\"🥳👍\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Ir9cY42MMAei" }, "source": [ "Some basic functions with strings can be found in `tf.strings`, including `tf.strings.split`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8k2K0VTFyj8e" }, "outputs": [], "source": [ "# You can use split to split a string into a set of tensors\n", "print(tf.strings.split(scalar_string_tensor, sep=\" \"))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zgGAn1dfR-04" }, "outputs": [], "source": [ "# ...but it turns into a `RaggedTensor` if you split up a tensor of strings,\n", "# as each string might be split into a different number of parts.\n", "print(tf.strings.split(tensor_of_strings))" ] }, { "cell_type": "markdown", "metadata": { "id": "HsAn1kPeO84m" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
Three strings split, shape: [3, None]
\n", "\"Splitting\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "st9OxrUxWSKY" }, "source": [ "And `tf.strings.to_number`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3nRtx3X9WRfN" }, "outputs": [], "source": [ "text = tf.constant(\"1 10 100\")\n", "print(tf.strings.to_number(tf.strings.split(text, \" \")))" ] }, { "cell_type": "markdown", "metadata": { "id": "r2EZtBbJBns4" }, "source": [ "Although you can't use `tf.cast` to turn a string tensor into numbers, you can convert it into bytes, and then into numbers." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fo8BjmH7gyTj" }, "outputs": [], "source": [ "byte_strings = tf.strings.bytes_split(tf.constant(\"Duck\"))\n", "byte_ints = tf.io.decode_raw(tf.constant(\"Duck\"), tf.uint8)\n", "print(\"Byte strings:\", byte_strings)\n", "print(\"Bytes:\", byte_ints)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uSQnZ7d1jCSQ" }, "outputs": [], "source": [ "# Or split it up as unicode and then decode it\n", "unicode_bytes = tf.constant(\"アヒル 🦆\")\n", "unicode_char_bytes = tf.strings.unicode_split(unicode_bytes, \"UTF-8\")\n", "unicode_values = tf.strings.unicode_decode(unicode_bytes, \"UTF-8\")\n", "\n", "print(\"\\nUnicode bytes:\", unicode_bytes)\n", "print(\"\\nUnicode chars:\", unicode_char_bytes)\n", "print(\"\\nUnicode values:\", unicode_values)" ] }, { "cell_type": "markdown", "metadata": { "id": "fE7nKJ2YW3aY" }, "source": [ "The `tf.string` dtype is used for all raw bytes data in TensorFlow. The `tf.io` module contains functions for converting data to and from bytes, including decoding images and parsing csv." ] }, { "cell_type": "markdown", "metadata": { "id": "ua8BnAzxkRKV" }, "source": [ "## Sparse tensors\n", "\n", "Sometimes, your data is sparse, like a very wide embedding space. TensorFlow supports `tf.sparse.SparseTensor` and related operations to store sparse data efficiently." ] }, { "cell_type": "markdown", "metadata": { "id": "mS5zgqgUTPRb" }, "source": [ "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
A `tf.SparseTensor`, shape: [3, 4]
\n", "\"An\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "B9nbO1E2kSUN" }, "outputs": [], "source": [ "# Sparse tensors store values by index in a memory-efficient manner\n", "sparse_tensor = tf.sparse.SparseTensor(indices=[[0, 0], [1, 2]],\n", " values=[1, 2],\n", " dense_shape=[3, 4])\n", "print(sparse_tensor, \"\\n\")\n", "\n", "# You can convert sparse tensors to dense\n", "print(tf.sparse.to_dense(sparse_tensor))" ] } ], "metadata": { "colab": { "collapsed_sections": [ "Tce3stUlHN0L" ], "name": "tensor.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }