Advanced NumPy¶
As described in detail in a previous lesson, NumPy is built around a new, n-dimensional array (ndarray) data structure that provides fast support for numerical computations. In this notebook, we introduce multi-dimensional NumPy arrays, and focus on extending concepts introduced to deal with one-dimensional arrays to higher dimensional arrays. Note that for simplicity we will rarely discuss anything other than two- or three-dimensional arrays.
Creating Multi-Dimensional Arrays¶
Higher dimensional arrays can be created in same way that a one-dimensional array is created (including all of the previously listed convenience functions, where a one-dimensional array with the correct number of elements is created, and subsequently reshaped into the correct dimensionality. For example, you can create a NumPy array with one hundred elements and reshape this new array into a ten by ten matrix.
import numpy as np
# Make a one hundred element one-dimensional array
data = np.arange(100)
print(data)
print()
# Reshape to a 10 x 10 array
mat = data.reshape(10, 10)
print(mat)
Special convenience functions also exist to create special multidimensional arrays. For example, you can create an identity matrix, where the diagonal elements are all one and all other elements are zero by using the eye method (since the Identity matrix is often denoted by the capital letter 'I'). You can also specify the diagonal (or certain off-diagonal) elements by using the diag function, where the input array is assigned to a set of diagonal elements in the new array. If the k parameter to the np.diag method is zero, the diagonal elements will be initialized. If the k parameter is a positive (negative) integer, the diagonal corresponding to the integer value of k is initialized. The size of the resulting array will be the minimum possible size to allow the input array to be properly initialized.
# Create special two-dimensional arrays
print("Matrix will be 4 x 4.\n", np.eye(4))
print("\nMatrix will be 4 x 4.\n", np.diag(np.arange(4), 0))
print("\nMatrix will be 5 x 5.\n", np.diag(np.arange(4), 1))
print("\nMatrix will be 5 x 5.\n", np.diag(np.arange(4), -1))
Slicing Multidimensional Arrays¶
Multi-dimensional arrays can be sliced (or indexed); the only trick is to remember the proper ordering for the elements. Each dimension is differentiated by a comma in the slicing operation, so a two-dimensional array is sliced with [start1:end1, start2:end2], while a three-dimensional array is sliced with [start1:end1, start2:end2. start3:end3], continuing on with higher dimensions. If only one dimension is specified, it will default to the first dimension. These concepts are demonstrated in the following two code cells, first for a two-dimensional array, followed by a three-dimensional array.
Note that each of these slicing operations (i.e., start:end) can also include an optional stride value as well.
# Two-dimensional slicing
b = np.arange(9).reshape((3,3))
print("3 x 3 array = \n", b)
print("\nSlice in first dimension (row 1): ", b[0])
print("\nSlice in first dimension (row 3): ", b[2])
print("\nSlice in second dimension (col 1): ", b[:,0])
print("\nSlice in second dimension (col 3): ", b[:,2])
print("\nSlice in first and second dimension: ", b[0:1, 1:2])
print("\nDirect Element access: ", b[0,1])
# Three-dimensional slicing
c = np.arange(27).reshape((3,3, 3))
print("3 x 3 x 3 array = \n", c)
print("\nSlice in first dimension (first x axis slice):\n", c[0])
print("\nSlice in first and second dimension: ", c[0, 1])
print("\nSlice in first dimension (third x axis slice):\n", c[2])
print("\nSlice in second dimension (first y axis slice):\n", c[:,0])
print("\nSlice in second dimension (third y axis slice):\n", c[:,2])
print("\nSlice in first and second dimension: ", c[0:1, 1:2])
print("\nSlice in first and second dimension:\n", c[0,1])
print("\nSlice in first and third dimension: ", c[0,:,1])
print("\nSlice in first, second, and third dimension: ", c[0:1,1:2,2:])
print("\nDirect element access: ", c[0,1, 2])
Student Exercise
In the empty Code cell below, write and execute code to make a four by four matrix that contains the integers 0 through 15. Slice and display the third row, the third column, and the element at the position (4, 4).
Special Indexing¶
NumPy also provides several other special indexing techniques. The first such technique is the use of an index array, where you use an array to specify the elements to be selected. The second technique is a Boolean mask array. In this case, the Boolean array is the same size as the primary NumPy array, and if the element in the mask array is True, the corresponding element in the primary array is selected, and vice-versa for a False mask array element. These two special indexing techniques are demonstrated in the following two code cells.
# Demonstration of a multi-dimensional index array
# Two-dimensional array
c = np.arange(10).reshape((2, 5))
print("\nStarting array:\n", c)
print("\nIndex Array access: \n", c[np.array([0, 1]) , np.array([3, 4])])
# Demonstrate Boolean mask access
# Two-dimensional example.
print("\n--------------------")
c = np.arange(25).reshape((5, 5))
print("\n Starting Array: \n", c)
# Build a mask that is True for all even elements with value greater than four
mask1 = (c > 4)
mask2 = (c % 2 == 0)
print("\nMask 1:\n", mask1)
print("\nMask 2:\n", mask2)
# We use the logical_and ufunc here, but it is described later
mask = np.logical_and(mask1, mask2)
print("\nMask :\n", mask)
print("\nMasked Array :\n", c[mask])
c[mask] *= -2
print("\nNew Array :\n", c)
Basic Operations¶
NumPy arrays naturally support basic mathematical operations, including addition, subtraction, multiplication, division, integer division, and remainder, allowing you to easily combine a scalar (a single number) with a vector (a one-dimensional array), or a matrix (a multi-dimensional array). In the next Code cell, we create a two-dimensional array and use that array to demonstrate how to operate on a scalar and a matrix.
# Create a two-dimensional array
b = np.arange(9).reshape((3,3))
print("Matrix = \n", b)
print("\nMatrix + 10.5 =\n", (b + 10.5))
print("\nMatrix * 0.25 =\n", (b * 0.25))
print("\nMatrix % 2 =\n", (b % 2))
print("\nMatrix / 3.0 =\n", ((b - 4.0) / 3.))
We also can combine arrays as long as they have the same dimensionality. In the next Code cell, we create a one-dimensional and a two-dimensional array and demonstrate how these two arrays can be combined.
# Create two arrays
a = np.arange(1, 10)
b = (10. - a).reshape((3, 3))
print(f"Array = \n {a}")
print(f"\nMatrix = \n {b}")
print("\nArray[0:3] + Matrix Row 1 = ",a[:3] + b[0,:,])
print("\nArray[0:3] + Matrix[:0] = ", a[:3] + b[:,0])
print("\nArray[3:6] + Matrix[0:] = ", a[3:6] + b[0, :])
# Now combine scalar operations
print("\n3.0 * Array[3:6] + (10.5 + Matrix[0:]) = ", 3.0 * a[3:6] + (10.5 + b[0, :]))
Summary Functions¶
NumPy provides convenience functions that can quickly summarize the values of an array, or the values along a specific axis of a multi-dimensional array. These functions include basic statistical measures (mean, median, var, std, min, and max), the total sum or product of all elements in the array (sum, prod), as well as running sums or products for all elements in the array (cumsum, cumprod). The last two functions actually produce arrays that are of the same size as the input array, where each element is replaced by the respective running sum/product up to and including the current element. Another function, trace, calculates the trace of an array, which simply sums up the diagonal elements in the multi-dimensional array.
# Demonstrate data processing convenience functions
# Make an array = [1, 2, 3, 4, 5]
a = np.arange(1, 10).reshape(3,3)
print(f'Array = \n{a}\n')
print("Mean value (all) = {}".format(np.mean(a)))
print("Mean value (columns) = {}".format(np.mean(a, axis = 0)))
print("Mean value (rows) = {}\n".format(np.mean(a, axis = 1)))
print("Std. Deviation (columns) = {}".format(np.std(a, axis = 0)))
print("Std. Deviation (rows) = {}\n".format(np.std(a, axis = 1)))
print("Minimum value (columns) = {}".format(np.min(a, axis = 0)))
print("Maximum value (columns) = {}\n".format(np.max(a, axis = 0)))
# Now compute trace of 5 x 5 diagonal matrix (= 5)
print('Trace of 5 x 5 identity matrix = ', np.trace(np.eye(5)))
Universal Functions¶
NumPy also includes methods that are universal functions, or ufuncs that are vectorized and thus operate on each element in the array, without the need for a loop. This topic was discussed previously in considerable detail in the Introduction to NumPy lesson; here we simply demonstrate the basic ideas with multi-dimensional arrays.
You have already seen examples of some of these functions at the start of this IPython Notebook when we compared the speed and simplicity of NumPy versus normal Python for numerical operations. These functions almost all include an optional out parameter that allows a pre-defined NumPy array to be used to hold the results of the calculation, which can often speed up the processing by eliminating the need for the creation and destruction of temporary arrays. These functions will all still return the final array, even if the out parameter is used.
For complete details, look at the official NumPy ufunc reference guide for more information on any of these functions, for example, the isnan function, since the user guide has a full breakdown of each function and sample code demonstrating how to use the function.
In the following Code cells, we demonstrate several of these ufuncs on multi-dimensional arrays.
# Make a 3 x 3 two-dimensional array
b = np.arange(1, 10).reshape(3, 3)
print('original array:\n', b)
# Peeform ufunc math
c = np.sin(b)
print('\nnp.sin : \n', c)
print('\nnp.log and np.abs : \n', np.log10(np.abs(c)))
print('\nnp.mod : \n', np.mod(b, 2))
print('\nnp.logical_and : \n', np.logical_and(np.mod(b, 2), True))
# Demonstrate Boolean tests with operators
d = np.arange(9).reshape(3, 3)
print("Greater Than or Equal Test: \n", d >= 5)
# Now combine to form Boolean Matrix
np.logical_and(d > 3, d % 2)
Student Exercise
In the empty Code cell below, write and execute code to make a four by four matrix that contains the integers 0 through 15. Add five to the first row, ten to the second row, fifteen to the third row, twenty to the fourth row, and display the resulting matrix. Next apply the cos function to the odd rows, and the sin function to the even rows (recall that Python indexing starts at zero), and display the resulting matrix. Finally, compute and display the cumulative sum for each row, and the cumulative product for each column.
Masked Arrays¶
As discussed in a previous lesson, NumPy provides support for masked arrays, where certain elements can be masked based on some criterion and ignored during subsequent calculations (i.e., these elements are masked). Masked arrays are in the numpy.ma package, and simply require a masked array to be created from a given array and a condition that indicates which elements should be masked. This new masked array can be used as a normal NumPy (multi-dimensional) array, except the masked elements are ignored. NumPy provides operations for masked arrays, allowing them to be used in a similar manner as normal NumPy arrays.
Correlation Measurements¶
In the Scatter Plots notebook, you learned to visually interpret relationships between data. Given two NumPy arrays, we can calculate analytic measures of the correlation between the two arrays to quantify any relationship. The two main techniques for computing a correlation are the Pearson and the Spearman correlation coefficients.
Pearson Correlation Coefficient¶
The Pearson correlation coefficient measures the linear relationship between two arrays. The computed value ranges between -1 and +1, with the former implying an exact negative linear relationship, and the latter an exact positive linear relationship. A value of zero indicates no correlation. The following figure, from Wikipedia, provides a visual illustration of two dimensional distributions of pints for various values of $r$.
In the spirit of this figure, there are several online games where you can view a two-dimensional distribution of points and guess the correlation. The first is an old-style arcade game, Guess the Correlation, while the second is a teaching game called Correlation Guessing Game. These games provide an excellent chance to improve your understanding of the relationship between an analytic correlation measure (introduced herein) and the visual appearance of a distribution of data (as covered in the Python Two-Dimensional Plotting notebook). This skill will be valuable as you learn to perform linear regression and more generally when working with multi-dimensional data in the future.
We will use the pearsonr function provided by the SciPy module, which computes and returns the $r$ value we are after as well as a two-tailed p-value, which we can safely ignore for now (this will be discussed in a subsequent course).
Spearman Correlation Coefficient¶
The Spearman correlation coefficient measures whether or not two arrays have a monotonic relationship, which basically means that as $x$ increases, $y$ continues to move in the same manner without changing direction (thus if $y$ is increasing, it never starts decreasing, and vice-versa). The computed value ranges between -1 and +1, with the former implying an exact negative monotonic relationship, and the latter an exact positive monotonic relationship. A value of zero indicates no correlation. We will use the spearmanr function provided by the SciPy module, which computes and returns the $rho$ value we are after as well as a two-tailed p-value for a hypothesis test, which we can safely ignore for now (this will be discussed in a subsequent course).
Covariance¶
In some cases, we do not want a single value indicating the relationship between two sets of data, and instead we want to know how each variable in an array (or matrix) varies in relationship to the other variables. This computation is known as the covariance, which produces a matrix whose diagonal elements represent the variance between the same variables and the off-diagonal elements are the covariances for a given pair of different variables. This is encoded in the following formula:
\begin{equation} \textrm{Cov(}x_i\textrm{, }y_i\textrm{)} = (x_i - \mu_x) (y_j - \mu_y) \end{equation}where $\mu_x$ and $\mu_y$ are the mean values for the $x$-array and $y$-array, respectively.
Given two equal-sized arrays, we can compute the covariance between them by using the NumPy cov function. To aid in the interpretation of the covariance, this matrix can be normalized by the variances to create a new matrix called the correlation matrix. The entries in this new matrix are Pearson product-moment correlation coefficients. We can calculate this correlation matrix, whose values are constrained to lie between -1 (perfect negative correlation) and +1 (perfect positive correlation) by using the NumPy corrcoef function.
The following code cell demonstrates these functions on correlated and random data, respectively.
import scipy.stats as st
# Create two correlated arrays
x = np.arange(10)
y = 9 - x
# Compute auto-correlations
pr1 = st.pearsonr(x, x)
sr1 = st.spearmanr(x, x)
# Compute cross-correlations
pr2 = st.pearsonr(x, y)
sr2 = st.spearmanr(x, y)
print('Pearsonr correlation for x,x = {0:6.4f}'.format(pr1[0]))
print('Spearmanr correlation for x,x = {0:6.4f}'.format(sr1[0]))
print('Pearsonr correlation for x,y = {0:6.4f}'.format(pr2[0]))
print('Spearmanr correlation for x,y = {0:6.4f}\n'.format(sr2[0]))
print('Covariance Matrix = \n', np.cov(x, y))
print()
print('Correlation Matrix = \n', np.corrcoef(x, y))
print()
# Generate random arrays
x = np.random.uniform(0, 1, 10)
y = np.random.uniform(0, 1, 10)
# Compute cross-correlations, should be close to null
prr = st.pearsonr(x, y)
srr = st.spearmanr(x, y)
print('Pearsonr correlation for random x,y = {0:6.4f}'.format(prr[0]))
print('Spearmanr correlation for random x,y = {0:6.4f}\n'.format(srr[0]))
Student Exercise
In the empty Code cell below, write and execute code to compute the Pearson and Spearman correlations for the columns in the Iris data set. You can load this data from the Seaborn module, and extract the columns as NumPy matrices to simplify the calculation. Do these values make sense in light of the scatter plots you generated previously?
Ancillary Information¶
The following links are to additional documentation that you might find helpful in learning this material. Reading these web-accessible documents is completely optional.
- NumPy tutorial
- NumPy cheatsheet
- NumPy lecture notes
- NumPy notebook demo
- The NumPy chapter from the book Python Data Science Handbook by Jake VanderPlas
© 2017: Robert J. Brunner at the University of Illinois.
This notebook is released under the Creative Commons license CC BY-NC-SA 4.0. Any reproduction, adaptation, distribution, dissemination or making available of this notebook for commercial use is not allowed unless authorized in writing by the copyright holder.